JDirect is (was?) an enabling technology that lets Java programs directly call operating system routines, without native glue libraries. JDirect was developed at Apple at a time when development tools were not bundled with the operating system and were very awkward to drive from inside an application. In contrast, UNIX has a long tradition of language compilers that are layered on top of the host C compiler (e.g. Sather, SISAL and many others).
Since the introduction of Mac OS X, Apple's version of gcc is freely available with the operating system itself. Although it isn't installed by default, anybody doing any development will have it installed, so for the purposes of this discussion, it is assumed that Apple's latest version of gcc is part of the operating system.
JNI Direct is a new implementation of JDirect which has the same advantages as older versions, but has the additional advantage of being written in 100% pure Java itself, which solves an important bootstrapping problem. If JDirect isn't built into the JVM, how does one obtain a working version of JDirect without distributing binary libraries? JNI Direct solves this problem by using the host operating system's C compiler to generate glue libraries that can be loaded dynamically. If binary distribution is acceptable, the results of running JNI Direct on a developer's machine can be cached and distributed with Java classes that use JNI Direct.
Apple's recent postings (late 2002) on the java development mailing list indicate that support for JDirect may be waning. Since the most modern version of JDirect is built on top of JNI, it is a shame that there isn't a cross-platform version of JDirect available that can be used on any operating system that has a JVM that supports JNI. This paper describes a new implementation of JDirect that remedies this situation.
Here is a Java class that uses JNIDirect to access ISO C89 standard C functions to read environment variables:
class Environment {
static String get(String name) {
int str = getenv((name + '\0').getBytes());
if (str != 0) {
int length = strlen(str);
byte[] buffer = new byte[length];
memcpy(buffer, str, length);
return new String(buffer);
}
return null;
}
static void put(String name, String value) {
putenv((name + '=' + value + '\0').getBytes());
}
private static native int getenv(byte[] name);
private static native int strlen(int str);
private static native int memcpy(byte[] dest, int source, int length);
private static native void putenv(byte[] buffer);
static {
jnidirect.Linker.link(Environment.class);
}
} |
This demonstrates a couple of key concepts about using JNIDirect. First, it is very low-level. When calling standard C library functions, you will need to understand how to represent low-level concepts like pointers in Java. In this example, the return value of getenv() is represented as an int, but it is in fact a pointer to a C string (char*). Since Java can't dereference pointers directly, there are a couple of ways to convert this (char*) into a Java string. The approach given here is to use strlen() to determine the length of the string, allocate a byte[] array of appropriate size, and copy the bytes using memcpy(). (I've purposefully glossed over details such as character encoding for brevity.)
A call to jnidirect.Linker.link() generates JNI glue for the native methods of the class passed as the first parameter. Each declared native method is analyzed using reflection, and appropriate C glue code is generated. All of the C code is accumulated into a temporary source file, compiled using the host C compiler, and linked into a shared library, which is dynamically loaded into the JVM using System.load().
class Environment {
public static String get(String name) {
int val = getenv((name + '\0').getBytes());
if (val != 0) {
StringBuffer buffer = new StringBuffer();
int ptr = val;
char ch;
while ((ch = (char) $read$byte$(ptr++)) != 0)
buffer.append(ch);
return buffer.toString();
}
return null;
}
static void put(String name, String value) {
putenv((name + '=' + value + '\0').getBytes());
}
private static native int getenv(byte[] name);
private static native byte $read$byte$(int pointer);
private static native void putenv(byte[] buffer);
static {
jnidirect.Linker.link(Environment.class);
}
} |
Native methods of the form type $read$type$(int pointer) and void $write$type$(int pointer, type value) are automatically generated by JNI Direct to perform the indicated memory primitive operation. The implementation of get() in the example above uses $read$byte$() to read the bytes from the (char*) pointer directly. Although this implementation uses fewer native methods to do the same thing as the previous example, it is more complex, and not for the faint of heart.
The previous examples made use of functions that are part of the standard C library. To specify a particular Mac OS X framework to link against, pass a String[] parameter to jnidirect.Linker.link():
class CFString {
private int ref;
public CFString(String string) {
ref = CFStringCreateWithCharacters(0, string.toCharArray(), string.length());
}
public void show() {
CFShow(ref);
}
protected void finalize() {
if (ref != 0)
CFRelease(ref);
}
private static native int CFStringCreateWithCharacters(int alloc, char[] chars, int numChars);
private static native void CFShow(int ref);
private static native void CFRelease(int ref);
static {
String[] frameworks = { "CoreFoundation" };
jnidirect.Linker.link(CFString.class, frameworks);
}
} |
Method closures provide a native callback mechanism, which combine a function pointer, a java.lang.reflect.Method, and a java.lang.Object into a single unit. Here's an example:
public class pthread {
private int start_closure;
private int pthread;
public pthread() {
start_closure = jnidirect.Linker.newMethodClosure(runMethod, this);
}
protected void finalize() {
jnidirect.Linker.disposeMethodClosure(start_closure);
}
public void start() {
int[] pthread_ptr = { 0 };
int [] attributes = default_attributes();
int rv = pthread_create(pthread_ptr, attributes, start_closure, 0);
if (rv != 0) throw new Error("pthread_create returned " + rv);
rv = pthread_attr_destroy(attributes);
if (rv != 0) throw new Error("pthread_attr_destroy returned " + rv);
pthread = pthread_ptr[0];
}
public void stop() {
int rv = pthread_cancel(pthread);
if (rv != 0) throw new Error("pthread_cancel returned " + rv);
}
protected int run(int arg) {
System.out.println("pthread.run() here!");
return 0;
}
private int[] default_attributes() {
// #define __PTHREAD_ATTR_SIZE__ 36
// struct _opaque_pthread_attr_t { long sig; char opaque[__PTHREAD_ATTR_SIZE__]; } pthread_attr_t;
int[] attr = new int[10];
int rv = pthread_attr_init(attr);
if (rv != 0) throw new Error("pthread_attr_init returned " + rv);
return attr;
}
private static native int pthread_attr_init(/* pthread_attr_t* */ int[] attr);
private static native int pthread_attr_destroy(/* pthread_attr_t* */ int[] attr);
private static native int pthread_create(/* pthread_t* */ int[] thread,
/* const pthread_attr_t* */ int[] attr,
/* void* (*start_routine)(void*) */ int start_closure,
/* void* */ int arg);
private static native int pthread_cancel(/* pthread_t */ int thread);
private static Method runMethod;
static {
try {
runMethod = pthread.class.getDeclaredMethod("run", new Class[] { Integer.TYPE });
jnidirect.Linker.link(pthread.class, new Method[] { runMethod });
} catch (Exception ex) {
}
}
} |
This is a simple wrapper for POSIX threads, which prints out a message when the run() method is executed. A C function pointer is created by the call to jnidirect.Linker.newMethodClosure(). The next section describes the technical details of how this works.
When calling jnidirect.Linker.link() with an array of methods, a JNI_OnLoad() function is generated that calls back to Linker.registerMethodClosureFunctions(), passing it the same array of methods, and an array of corresponding glue function pointers. For the pthread example above, a static C function is generated that looks like this:
static jint Closure_pthread_run(jint arg0)
{
JNIEnv* env;
struct { void* glue; jobject self; jmethodID method; } *closure;
asm("mr %0,r12" : "=r" (closure));
if ((*theJVM)->GetEnv(theJVM, (void**)&env, JNI_VERSION_1_2) != JNI_OK) {
jint rv;
(*theJVM)->AttachCurrentThread(theJVM, (void**)&env, NULL);
rv = (*env)->CallIntMethod(env, closure->self, closure->method, arg0);
(*theJVM)->DetachCurrentThread(theJVM);
} else {
return (*env)->CallIntMethod(env, closure->self, closure->method, arg0);
}
}; |
A call to newMethodClosure() generates PowerPC glue code that wraps the closure function pointer, passing it a pointer to the 12-byte closure structure in register r12. Multiple closure instances can be created, wrapping distinct object instances. This is crucial, so that a developer has the flexibility to use multiple pthread instances, with each having different behavior.
Here's what the glue looks like:
ClosureGlue:
lis r12,0x0000 ; load r12 with address of ClosureVector
ori r12,r12,0x0000
lwz r0,0(r12)
mtctr r0
bctr
ClosureVector:
dc.l 0 ; address of Closure_pthread_run
dc.l 0 ; weak global reference to closure instance object
dc.l 0 ; jmethodID of method to call |
This design provides a way to reuse the same closure function for as many instances as are necessary.
When newMethodClosure() is called, a weak JNI global reference to the object is created, so that the native callback function can refer to the object from an arbitrary Java thread, but won't keep the object from being garbage collected.
Because a pthread can be garbage collected, to prevent referencing dangling Java objects, a finalize() method is provided which removes the signal handler, and frees the memory of the method closure object.
A Windows version of the jnidirect.Linker class is planned. Ironically, the first generation implementation of the JNIDirect concept was initially written for Windows, and then it was ported to Mac OS X as JDirect evolved.
The objective C runtime provides an API whereby classes can be generated at runtime. Therefore, a mechanism similar to method closures could be implemented that binds a Java method to an objective C method using function pointers. The relevant APIs are here:
/usr/include/objc/objc-class.h
/usr/include/objc/objc-runtime.h
It will need a metaclass if it has any class methods (that's where those go), otherwise the metaclass can be the NSObject's metaclass?
Some sample code from: Introduction to The Objective-C Programming Language
BOOL CreateClassDefinition( const char * name, const char * superclassName)
{
struct objc_class * meta_class;
struct objc_class * super_class;
struct objc_class * new_class;
struct objc_class * root_class;
//
// Ensure that the superclass exists and that someone
// hasn't already implemented a class with the same name
//
super_class = (struct objc_class *) objc_lookUpClass(superclassName);
if (super_class == nil)
{
return NO;
}
if (objc_lookUpClass (name) != nil)
{
return NO;
}
// Find the root class
root_class = super_class;
while( root_class->super_class != nil )
{
root_class = root_class->super_class;
}
// Allocate space for the class and its meta class
new_class = calloc(2, sizeof(struct objc_class));
meta_class = &new_class[1];
// setup class
new_class->isa = meta_class;
new_class->info = CLS_CLASS;
meta_class->info = CLS_META;
//
// Create a copy of the class name.
// For efficiency, we have the metaclass and the class itself
// to share this copy of the name, but this is not a requirement
// imposed by the runtime.
//
new_class->name = malloc(strlen (name) + 1);
strcpy((char*)new_class->name, name);
meta_class->name = new_class->name;
//
// Allocate empty method lists
// We can add methods later.
//
new_class->methodLists = calloc(1, sizeof(struct objc_method_list *));
meta_class->methodLists = calloc(1, sizeof(struct objc_method_list *));
//
// Connect the class definition to the class hierarchy.
// First, connect the class to the superclass
// Then connect the metaclass to the metaclass of the superclass
// Then connect the metaclass of the metaclass to
// the metaclass of the root class
new_class->super_class = super_class;
meta_class->super_class = super_class->isa;
meta_class->isa = (void *)root_class->isa;
// Finally, register the class with the runtime.
objc_addClass(new_class);
return YES;
} |
class_addMethods().
typedef struct objc_class *Class;
typedef struct objc_object {
Class isa;
} *id;
typedef struct objc_selector *SEL;
typedef id (*IMP)(id, SEL, ...);
struct objc_method {
SEL method_name;
char *method_types;
IMP method_imp;
}; |
Have to make sure each selector is mapped with sel_registerName().
Could just generate .m files with appropriate wrapper classes. A strong reference to the Java object would be held by the wrapper object, which would be released when the reference count goes to 0. These would all have to manage attaching to the JNIEnv on calls in from non-Java threads, the same as the method closures.
Alternatively, could use method closures as a quick way
Reimplementing the method closure mechanism proved to be a fairly challenging undertaking. Various approaches were considered, but a constraint of never straying from the JNI Direct concept itself was adhered to. This appendix describes additional primitive operations that were created to make method closures work.
The main problem to be solved for method closures was how to set up the 12-byte
closure vectors, and how to make them available to the generated closure
functions. Since the closure functions are C functions that must use the JNI API
to call into Java, a way to convert Java object references into JNI references
was needed. An inspiration that came to the author was that additional primitive
operations similar to $read$type$ and $write$type$ could be created, that would
provide ways to perform JNI operations. Two primitives were created,
$jni$NewWeakGlobalRef$ and $jni$FromReflectedMethod$. $jni$NewWeakGlobalRef$
converts a Java object reference into JNI weak global reference, and returns it
as a Java int value. $jni$FromReflectedMethod converts a
java.lang.reflect.Method reference to a jmethodID and returns it as an int
value. The main epiphany here was that the jnidirect.Linker class could itself
make use of its own facilities to make this work. While this may seem like a
circular dependence, it really isn't, because only the method closure mechanism
has to use these primitives.
Clearly, a whole family of calls to JNI functions could be created, such as
methods to convert from jmethodID values back to Method references. The
JNIDirect architecture is proving to be remarkably flexible.
A wrapper class to load .dylib images, and to look up symbols in them was
implemented. To make this truly useful, a $call$ primitive was added to
jnidirect.Linker to provide a way to call an arbitrary function pointer from
Java:
public static void test() {
try {
NSImage system = new NSImage(new File("/usr/lib/libSystem.dylib"));
int printf = system.getSymbolAddress("printf");
$call$(printf, "hello indirect printf.\n\0".getBytes());
} catch (IOException ex) {
ex.printStackTrace(System.err);
}
}
private static native int $call$(int f, byte[] format); |
This is very similar to the Mixed Mode manager API, CallUniversalProc, but it is
far simpler. It simply creates a C glue function like this:
JNIEXPORT jint JNICALL Java_jnidirect_NSImage__00024call_00024(JNIEnv *env, jclass thisClass, jint arg0, jbyteArray arg1)
{
jint rv;
jbyte* marg1;
jboolean isCopy;
jint (*f) () = (void*)arg0;
marg1 = (arg1 ? (*env)->GetByteArrayElements(env, arg1, &isCopy) : NULL);
rv = (*f)(marg1);
if (arg1) (*env)->ReleaseByteArrayElements(env, arg1, marg1, JNI_COMMIT);
return rv;
} |
This shares the same implementation as glue for other native functions, but instead of calling a named function, it instead makes an indirect call through the function pointer passed in the first argument.
Copyright ©2003 by Patrick C. Beard. All rights reserved.