www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D called from Java

reply Frank Benoit <keinfarbton googlemail.com> writes:
I am working on a framework to automate the process of binding Java and
D. See the dsource project TioLink.

The main problem i see currently is, that both available D runtime
libraries - Phobos and Tango - do not have support for registering
Threads that are created "outside".

If a dll (or .so) is called from Java, it is called always with several
threads. Even in a single threaded Java application. Java runs the
finalizer in an own Thread, so it is essential to be able to register
all calling threads on the library entry points. Only with this, the D
GC can savely do its job.

Hopefully both runtime libs will have support for multithreaded
libraries in near future :)
Aug 18 2007
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Frank Benoit wrote:

 I am working on a framework to automate the process of binding Java and
 D. See the dsource project TioLink.
 
 The main problem i see currently is, that both available D runtime
 libraries - Phobos and Tango - do not have support for registering
 Threads that are created "outside".


...

 Hopefully both runtime libs will have support for multithreaded
 libraries in near future :)


Tango will have this once I have time for it.  Likely not long after the 
conference :-)


Sean
Aug 18 2007
parent reply Frank Benoit <keinfarbton googlemail.com> writes:
Sean Kelly schrieb:

 Tango will have this once I have time for it.  Likely not long after the
 conference :-)


The more i think about it, how will it work?

The d runtime (drt) needs to know all running threads, for the following
reason:
 * be able to stop all threads while the GC is running
 * provide a thread local storage container.
 * synchronization?

But does the drt also need to know about the termination of a thread?

If so, the registration must be paired with a deregistration on each
library entry point. This also means, register and deregister need to be
really fast because they are called every time the lib is entered or left.

export extern(System)
jint Java_test_Test_test( JNIEnv* env, jobject obj ) {
    Thread.register();
    scope( exit ) Thread.deregister();
    // ...
}

In the scenario, were the D GC is actually running and has all known
threads suspended, the Thread.register() call needs to block until the
GC completes. Probably the methods are better called like enterD/leaveD.

export extern(System)
jint Java_test_Test_test( JNIEnv* env, jobject obj ) {
    Thread.enterD();
    scope( exit ) Thread.leaveD();
    // ...
}

Hopefully this will not raise some new deadlock conditions.
1. D ctor remove held Java global references
2. this removal triggers the Java finalizer
3. another java object.finalize calls therefore the D lib, to signal the
end of the javas object lifecycle.
4. the call blocks and the dtor call from (1) does not return.
Aug 19 2007
parent Sean Kelly <sean f4.ca> writes:
Frank Benoit wrote:

 Sean Kelly schrieb:

 Tango will have this once I have time for it.  Likely not long after the
 conference :-)


 
 The more i think about it, how will it work?
 
 The d runtime (drt) needs to know all running threads, for the following
 reason:
  * be able to stop all threads while the GC is running
  * provide a thread local storage container.
  * synchronization?


The user would call some routine from each thread he wants to register, 
or perhaps simply pass the thread id.  The threads would have to be 
de-registered when they terminate as well.


 But does the drt also need to know about the termination of a thread?


Yes.  It may be possible to get away without explicitly notifying the GC 
of thread termination on Windows, based on how that code works, but on 
Unix I think such notification is necessary.


 If so, the registration must be paired with a deregistration on each
 library entry point. This also means, register and deregister need to be
 really fast because they are called every time the lib is entered or left.


The registration would involve a memory allocation for the proxy thread 
object and a mutex lock to place it in a list.


 export extern(System)
 jint Java_test_Test_test( JNIEnv* env, jobject obj ) {
     Thread.register();
     scope( exit ) Thread.deregister();
     // ...
 }
 
 In the scenario, were the D GC is actually running and has all known
 threads suspended, the Thread.register() call needs to block until the
 GC completes. Probably the methods are better called like enterD/leaveD.


I /think/ the only risk of being blocked by the GC will be allocating 
the proxy thread object.  It should be okay if a GC cycle is running 
after this time when the object is enlisted, though it would mean an 
attempted resume on a running thread.  This should work fine given how 
things are implemented, but I'll have to see if I can test it somehow to 
be sure.


 export extern(System)
 jint Java_test_Test_test( JNIEnv* env, jobject obj ) {
     Thread.enterD();
     scope( exit ) Thread.leaveD();
     // ...
 }
 
 Hopefully this will not raise some new deadlock conditions.
 1. D ctor remove held Java global references
 2. this removal triggers the Java finalizer
 3. another java object.finalize calls therefore the D lib, to signal the
 end of the javas object lifecycle.
 4. the call blocks and the dtor call from (1) does not return.


If the D GC is running then all threads it knows about will be suspended 
until the collection completes.  The mutex guarding memory allocations 
will be locked as well.  So operations in D object dtors must consider 
this.  It's possible to deadlock even without this Java integration if 
an attempt is made to enter a synchronized block that may already be 
held by a suspended thread.  In fact, this is why Tango does not have 
notifyRegister--using the routine to unregister a slot could easily 
deadlock for this exact reason.


Sean
Aug 19 2007
prev sibling parent reply Juan Jose Comellas <jcomellas gmail.com> writes:
You might have a problem if you have the D runtime and the JVM running
within the same process on Linux/Unix. Both the D and the Java garbage
collector (at least on JDK 1.4) use the SIGUSR1 and SIGUSR2 signals to
suspend and resume the garbage collection process, so you might want to
check whether there is any problem if a set of threads created in one
section of the process (e.g. D runtime) get a signal to start the garbage
collection from the other part of the process (e.g. JVM). I'm not sure if
the D runtime is prepared to handle this situation.

I had a problem with this when embedding the JVM in a program written in C
that already used these signals for its own purposes.


Frank Benoit wrote:


 I am working on a framework to automate the process of binding Java and
 D. See the dsource project TioLink.
 
 The main problem i see currently is, that both available D runtime
 libraries - Phobos and Tango - do not have support for registering
 Threads that are created "outside".
 
 If a dll (or .so) is called from Java, it is called always with several
 threads. Even in a single threaded Java application. Java runs the
 finalizer in an own Thread, so it is essential to be able to register
 all calling threads on the library entry points. Only with this, the D
 GC can savely do its job.
 
 Hopefully both runtime libs will have support for multithreaded
 libraries in near future :)
Aug 19 2007
parent reply Juan Jose Comellas <jcomellas gmail.com> writes:
Sorry, a small correction, both the D and the Java garbage collectors use
the SIGUSR1 and SIGUSR2 Unix signals to suspend and resume the process'
threads while the garbage collector is running.

Juan Jose Comellas wrote:


 You might have a problem if you have the D runtime and the JVM running
 within the same process on Linux/Unix. Both the D and the Java garbage
 collector (at least on JDK 1.4) use the SIGUSR1 and SIGUSR2 signals to
 suspend and resume the garbage collection process, so you might want to
 check whether there is any problem if a set of threads created in one
 section of the process (e.g. D runtime) get a signal to start the garbage
 collection from the other part of the process (e.g. JVM). I'm not sure if
 the D runtime is prepared to handle this situation.
 
 I had a problem with this when embedding the JVM in a program written in C
 that already used these signals for its own purposes.
 
 
 Frank Benoit wrote:
 

 I am working on a framework to automate the process of binding Java and
 D. See the dsource project TioLink.
 
 The main problem i see currently is, that both available D runtime
 libraries - Phobos and Tango - do not have support for registering
 Threads that are created "outside".
 
 If a dll (or .so) is called from Java, it is called always with several
 threads. Even in a single threaded Java application. Java runs the
 finalizer in an own Thread, so it is essential to be able to register
 all calling threads on the library entry points. Only with this, the D
 GC can savely do its job.
 
 Hopefully both runtime libs will have support for multithreaded
 libraries in near future :)
Aug 19 2007
parent reply Frank Benoit <keinfarbton googlemail.com> writes:
Is this a show stopper for Java/D bridges on linux?

Strange is, that there are only these two signals available for
application use.

Well you said 'same process'. I don't know, do you think it would be
possible spawning a new process and use IPC? Sounds very expensive.
Aug 19 2007
parent Juan Jose Comellas <jcomellas gmail.com> writes:
I don't know if it's a showstopper with D because I really haven't tried it,
but it was a showstopper for me. There is a way to remap the signals used
by the JVM but it was very cumbersome and it still presented problems
because the signals generated by the JVM kept interrupting system calls and
I had to patch a lot of code that did not belong to me to make everything
work (I was embedding the JVM in an Asterisk module).

I finally moved the JVM out of the process and communicated with it via a
socket. This change solved all my problems and made the system much easier
to debug. It did introduce more latency but it was acceptable for what I
needed to do. Besides avoiding the difficult task of analyzing crashes
where the Java code was part of the call stack, I also could forget about
dealing with all the JVM/JNI quirks (e.g. if you load the same dynamic
library from your C code and from the Java code at the same time all hell
breaks loose, etc.)

To sum it up, I'd really think twice before embedding the JVM inside another
process again.


Frank Benoit wrote:


 Is this a show stopper for Java/D bridges on linux?
 
 Strange is, that there are only these two signals available for
 application use.
 
 Well you said 'same process'. I don't know, do you think it would be
 possible spawning a new process and use IPC? Sounds very expensive.
Aug 19 2007