www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - C# Interop

reply Eelco Hoogendoorn <hoogendoorn.eelco gmail.com> writes:
Hi all,



Ive been using D for quite a while for hobby projects, and needless to say, it
makes C++ feel like masochism. Id like to convince my coworkers that adding D
in the mix is more than just another complication.


will no longer do me any good I fear.

Do I just create a D DLL with a bunch of free extern(C) function for
communication? What about marshalling? Is using unsafe pointers back and forth




compilation.)
Jan 31 2011
next sibling parent David Gileadi <gileadis NSPMgmail.com> writes:
On 1/31/11 2:25 PM, Eelco Hoogendoorn wrote:

 Hi all,



 Ive been using D for quite a while for hobby projects, and needless to say, it
 makes C++ feel like masochism. Id like to convince my coworkers that adding D
 in the mix is more than just another complication.


 will no longer do me any good I fear.

 Do I just create a D DLL with a bunch of free extern(C) function for
 communication? What about marshalling? Is using unsafe pointers back and forth




 compilation.)


It's been a number of years and it was in D1, so take this for the 
little value it may have, but the last time I tried making a D DLL used 

garbage collectors but didn't have the time to debug it (it was for a 
school project).  I'd be interested to hear if someone else has gotten 
this working.

-Dave
Jan 31 2011
prev sibling next sibling parent Michal Minich <michal.minich gmail.com> writes:
On Mon, 31 Jan 2011 21:25:11 +0000, Eelco Hoogendoorn wrote:


 Do I just create a D DLL with a bunch of free extern(C) function for
 communication?


Yes, why not.
You can pass struts in extern(C) functions...



 What about marshalling? Is using unsafe pointers back and
 forth the best I can do? 


No.






Yes. 



seamlessly and heard of some D projects using it.

one starter link:
http://www.dsource.org/projects/juno/wiki/ComProgramming
Jan 31 2011
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 31 Jan 2011 16:25:11 -0500, Eelco Hoogendoorn  
<hoogendoorn.eelco gmail.com> wrote:


 Hi all,


 interop.

 Ive been using D for quite a while for hobby projects, and needless to  
 say, it
 makes C++ feel like masochism. Id like to convince my coworkers that  
 adding D
 in the mix is more than just another complication.


 CLI
 will no longer do me any good I fear.

 Do I just create a D DLL with a bunch of free extern(C) function for
 communication?


Basically, yes. I've been doing this for a project and it works reasonably  
well. I have one module based on From D's/bugzilla's public domain example  
code to properly enable the dll and then declare the interop functions as  
export extern(C) {}. There's also some general helper functions you'll  
want to write. Although exceptions can propagate to .NET, they just turn  
into a System.Runtime.InteropServices.SEHException exception, so you'll  
want to wrap all your export function in a try-catch block and save the  
error message to a global variable. Then you can call a lastError function  
to get the actual error string. Oh, and remember .NET defaults to  
wstrings, not strings. The other helper function you'll want is a way to  



 What about marshalling?



anything other then calling basic C system calls. Lucky, you really only  
need it for arrays and structs. Objects can be be treated as handles  

functions whose first argument is the object's handle. But you'll also  

a lot of this, I'd recommend writing a mixin to generate all the free  
functions on the D side, and looking into the dynamic language features of  

yet)


 Is using unsafe pointers back and forth

 mode,



I don't know if it's the best you can do, but it does work.



 compilation.)


Lastly, D DLLs will only work on Vista/Windows 7/later. They will not work  
on XP. This is due to a long known bug with DLLs and thread local storage  

isn't 64-bit compatible yet. (Walter is hard at work on a 64-bit version  
of DMD, but it will be Linux only at first, with Windows following  
sometime later)

I've listed some example code from my project below:

// Written in the D Programming Language (www.digitalmars.com/d)
///Basic DLL setup and teardown code. From D's/bugzilla's public domain  
example code.
module dll;

import std.c.windows.windows;
import std.c.stdlib;
import core.runtime;
import core.memory;

extern (Windows)
BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved) {
     switch (ulReason) {
         case DLL_PROCESS_ATTACH:
             Runtime.initialize();
             break;
         case DLL_PROCESS_DETACH:
             Runtime.terminate();
             break;
         case DLL_THREAD_ATTACH:
         case DLL_THREAD_DETACH:
             return false;
     }
     return true;
}


D code:
private {
     wstring last_error_msg = "No error";                    /// The last  
encountered by the dose server

     __gshared Object[Object] pinned;                        /// Hash of  
pinned objects
     void pin(Object value)   { pinned[ value ] = value;  }  /// Pin   an  
object

     /// Stores a string as the last error
     void lastError(string str) {
         wstring err;
         auto app = appender(&err);
         foreach (dchar c; str) app.put(c);
         last_error_msg = err;
         enforce(false);
     }
}
export extern(C) {
     ref wstring lastError() { return  last_error_msg; }     /// returns:  
the last error message, used for exception marshaling
}


         [DllImport(gpu_dll)]
         [return: MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef =  
typeof(DString))]
         private static extern string lastError();

         /// <summary>
         /// A custom marshaller for D's strings (char[]) passed by ref
         /// </summary>
         private class DString : ICustomMarshaler
         {
             //[ThreadStatic]
             //private static ICustomMarshaler _instance = new DString();
             private IntPtr ptr = IntPtr.Zero;

             /// <summary>
             /// Factory method
             /// </summary>
             /// <param name="pstrCookie"></param>
             /// <returns></returns>
             static ICustomMarshaler GetInstance(String pstrCookie)
             {
                 return new DString();
             }

             /// <summary>
             /// Convert a pointer to a D array to a managed T[]
             /// </summary>
             /// <param name="pNativeData"></param>
             /// <returns></returns>
             public Object MarshalNativeToManaged(IntPtr pNativeData)
             {
                 Int32 length = Marshal.ReadInt32(pNativeData, 0);
                 IntPtr data = Marshal.ReadIntPtr(pNativeData, 4);
                 char[] result = new char[length];
                 Marshal.Copy(data, result, 0, length);
                 return new string(result);
             }

             /// <summary>
             /// Convert a managed T[] to a pointer to a D array
             /// </summary>
             /// <param name="ManagedObj"></param>
             /// <returns></returns>
             public IntPtr MarshalManagedToNative(Object ManagedObj)
             {
                 char[] managed = ((string)ManagedObj).ToCharArray(); ;
                 ptr = Marshal.AllocHGlobal(8 + managed.Length * 2); //  
unicode = 16 bytes, sigh
                 IntPtr data = (IntPtr)((UInt32)ptr + 8);
                 Marshal.Copy(managed, 0, data, managed.Length);
                 Marshal.WriteInt32(ptr, 0, managed.Length);
                 Marshal.WriteIntPtr(ptr, 4, data);
                 return ptr;
             }

             /// <summary>
             /// Delete the marshaled D array
             /// </summary>
             /// <param name="pNativeData"></param>
             public void CleanUpNativeData(IntPtr pNativeData)
             {
                 if (ptr == pNativeData)
                     Marshal.FreeHGlobal(pNativeData);
             }

             /// <summary>
             /// Clean up managed data (i.e. do nothing)
             /// </summary>
             /// <param name="ManagedObj"></param>
             public void CleanUpManagedData(Object ManagedObj) { }

             /// <returns>The size of a D array struct (8)</returns>
             public int GetNativeDataSize()
             {
                 return 8;
             }
         }
Jan 31 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/1/11, Robert Jacques <sandford jhu.edu> wrote:

 Lastly, D DLLs will only work on Vista/Windows 7/later. They will not work
 on XP. This is due to a long known bug with DLLs and thread local storage
 in general on XP.


Is there a bugzilla link for this?
Jan 31 2011
parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 31 Jan 2011 22:32:35 -0500, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 On 2/1/11, Robert Jacques <sandford jhu.edu> wrote:

 Lastly, D DLLs will only work on Vista/Windows 7/later. They will not  
 work
 on XP. This is due to a long known bug with DLLs and thread local  
 storage
 in general on XP.


 Is there a bugzilla link for this?


This isn't a D issue. It's a well documented problem with thread local  
storage with all DLLs on Windows XP. I haven't seen a specific bugzilla on  
this issue, but there are several issues with writting/loading DLLs in D,  
which I assume will be addressed at/about the time Linex .SOs are, which  
are listed as next/in progress after 64-bit support.
Jan 31 2011
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Robert Jacques wrote:

 On Mon, 31 Jan 2011 16:25:11 -0500, Eelco Hoogendoorn 
 <hoogendoorn.eelco gmail.com> wrote:
 [...]
 
 Lastly, D DLLs will only work on Vista/Windows 7/later. They will not 
 work on XP. This is due to a long known bug with DLLs and thread local 

 as DMD isn't 64-bit compatible yet. (Walter is hard at work on a 64-bit 
 version of DMD, but it will be Linux only at first, with Windows 
 following sometime later)
 


XP TLS support with dynamically loaded DLLs is fixed for some time now 
with a workaround implemented in druntime. Also, DLLs can be used in 
multi-threading environments.


 I've listed some example code from my project below:

 // Written in the D Programming Language (www.digitalmars.com/d)
 ///Basic DLL setup and teardown code. From D's/bugzilla's public domain
 example code.
 module dll;

 import std.c.windows.windows;
 import std.c.stdlib;
 import core.runtime;
 import core.memory;

 extern (Windows)
 BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved) {
     switch (ulReason) {
         case DLL_PROCESS_ATTACH:
             Runtime.initialize();
             break;
         case DLL_PROCESS_DETACH:
             Runtime.terminate();
             break;
         case DLL_THREAD_ATTACH:
         case DLL_THREAD_DETACH:
             return false;
     }
     return true;
 }


This DLLMain code is a bit outdated (is it D1?), the current proposed 
version is here: http://www.digitalmars.com/d/2.0/dll.html

Unfortunately, there is a regression in the latest dmd release (2.051): 
http://d.puremagic.com/issues/show_bug.cgi?id=5382 that causes TLS not 
to be initialized for new threads created by the host application.

Rainer
Feb 01 2011
next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 01 Feb 2011 03:05:13 -0500, Rainer Schuetze <r.sagitario gmx.de>  
wrote:

 Robert Jacques wrote:

 On Mon, 31 Jan 2011 16:25:11 -0500, Eelco Hoogendoorn  
 <hoogendoorn.eelco gmail.com> wrote:
 [...]
  Lastly, D DLLs will only work on Vista/Windows 7/later. They will not  
 work on XP. This is due to a long known bug with DLLs and thread local  

 as DMD isn't 64-bit compatible yet. (Walter is hard at work on a 64-bit  
 version of DMD, but it will be Linux only at first, with Windows  
 following sometime later)


 XP TLS support with dynamically loaded DLLs is fixed for some time now  
 with a workaround implemented in druntime. Also, DLLs can be used in  
 multi-threading environments.


Yes, I pointed out in another thread that D loading D DLLs can work around  
this issue, but the original post was about calling a D DLL from another  




  > I've listed some example code from my project below:


[snip]


 This DLLMain code is a bit outdated (is it D1?), the current proposed  
 version is here: http://www.digitalmars.com/d/2.0/dll.html


Thanks. It was D2, but it was forked a while ago. Given that the  
recommended way of doing this might change in the future, a string mixin  
in core.dll_helper might be appropriate.
Feb 01 2011
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Robert Jacques wrote:

 On Tue, 01 Feb 2011 03:05:13 -0500, Rainer Schuetze <r.sagitario gmx.de> 
 wrote:

 XP TLS support with dynamically loaded DLLs is fixed for some time now 
 with a workaround implemented in druntime. Also, DLLs can be used in 
 multi-threading environments.


 
 Yes, I pointed out in another thread that D loading D DLLs can work 
 around this issue, but the original post was about calling a D DLL from 


 Hmm...)


The workaround is not about D loading a D DLL. Visual D lives happily in 

dll_process_attach() that sets up TLS for existing threads and patches 
the loader structures to make XP think the DLL was loaded at startup 
(where implicite TLS works). The downside is that the DLL cannot be 
unloaded, though.


 

  > I've listed some example code from my project below:


 
 [snip]
 

 This DLLMain code is a bit outdated (is it D1?), the current proposed 
 version is here: http://www.digitalmars.com/d/2.0/dll.html


 
 Thanks. It was D2, but it was forked a while ago. Given that the 
 recommended way of doing this might change in the future, a string mixin 
 in core.dll_helper might be appropriate.


I don't like mixins too much, but a standard_DllMain that you can 
forward to from DllMain, might be a good idea to include into the 
runtime library.
Feb 01 2011
parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 01 Feb 2011 13:33:20 -0500, Rainer Schuetze <r.sagitario gmx.de>  
wrote:


 Robert Jacques wrote:

 On Tue, 01 Feb 2011 03:05:13 -0500, Rainer Schuetze  
 <r.sagitario gmx.de> wrote:

 XP TLS support with dynamically loaded DLLs is fixed for some time now  
 with a workaround implemented in druntime. Also, DLLs can be used in  
 multi-threading environments.


  Yes, I pointed out in another thread that D loading D DLLs can work  
 around this issue, but the original post was about calling a D DLL from  


 Hmm...)


 The workaround is not about D loading a D DLL. Visual D lives happily in  

 dll_process_attach() that sets up TLS for existing threads and patches  
 the loader structures to make XP think the DLL was loaded at startup  
 (where implicite TLS works). The downside is that the DLL cannot be  
 unloaded, though.


Thanks, again. Though the pros and cons of this should be listed in the  
docs somewhere.


  > I've listed some example code from my project below:


  [snip]


 This DLLMain code is a bit outdated (is it D1?), the current proposed  
 version is here: http://www.digitalmars.com/d/2.0/dll.html


  Thanks. It was D2, but it was forked a while ago. Given that the  
 recommended way of doing this might change in the future, a string  
 mixin in core.dll_helper might be appropriate.


 I don't like mixins too much, but a standard_DllMain that you can  
 forward to from DllMain, might be a good idea to include into the  
 runtime library.


Yes, on second thought, a standard_DllMain is the better solution.
Feb 01 2011
prev sibling parent reply Richard Webb <webby beardmouse.org.uk> writes:
 Also, DLLs can be used in multi-threading environments.




On a related note, i'm having a bit of a problem with a D dll at the moment.

I have an Outlook COM Addin that is written in D2 using the Juno library and
that
is running ok, but i'm now trying to use a D dll from a COM addin written in C++
and i'm getting a crash inside dll_process_attach.

The only thing i get in the call stack is gcx.mark, but it appears that the
crash
is due to the GC running midway through the thread setup code. Disabling the GC
during the thread_moduleTlsCtor() calls avoids the crash, but that might just be
covering the problem up.

This is with the latest druntime/phobos code from a couple of days ago plus your


Any ideas?
Feb 02 2011
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Richard Webb wrote:

 Also, DLLs can be used in multi-threading environments.




 
 On a related note, i'm having a bit of a problem with a D dll at the moment.
 
 I have an Outlook COM Addin that is written in D2 using the Juno library and
that
 is running ok, but i'm now trying to use a D dll from a COM addin written in
C++
 and i'm getting a crash inside dll_process_attach.
 
 The only thing i get in the call stack is gcx.mark, but it appears that the
crash
 is due to the GC running midway through the thread setup code. Disabling the GC
 during the thread_moduleTlsCtor() calls avoids the crash, but that might just
be
 covering the problem up.


If there is a garbage collection going on, I assume that you do quite a 
lot of initialization for TLS. Are you using callbacks into the C++ DLL 
from the module ctors? This might be dangerous...

You might want to try this version of impersonate thread(), that might 
work better if the C++ part also uses thread local storage:

         // execute function on the TLS for the given thread
         static void impersonate_thread( uint id, void function() fn )
         {
             if( id == GetCurrentThreadId() )
             {
                 fn();
                 return;
             }

             // temporarily set current TLS array pointer to the array 
pointer of the referenced thread
             void** curteb = getTEB();
             void** teb    = getTEB( id );
             assert( teb && curteb );

             void** curtlsarray = cast(void**) curteb[11];
             void** tlsarray    = cast(void**) teb[11];
             if( !curtlsarray || !tlsarray )
                 return;

             curteb[11] = tlsarray;
             fn();
             curteb[11] = curtlsarray;
         }

Especially loading more DLLs from the TLS init code will still be 
troublesome...

Rainer
Feb 02 2011
parent reply Richard Webb <webby beardmouse.org.uk> writes:
 I assume that you do quite a lot of initialization for TLS




I'm not directly, but Juno has a number of static class constructors that
allocate
static members, and Outlook creates lots of threads (for example, i see the
static
constructor of the module juno.com.core get called 20+ times on startup).
The Juno lib predates the TLS changes in D2 so i guess that some of these static
variables don't need to be thread local, but i haven't had chance to
investigate that.



Are you using callbacks into the C++ DLL from the module ctors




The D dll just exports a couple of 'C' functions that the C++ dll calls. There
are
no calls in the other direction.



 that might work better if the C++ part also uses thread local storage




The C++ dll uses ATL for it's COM stuff, and i'm not sure what that does with
TLS
in the background.

I'll try your suggested change tommorow.

Thanks,
Richard Webb
Feb 02 2011
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
I forgot one more thing: I don't know how juno deals with it, but I 
consider the default memory management of COM objects in phobos/druntime 
rather unusable. It leaks memory and can easily cause garbage collection 
on objects still referenced by COM objects. That's why I use 
http://d.puremagic.com/issues/show_bug.cgi?id=4092

Rainer

Richard Webb wrote:

 I assume that you do quite a lot of initialization for TLS




 
 I'm not directly, but Juno has a number of static class constructors that
allocate
 static members, and Outlook creates lots of threads (for example, i see the
static
 constructor of the module juno.com.core get called 20+ times on startup).
 The Juno lib predates the TLS changes in D2 so i guess that some of these
static
 variables don't need to be thread local, but i haven't had chance to
investigate that.
 
 

 Are you using callbacks into the C++ DLL from the module ctors




 
 The D dll just exports a couple of 'C' functions that the C++ dll calls. There
are
 no calls in the other direction.
 
 

 that might work better if the C++ part also uses thread local storage




 
 The C++ dll uses ATL for it's COM stuff, and i'm not sure what that does with
TLS
 in the background.
 
 I'll try your suggested change tommorow.
 
 Thanks,
 Richard Webb
Feb 03 2011
parent Richard Webb <webby beardmouse.org.uk> writes:
It overrides new and allocates class instances using malloc. The official
version
calls addRange from new, and removeRange/free from Release when the reference
count is 0, but i changed my local version to use add/remove(Root) instead of
range at some point.
I'm only currently trying to use external com objects rather than creating my
own,
so this isn't an issue here.


fwiw, i just marked some static variables in juno as __gshared to work around
the
problem for now, and it seems to be loading ok. Juno could really do with some
official maintainance work to keep up with D2 changes, but the author seems to
have left a while back  :-(
Feb 06 2011