www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D struct with String type accessing from Python

reply alchemypy <me alchemypy.com> writes:
Hi Everyone, I am new to Dlang and i am trying to consume .so 
(Dlang based dynamic library) from Python.

My D program has something like below,


	struct gph
	{
		string x;
	}

	struct myStruct
	{
		pragma(mangle, "Print_gph")
		void Print_gph(gph g)
		{
			stderr.writeln("here: gph ");
			stderr.writeln(g.x);
		
		}
	}

i have created a .so for the same and my Python code looks as 
follows:

	from ctypes import *

	class gph(Structure):
		_fields_ = (
			('x_p', c_char_p),
			('x_len', c_size_t)
		)
		def __init__(self, x):
			self.x_p = x
			self.x_len = len(x)


	so_path = "./test.so"
	lib = CDLL(so_path)

	sample_struct = gph(b'gph')
	lib.SubOcc_gph(c_void_p(),sample_struct)


When i executed i got bellow error,


	here: gph
	src/rt/dwarfeh.d:330: uncaught exception reached top of stack
	This might happen if you're missing a top level catch in your 
fiber or signal handler
	std.exception.ErrnoException /usr/include/dmd/phobos/std/stdio.d(3170):
Enforcement failed (Bad address)
	Aborted


Can you please help me what went wrong here ?
Dec 25 2022
parent reply cc <cc nevernet.com> writes:
On Monday, 26 December 2022 at 03:05:33 UTC, alchemypy wrote:

 Can you please help me what went wrong here ?



believe D stores the size of a slice before the pointer.

Not sure if this will help, I abandoned this idea after playing 
with it for a while.  I think I had trouble getting the D 
compiler to send one of its own strings/slices to an extern(C) 
function.  I believe it worked fine if I sent the string result 
AS a struct that separated out the size and pointer, but failed 
for unexplained reasons if I just tried to send the basic 
string/immutable(char)[] as-is.



		[StructLayout(LayoutKind.Sequential, Size=16, Pack=1)]
		public struct DString {
			public ulong length;
			public IntPtr ptr;
			public string str {
				get {
					byte[] b = new byte[length];
					for (int i = 0; i < (int)length; i++) {
						b[i] = Marshal.ReadByte(ptr, i);
					}
					return Encoding.UTF8.GetString(b);
				}
			}
		}
		[DllImport("mydll.dll")]
		private static extern void DDLL_testString(out DString dd);
		public static string testString() {
			DString d;
			DDLL_testString(out d);
			return d.str;
		}
```
```d
// D DLL
static struct DString {
	size_t length;
	immutable(char)* ptr;
	this(string str) {
		length = str.length;
		ptr = str.ptr;
	}
	void opAssign(string str) {
		length = str.length;
		ptr = str.ptr;
	}
}
//extern(C) export DString testString() {
extern(C) export void testString(out string ret) {
	string str = "hello王".idup;
	//return DString(str);
	ret = str;
}
```
Dec 25 2022
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 12/25/22 20:41, cc wrote:

 D stores the size of a slice before the pointer.


And that's in the ABI spec:

   https://dlang.org/spec/abi.html#arrays

Another important topic is object lifetimes because Python and D garbage 
collectors not necessarily know about objects created by the other side.

In any case, I have a presentation "Exposing a D Library to Python 
Through a C API", which touches up on these topics:

   https://www.youtube.com/watch?v=FNL-CPX4EuM

(Note: Yes, that was covid hair. :) )

Ali
Dec 26 2022
parent reply cc <cc nevernet.com> writes:
On Monday, 26 December 2022 at 16:44:37 UTC, Ali Çehreli wrote:

 In any case, I have a presentation "Exposing a D Library to 
 Python Through a C API", which touches up on these topics:


I found the problem I was running into before.  Why does 
returning a struct, or sending a D string via an out parameter, 
work here, but simply returning a D string does not?

D DLL
```d
static struct DString {
	size_t length;
	immutable(char)* ptr;
	this(string str) {
		length = str.length;
		ptr = str.ptr;
	}
	void opAssign(string str) {
		length = str.length;
		ptr = str.ptr;
	}
}

static this() {
	cache = null; // DLL memory error if not initialized here
}
string[string] cache;

export string MyDLL_testStringReturn() {
	string s = "hello";
	cache.require(s);
	writefln("[D] pass: %s", s);
	return s;
}
export DString MyDLL_testStringReturnStruct() {
	string s = "hello";
	cache.require(s);
	auto dstr = DString(s);
	writefln("[D] pass: %s", dstr);
	return dstr;
}
export void MyDLL_testStringOut(out string str) {
	string s = "hello";
	cache.require(s);
	str = s;
	writefln("[D] pass: %s", str);
}
```



[StructLayout(LayoutKind.Sequential, Size=16, Pack=1)]
public struct DString {
	public ulong length;
	public IntPtr ptr;
	public string str {
		get {
			byte[] b = new byte[length];
			for (int i = 0; i < (int)length; i++) {
				b[i] = Marshal.ReadByte(ptr, i);
			}
			return Encoding.UTF8.GetString(b);
		}
	}
}

[DllImport("mydll.dll")]
private static extern DString MyDLL_testStringReturn();
[DllImport("mydll.dll")]
private static extern DString MyDLL_testStringReturnStruct();
[DllImport("mydll.dll")]
private static extern void MyDLL_testStringOut(out DString dd);

public static void testString() {
	System.Console.WriteLine("Test [RETURN D string]");
	DString d1 = MyDLL_testStringReturn();
	System.Console.WriteLine("ok");
	System.Console.WriteLine("len: {0}", d1.length);
	System.Console.WriteLine("d1: {0}", d1.str);

	System.Console.WriteLine("Test [RETURN D struct]");
	DString d2 = MyDLL_testStringReturnStruct();
	System.Console.WriteLine("ok");
	System.Console.WriteLine("len: {0}", d2.length);
	System.Console.WriteLine("d2: {0}", d2.str);

	System.Console.WriteLine("Test [OUT D string]");
	DString d3;
	MyDLL_testStringOut(out d3);
	System.Console.WriteLine("ok");
	System.Console.WriteLine("len: {0}", d3.length);
	System.Console.WriteLine("d3: {0}", d3.str);
}
```

Output
```
Test [RETURN D string]
[D] pass: hello
ok
len: 0         !! Should be 5!
d1:            !! Should be hello!
Test [RETURN D struct]
[D] pass: DString(5, 7FFA497BCDD0)
ok
len: 5
d2: hello
Test [OUT D string]
[D] pass: hello
ok
len: 5
d3: hello
```
Dec 26 2022
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 12/26/22 22:42, cc wrote:


 Why does returning a
 struct, or sending a D string via an out parameter, work here, but
 simply returning a D string does not?







 static struct DString {
      size_t length;


[...]


 [StructLayout(LayoutKind.Sequential, Size=16, Pack=1)]
 public struct DString {
      public ulong length;


[...]

Apparently that works in your case but D's size_t will be 32 bits on a 


Ali
Dec 27 2022
parent reply cc <cc nevernet.com> writes:
On Tuesday, 27 December 2022 at 17:04:03 UTC, Ali Çehreli wrote:







how does D pass a D slice as a return value under extern(C)?  
Does it just return nothing at all?  Returning a struct that is 
the same size as a D slice and memcpy'd from one works.

Here it is under C instead:

C
```c
#include <stdio.h>
#pragma comment(lib, "mydll.lib")
struct DString {
	unsigned __int64 len;
	char *ptr;
};
extern __declspec(dllimport) struct DString __cdecl 
MyDLL_testStringReturn(void);

int main(int argc, char** argv) {
	printf("[CMain] START\n");
	struct DString d = MyDLL_testStringReturn();
	printf("[CMain] len: %I64u\n", d.len);
	printf("[CMain] str: ");
	for (int i = 0; i < d.len; i++) {
		printf("%c", d.ptr[i]);
	}
	printf("\n");
	printf("[CMain] END\n");
}
```

This doesn't work (D):
```d
export auto MyDLL_testStringReturn() {
	string s = "hello";
	return s;
}
```
```
Segmentation fault
```

But this does (D):
```d
export auto MyDLL_testStringReturn() {
	import core.stdc.string : memcpy;
	string s = "hello";
	static struct N {
		ubyte[string.sizeof] b;
	}
	static assert(N.sizeof == string.sizeof);
	N n;
	memcpy(&n, &s, string.sizeof);
	return n;
}
```
```
[CMain] START
[CMain] len: 5
[CMain] str: hello
[CMain] END
```
Dec 27 2022
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 12/27/22 10:31, cc wrote:





Sure but you are not using extern(C) for your D example. (?)

I find quotes like this: "C defines no ABI. In fact, it bends over 
backwards to avoid defining an ABI."

At least we have this guarantee: "D is designed to fit comfortably with 
a C compiler for the target system.":

   https://dlang.org/spec/interfaceToC.html#structs_and_unions


 This doesn't work (D):
 ```d
 export auto MyDLL_testStringReturn() {
      string s = "hello";
      return s;
 }


But that's not extern(C).

There must be clues on the following page but it doesn't show anything 
special for returning arrays nor it specifies whether an array is the 
same as a struct with 2 members. (We know it is equivalent but it 
doesn't say it is the same as a struct definition.)

   https://dlang.org/spec/abi.html#return_value

Ali
Dec 27 2022
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 12/27/22 11:02, Ali Çehreli wrote:


 whether an array is the same as a struct with 2 members. (We
 know it is equivalent but it doesn't say it is the same as a
 struct definition.)


I've just checked. No, D arrays are not returned as a struct that has 
two members.

Source code and the output of -vasm compilation:

$ tail -n 17 deneme.d
struct S {
     size_t length;
     char * ptr;
}

S returns_S() {
     return S();
}

string returns_string() {
     return "hello world";
}

void main() {
     auto a = returns_S();
     auto b = returns_string();
}

$ dmd -vasm deneme.d
_D6deneme9returns_SFZSQu1S:
0000:   55                       push      RBP
0001:   48 8B EC                 mov       RBP,RSP
0004:   48 83 EC 10              sub       RSP,010h
0008:   48 C7 45 F0 00 00 00 00  mov       qword ptr -010h[RBP],0
0010:   48 C7 45 F8 00 00 00 00  mov       qword ptr -8[RBP],0
0018:   48 8B 55 F8              mov       RDX,-8[RBP]
001c:   48 8B 45 F0              mov       RAX,-010h[RBP]
0020:   C9                       leave
0021:   C3                       ret
_D6deneme14returns_stringFZAya:
0000:   48 8D 15 FC FF FF FF     lea       RDX,[0FFFFFFFCh][RIP]
0007:   B8 0B 00 00 00           mov       EAX,0Bh
000c:   C3                       ret
_Dmain:
0000:   55                       push      RBP
0001:   48 8B EC                 mov       RBP,RSP
0004:   E8 00 00 00 00           call      L0
0009:   E8 00 00 00 00           call      L0
000e:   31 C0                    xor       EAX,EAX
0010:   5D                       pop       RBP
0011:   C3                       ret
main:
0000:   55                       push      RBP
0001:   48 8B EC                 mov       RBP,RSP
0004:   48 8B 15 FC FF FF FF     mov       RDX,[0FFFFFFFCh][RIP]
000b:   E8 00 00 00 00           call      L0
0010:   5D                       pop       RBP
0011:   C3                       ret

A string is passed by registers while a struct object is copied to 
stack. For strings, EAX carries the length because 0Bh is 11, the length 
of "hello world".

As expected, compiling with -O changes how the struct object is 
returned; now by registers like a string is:

_D6deneme9returns_SFZSQu1S:
0000:   31 C0                    xor       EAX,EAX
0002:   31 D2                    xor       EDX,EDX
0004:   C3                       ret

Ali
Dec 27 2022
parent Siarhei Siamashka <siarhei.siamashka gmail.com> writes:
On Wednesday, 28 December 2022 at 00:15:42 UTC, Ali Çehreli wrote:

 A string is passed by registers while a struct object is copied 
 to stack.


It looks like they both are returned in the RDX:RAX pair in all 
cases with and without optimizations. I don't see any difference 
at all.
Dec 27 2022
prev sibling parent cc <cc nevernet.com> writes:
On Tuesday, 27 December 2022 at 19:02:51 UTC, Ali Çehreli wrote:

 Sure but you are not using extern(C) for your D example. (?)
 But that's not extern(C).


I left off the rest of the DLL boilerplate and the `extern(C):` 
above the exported functions was cut off.
Dec 28 2022
prev sibling parent alchemypy <me alchemypy.com> writes:
On Tuesday, 27 December 2022 at 18:31:34 UTC, cc wrote:

 On Tuesday, 27 December 2022 at 17:04:03 UTC, Ali Çehreli wrote:







 However, how does D pass a D slice as a return value under 
 extern(C)?  Does it just return nothing at all?  Returning a 
 struct that is the same size as a D slice and memcpy'd from one 
 works.



I may be wrong (Please ignore this if i am wrong) , This example 

challenge i am facing is) i am not able to call a D function 
which has Struct parameter with String property.

I am trying to see how i can call Print_gph function which has 
Struct parameter with String property.

	struct gph
	{
		string x;
	}

	struct myStruct
	{
		pragma(mangle, "Print_gph")
		void Print_gph(gph g)
		{
			stderr.writeln("here: gph ");
			stderr.writeln(g.x);
		
		}
	}
Dec 27 2022