www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ldc - WebAssembly + Custom Runtime + Associative Arrays

reply Remi <remigillig gmail.com> writes:
Hi all,

I've been trying to compile a game project to WebAssembly for a 
while. It had been going quite well but I'm hitting an issue with 
associative arrays now and I don't know if it's a compiler bug or 
something on my side.

I uploaded some code with a broken example here: 
https://github.com/speps/dwasmaa

If you clone it then run a HTTP server and navigate to index.htm, 
it will output the wrong message as described in the README.

If you compile+run it with `dub --arch=x86` it displays the 
correct message (sorry that arch only works in Windows for now as 
I'm not using the runtime and only hooked up Win32 API console 
output).

Anyway, where am I going wrong? Thanks.

- Remi Gillig.
Nov 10 2020
next sibling parent reply Ferhat =?UTF-8?B?S3VydHVsbXXFnw==?= <aferust gmail.com> writes:
On Tuesday, 10 November 2020 at 18:24:03 UTC, Remi wrote:

 Hi all,

 I've been trying to compile a game project to WebAssembly for a 
 while. It had been going quite well but I'm hitting an issue 
 with associative arrays now and I don't know if it's a compiler 
 bug or something on my side.

 I uploaded some code with a broken example here: 
 https://github.com/speps/dwasmaa

 If you clone it then run a HTTP server and navigate to 
 index.htm, it will output the wrong message as described in the 
 README.

 If you compile+run it with `dub --arch=x86` it displays the 
 correct message (sorry that arch only works in Windows for now 
 as I'm not using the runtime and only hooked up Win32 API 
 console output).

 Anyway, where am I going wrong? Thanks.

 - Remi Gillig.


AA needs druntime. Adam's repo does not include a custom AA 
implementation.
Nov 10 2020
parent Ferhat =?UTF-8?B?S3VydHVsbXXFnw==?= <aferust gmail.com> writes:
On Tuesday, 10 November 2020 at 19:20:42 UTC, Ferhat Kurtulmuş 
wrote:

 On Tuesday, 10 November 2020 at 18:24:03 UTC, Remi wrote:

 [...]


 AA needs druntime. Adam's repo does not include a custom AA 
 implementation.


Uhh you did it in your repo. Sorry.
Nov 10 2020
prev sibling next sibling parent Johan <j j.nl> writes:
On Tuesday, 10 November 2020 at 18:24:03 UTC, Remi wrote:

 Hi all,

 I've been trying to compile a game project to WebAssembly for a 
 while. It had been going quite well but I'm hitting an issue 
 with associative arrays now and I don't know if it's a compiler 
 bug or something on my side.


I know almost nothing about webassembly, but I think 0 (`null`) 
is a valid memory address. Could that be the reason for the 
troubles? (some object allocated on address 0?)

-Johan
Nov 10 2020
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 10 November 2020 at 18:24:03 UTC, Remi wrote:

 Anyway, where am I going wrong? Thanks.


Looks like you fixed the array append function but the one in my 
repo was broken af until... well right now, I just pushed. The 
way that function works is bizarre.

But I think you did it right here... just like... that's where 
I'd focus attention first to confirm it actually works before 
digging much deeper.
Nov 10 2020
prev sibling next sibling parent reply Ferhat =?UTF-8?B?S3VydHVsbXXFnw==?= <aferust gmail.com> writes:
On Tuesday, 10 November 2020 at 18:24:03 UTC, Remi wrote:

 Hi all,

 I've been trying to compile a game project to WebAssembly for a 
 while. It had been going quite well but I'm hitting an issue 
 with associative arrays now and I don't know if it's a compiler 
 bug or something on my side.

 I uploaded some code with a broken example here: 
 https://github.com/speps/dwasmaa

 If you clone it then run a HTTP server and navigate to 
 index.htm, it will output the wrong message as described in the 
 README.

 If you compile+run it with `dub --arch=x86` it displays the 
 correct message (sorry that arch only works in Windows for now 
 as I'm not using the runtime and only hooked up Win32 API 
 console output).

 Anyway, where am I going wrong? Thanks.

 - Remi Gillig.


It is weird, but if you pass pointer aa as ref it works.

extern (C) void* _aaGetY(ref AA* aa, const 
TypeInfo_AssociativeArray aati, in size_t valuesize, in void* 
pkey)
Nov 10 2020
parent reply Remi <remigillig gmail.com> writes:
On Tuesday, 10 November 2020 at 21:23:07 UTC, Ferhat Kurtulmuş 
wrote:

 On Tuesday, 10 November 2020 at 18:24:03 UTC, Remi wrote:

 [...]


 It is weird, but if you pass pointer aa as ref it works.

 extern (C) void* _aaGetY(ref AA* aa, const 
 TypeInfo_AssociativeArray aati, in size_t valuesize, in void* 
 pkey)


Wow thanks! Can't believe I didn't think of that. Seems to work 
indeed.
Nov 10 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 10 November 2020 at 22:36:40 UTC, Remi wrote:

 Wow thanks! Can't believe I didn't think of that. Seems to work 
 indeed.


I don't understand why that would work though..

you don't change aa itself, just the aa.impl which should go 
through the pointer already.....
Nov 10 2020
parent reply Remi <remigillig gmail.com> writes:
On Tuesday, 10 November 2020 at 22:38:30 UTC, Adam D. Ruppe wrote:

 On Tuesday, 10 November 2020 at 22:36:40 UTC, Remi wrote:

 Wow thanks! Can't believe I didn't think of that. Seems to 
 work indeed.


 I don't understand why that would work though..

 you don't change aa itself, just the aa.impl which should go 
 through the pointer already.....


Well, it seems my troubles aren't finished though. I'm having 
another _aaGetY crash to figure out by using the WASM debugger in 
Firefox...
Nov 10 2020
parent reply Remi <remigillig gmail.com> writes:
On Tuesday, 10 November 2020 at 22:53:59 UTC, Remi wrote:

 On Tuesday, 10 November 2020 at 22:38:30 UTC, Adam D. Ruppe 
 wrote:

 On Tuesday, 10 November 2020 at 22:36:40 UTC, Remi wrote:

 Wow thanks! Can't believe I didn't think of that. Seems to 
 work indeed.


 I don't understand why that would work though..

 you don't change aa itself, just the aa.impl which should go 
 through the pointer already.....


 Well, it seems my troubles aren't finished though. I'm having 
 another _aaGetY crash to figure out by using the WASM debugger 
 in Firefox...


Also that fixes makes it crash in the x86 build now. So yeah Adam 
you might be right, it doesn't really fix it in the right way. 
I'm still not sure why it's doing that.
Nov 10 2020
parent reply Ferhat =?UTF-8?B?S3VydHVsbXXFnw==?= <aferust gmail.com> writes:
On Tuesday, 10 November 2020 at 22:57:27 UTC, Remi wrote:

 On Tuesday, 10 November 2020 at 22:53:59 UTC, Remi wrote:

 On Tuesday, 10 November 2020 at 22:38:30 UTC, Adam D. Ruppe 
 wrote:

 On Tuesday, 10 November 2020 at 22:36:40 UTC, Remi wrote:

 Wow thanks! Can't believe I didn't think of that. Seems to 
 work indeed.


 I don't understand why that would work though..

 you don't change aa itself, just the aa.impl which should go 
 through the pointer already.....


 Well, it seems my troubles aren't finished though. I'm having 
 another _aaGetY crash to figure out by using the WASM debugger 
 in Firefox...


 Also that fixes makes it crash in the x86 build now. So yeah 
 Adam you might be right, it doesn't really fix it in the right 
 way. I'm still not sure why it's doing that.


As kinke mentioned, your AA does not implement everything needed. 
You should port the entire thing in 
druntime/blob/master/src/rt/aaA.d. Some time ago, I created a 
malloc-free-based port of it. It does not provide required 
symbols to compiler as you did there though.

https://github.com/aferust/bcaa/blob/master/source/bcaa.d
Nov 10 2020
parent Remi <remigillig gmail.com> writes:
On Wednesday, 11 November 2020 at 05:18:25 UTC, Ferhat Kurtulmuş 
wrote:

 On Tuesday, 10 November 2020 at 22:57:27 UTC, Remi wrote:
 [...]

 As kinke mentioned, your AA does not implement everything 
 needed. You should port the entire thing in 
 druntime/blob/master/src/rt/aaA.d. Some time ago, I created a 
 malloc-free-based port of it. It does not provide required 
 symbols to compiler as you did there though.

 https://github.com/aferust/bcaa/blob/master/source/bcaa.d


Thanks I'll probably have a look if I find my implementation too 
slow, which will probably be the case :)
Nov 11 2020
prev sibling parent reply kinke <noone nowhere.com> writes:
On Tuesday, 10 November 2020 at 18:24:03 UTC, Remi wrote:

 Anyway, where am I going wrong? Thanks.


[I've just glanced at the code, not run and debugged anything.] I 
don't see where you actually grow the heap (wasm_growMemory() 
never used AFAICT), certainly not in your malloc(), which 
initializes `nextFree` with &__heap_base and then bumps that 
pointer for the next allocation (likely breaking some alignment 
btw - rounding up to the next multiple of 8 or 16 should be safe).

Note that you shouldn't have to get the size of and grow the 
linear memory in JS, there are LLVM intrinsics you can use (just 
copy the declarations): 
https://github.com/ldc-developers/druntime/blob/412467a452e2d12f561a2eace933dd44014af3c6/src/ldc/intrinsics.di#L678-L697
Nov 10 2020
parent reply kinke <noone nowhere.com> writes:
Ouch, it's compiler bug - the compiler's hardcoded prototype for 
_aaInX() uses a `void*` as first param, not a `AA` struct 
wrapping the impl pointer. So this assumes the ABI/calling 
convention treats both types identically, which isn't currently 
the case for LDC's wasm ABI.

So using `void* _aaInX(AAImpl* aaimpl, ...)` works around this.
Nov 10 2020
parent reply Remi <remigillig gmail.com> writes:
On Wednesday, 11 November 2020 at 02:05:25 UTC, kinke wrote:

 Ouch, it's compiler bug - the compiler's hardcoded prototype 
 for _aaInX() uses a `void*` as first param, not a `AA` struct 
 wrapping the impl pointer. So this assumes the ABI/calling 
 convention treats both types identically, which isn't currently 
 the case for LDC's wasm ABI.

 So using `void* _aaInX(AAImpl* aaimpl, ...)` works around this.


That's kind of what I saw, AA* worked in x86 but AA** worked in 
WASM and vice versa. I did try to find where the compiler 
actually generated the call to _aaInX but couldn't find it by a 
simple search. Thanks for the help, it seems to work in both 
architectures now :)
Nov 11 2020
parent Ferhat =?UTF-8?B?S3VydHVsbXXFnw==?= <aferust gmail.com> writes:
On Wednesday, 11 November 2020 at 08:30:09 UTC, Remi wrote:

 On Wednesday, 11 November 2020 at 02:05:25 UTC, kinke wrote:

 Ouch, it's compiler bug - the compiler's hardcoded prototype 
 for _aaInX() uses a `void*` as first param, not a `AA` struct 
 wrapping the impl pointer. So this assumes the ABI/calling 
 convention treats both types identically, which isn't 
 currently the case for LDC's wasm ABI.

 So using `void* _aaInX(AAImpl* aaimpl, ...)` works around this.


 That's kind of what I saw, AA* worked in x86 but AA** worked in 
 WASM and vice versa. I did try to find where the compiler 
 actually generated the call to _aaInX but couldn't find it by a 
 simple search. Thanks for the help, it seems to work in both 
 architectures now :)


Please contribute to Adam's WebAssembly when your druntime works 
is complete, or publish your own library so that others and I can 
use it :)
Nov 11 2020