www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - TLS + LDC + Android (ARM) = FAIL

reply Igor Shirkalin <mathsoft inbox.ru> writes:
We solved the subject with modifying druntime source related with 
tls. Imaging, we have lost a lot of D's features.
As far as I know DMD or GDC are not available for ARM 
architecture. So we need LDC.
A short story: we have big C/C++ project that links D (LDC) code 
for different platforms.

Does new "-betterC" mean we may use parallelism with using 
separate linker?

- IS
Nov 01 2017
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 1 November 2017 at 18:24, Igor Shirkalin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 We solved the subject with modifying druntime source related with tls.
 Imaging, we have lost a lot of D's features.
 As far as I know DMD or GDC are not available for ARM architecture. So we
 need LDC.


GDC supports the same or maybe more platforms than LDC. :-)
Nov 01 2017
next sibling parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw wrote:

 On 1 November 2017 at 18:24, Igor Shirkalin via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 We solved the subject with modifying druntime source related 
 with tls.
 Imaging, we have lost a lot of D's features.
 As far as I know DMD or GDC are not available for ARM 
 architecture. So we
 need LDC.


 GDC supports the same or maybe more platforms than LDC. :-)


Could you please give the reference how to build it in linux or 
windows?
Nov 01 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw wrote:

 GDC supports the same or maybe more platforms than LDC. :-)


Or quite possibly fewer, depending on what one understands 
"platform" and "support" to mean. ;)

What is the state of GDC on Android/ARM – has anyone been using 
it recently?

  — David
Nov 01 2017
parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Nov 2017 17:42:22 +0000
schrieb David Nadlinger <code klickverbot.at>:


 On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw wrote:

 GDC supports the same or maybe more platforms than LDC. :-) =20


=20
 Or quite possibly fewer, depending on what one understands=20
 "platform" and "support" to mean. ;)
=20
 What is the state of GDC on Android/ARM =E2=80=93 has anyone been using=20
 it recently?
=20
   =E2=80=94=C2=A0David
=20


ARM: Fine. Android: probably won't work well. AFAIK we're only
missing emulated TLS / GC integration, so most test will pass but real
apps will crash because of GC memory corruption. I guess I should
finally get back to fixing that problem ;-) OTOH Android doesn't even
support GCC anymore, so I don't really see much benefit in maintaining
GDC Android support.


-- Johannes
Nov 01 2017
next sibling parent reply Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau 
wrote:

 Am Wed, 01 Nov 2017 17:42:22 +0000
 schrieb David Nadlinger <code klickverbot.at>:


 On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw 
 wrote:

 [...]


 
 Or quite possibly fewer, depending on what one understands 
 "platform" and "support" to mean. ;)
 
 What is the state of GDC on Android/ARM – has anyone been 
 using it recently?
 
   — David
 


 ARM: Fine. Android: probably won't work well. AFAIK we're only 
 missing emulated TLS / GC integration, so most test will pass 
 but real apps will crash because of GC memory corruption. I 
 guess I should finally get back to fixing that problem ;-) OTOH 
 Android doesn't even support GCC anymore,



 so I don't really see much benefit in maintaining GDC Android 
 support.
 -- Johannes


That's too bad. I'd do it here for food.

- Igor
Nov 01 2017
parent Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Nov 2017 18:32:37 +0000
schrieb Igor Shirkalin <mathsoft inbox.ru>:


 On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau=20
 wrote:

 Am Wed, 01 Nov 2017 17:42:22 +0000
 schrieb David Nadlinger <code klickverbot.at>:
 =20

 On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw=20
 wrote: =20

 [...] =20


=20
 Or quite possibly fewer, depending on what one understands=20
 "platform" and "support" to mean. ;)
=20
 What is the state of GDC on Android/ARM =E2=80=93 has anyone been=20
 using it recently?
=20
   =E2=80=94=C2=A0David
  =20


 ARM: Fine. Android: probably won't work well. AFAIK we're only=20
 missing emulated TLS / GC integration, so most test will pass=20
 but real apps will crash because of GC memory corruption. I=20
 guess I should finally get back to fixing that problem ;-) OTOH=20
 Android doesn't even support GCC anymore, =20


=20

 so I don't really see much benefit in maintaining GDC Android=20
 support.
 -- Johannes =20


=20
 That's too bad. I'd do it here for food.
=20
 - Igor
=20


I understand that D support for Android is important. I just think that
if google now supports only LLVM for Android, focusing on a LLVM based
compiler such as LDC is a more reasonable way to support Android.


-- Johannes
Nov 01 2017
prev sibling next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 1 November 2017 at 19:28, Johannes Pfau via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am Wed, 01 Nov 2017 17:42:22 +0000
 schrieb David Nadlinger <code klickverbot.at>:


 On Wednesday, 1 November 2017 at 17:30:05 UTC, Iain Buclaw wrote:

 GDC supports the same or maybe more platforms than LDC. :-)


 Or quite possibly fewer, depending on what one understands
 "platform" and "support" to mean. ;)

 What is the state of GDC on Android/ARM – has anyone been using
 it recently?

   — David


 ARM: Fine. Android: probably won't work well. AFAIK we're only
 missing emulated TLS / GC integration, so most test will pass but real
 apps will crash because of GC memory corruption. I guess I should
 finally get back to fixing that problem ;-) OTOH Android doesn't even
 support GCC anymore, so I don't really see much benefit in maintaining
 GDC Android support.


 -- Johannes


What's the thread model on Android?  You could perhaps get away with
--enable-threads=single.
Nov 01 2017
parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 18:33:45 UTC, Iain Buclaw wrote:

 On 1 November 2017 at 19:28, Johannes Pfau via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 Am Wed, 01 Nov 2017 17:42:22 +0000
 schrieb David Nadlinger <code klickverbot.at>:


[...]


 ARM: Fine. Android: probably won't work well. AFAIK we're only 
 missing emulated TLS / GC integration, so most test will pass 
 but real apps will crash because of GC memory corruption. I 
 guess I should finally get back to fixing that problem ;-) 
 OTOH Android doesn't even support GCC anymore, so I don't 
 really see much benefit in maintaining GDC Android support.


 -- Johannes


 What's the thread model on Android?  You could perhaps get away 
 with --enable-threads=single.


I didn't pick this option up. We need multithreading.
Nov 01 2017
prev sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau 
wrote:

 ARM: Fine. Android: probably won't work well. AFAIK we're only 
 missing emulated TLS / GC integration, so most test will pass 
 but real apps will crash because of GC memory corruption. I 
 guess I should finally get back to fixing that problem ;-) OTOH 
 Android doesn't even support GCC anymore, so I don't really see 
 much benefit in maintaining GDC Android support.


I don't see what their deciding to drop gcc has to do with 
whether gdc should support Android.

On Wednesday, 1 November 2017 at 18:56:00 UTC, Igor Shirkalin 
wrote:

 Everything was beautiful for Win/Linux/iOS/Android.
 Some day we had to use LDC (for some obvious reasons). I marked 
 every function with  nogc and pure (it helped me to optimize 
 the code). The problem is TLS. Android doesn't support it. At 
 all. If you understand I talk about -betterC feature.


Android doesn't have native TLS, but __thread-annotated variables 
will work with clang: it uses emulated TLS.

betterC is not going to save you, many parts of the standard 
library won't work with it.

Please open an issue for ldc.  In it, give us as much relevant 
detail as you can.  Are you building the D component as a 
separate shared library or does it just get linked into a single 
binary with your C/C++ code?  Are you making sure to initialize 
the D runtime using rt_init, as described here?

https://wiki.dlang.org/Runtime_internals

Have you followed the three linking rules for emulated TLS that I 
laid out in an earlier post?

Put that info in an issue on the ldc github and we can help you 
out.
Nov 01 2017
parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Nov 2017 19:24:42 +0000
schrieb Joakim <dlang joakim.fea.st>:


 On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau 
 wrote:

 ARM: Fine. Android: probably won't work well. AFAIK we're only 
 missing emulated TLS / GC integration, so most test will pass 
 but real apps will crash because of GC memory corruption. I 
 guess I should finally get back to fixing that problem ;-) OTOH 
 Android doesn't even support GCC anymore, so I don't really see 
 much benefit in maintaining GDC Android support.  


 
 I don't see what their deciding to drop gcc has to do with 
 whether gdc should support Android.


If there's a backend bug in GCC related to Android nobody will take
responsibility as it's not officially supported. All tutorials and
documentation will focus on LLVM based compilers. It's certainly still
interesting to use GDC for Android, but it is more work (especially
considering Android is one of the few systems requiring emutls) for
little benefit, especially if most users are going to use LLVM
anyway.

With the limited time available I think GDC should focus on systems
where GCC is a first class or even the preferred compiler.
X86/MIPS/ARM/PPC/Linux as some distributions such as debian might prefer
a GCC based compiler. Then there are embedded toolchains which
primarily use GCC: MSP and ARM (GCC Arm Embedded project). Also many
console homebrew toolchains exclusively use GCC.

I just don't think we have to support two compilers for any target
with the little resources we have.


-- Johannes
Nov 01 2017
parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 21:50:09 UTC, Johannes Pfau 
wrote:

 Am Wed, 01 Nov 2017 19:24:42 +0000
 schrieb Joakim <dlang joakim.fea.st>:


 On Wednesday, 1 November 2017 at 18:28:12 UTC, Johannes Pfau 
 wrote:

 ARM: Fine. Android: probably won't work well. AFAIK we're 
 only missing emulated TLS / GC integration, so most test 
 will pass but real apps will crash because of GC memory 
 corruption. I guess I should finally get back to fixing that 
 problem ;-) OTOH Android doesn't even support GCC anymore, 
 so I don't really see much benefit in maintaining GDC 
 Android support.


 
 I don't see what their deciding to drop gcc has to do with 
 whether gdc should support Android.


 If there's a backend bug in GCC related to Android nobody will 
 take responsibility as it's not officially supported. All 
 tutorials and documentation will focus on LLVM based compilers. 
 It's certainly still interesting to use GDC for Android, but it 
 is more work (especially considering Android is one of the few 
 systems requiring emutls) for little benefit, especially if 
 most users are going to use LLVM anyway.

 With the limited time available I think GDC should focus on 
 systems where GCC is a first class or even the preferred 
 compiler. X86/MIPS/ARM/PPC/Linux as some distributions such as 
 debian might prefer a GCC based compiler. Then there are 
 embedded toolchains which primarily use GCC: MSP and ARM (GCC 
 Arm Embedded project). Also many console homebrew toolchains 
 exclusively use GCC.

 I just don't think we have to support two compilers for any 
 target with the little resources we have.


Here's the thing though: Android is basically just linux/ARM 
without native TLS and replacing glibc with the bionic C runtime. 
  If you're going to support linux/ARM with glibc, it's not going 
to take much to support Android too, particularly since I 
upstreamed all my Bionic declarations and support into stock 
druntime.

Far be it from me to tell you what to work on, and I realize 
getting emulated TLS working with the garbage-collector is a 
pain, but it won't take much more than emulated TLS for gdc to 
support Android.  If you grep the llvm codebase for Android, you 
find almost nothing, basically just emulated TLS.

I agree that we don't need two D compilers for every platform, 
but it'd be nice to have for the most widely deployed OS platform 
on the planet. ;)
Nov 01 2017
prev sibling next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
wrote:

 We solved the subject with modifying druntime source related 
 with tls. Imaging, we have lost a lot of D's features.


You'd have been better off opening an issue for ldc:

https://github.com/ldc-developers/ldc/issues

TLS should work fine, though it's emulated, as Android doesn't 
support native TLS.  You have to be careful how you link the 
emulated TLS, and you have to include a main function even for a 
shared library:

https://wiki.dlang.org/Build_LDC_for_Android#Directions_for_future_work

You'd be better off talking to the ldc devs- this is the first 
I'm hearing about this- rather than going in and making changes 
to druntime.


 As far as I know DMD or GDC are not available for ARM 
 architecture. So we need LDC.
 A short story: we have big C/C++ project that links D (LDC) 
 code for different platforms.

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?


You'll need to expand on this: you want to use std.parallelism, 
which isn't working for you now, or something else to do with the 
linker is going wrong?
Nov 01 2017
parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 17:44:11 UTC, Joakim wrote:

 On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
 wrote:

 We solved the subject with modifying druntime source related 
 with tls. Imaging, we have lost a lot of D's features.


 You'd have been better off opening an issue for ldc:

 https://github.com/ldc-developers/ldc/issues

 TLS should work fine, though it's emulated, as Android doesn't 
 support native TLS.  You have to be careful how you link the 
 emulated TLS, and you have to include a main function even for 
 a shared library:

 https://wiki.dlang.org/Build_LDC_for_Android#Directions_for_future_work

 You'd be better off talking to the ldc devs- this is the first 
 I'm hearing about this- rather than going in and making changes 
 to druntime.


 As far as I know DMD or GDC are not available for ARM 
 architecture. So we need LDC.
 A short story: we have big C/C++ project that links D (LDC) 
 code for different platforms.

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?


 You'll need to expand on this: you want to use std.parallelism, 
 which isn't working for you now, or something else to do with


We use external D libraries in C project. Using parallelisms 
means we have to initialize druntime. But tls stops it (therefore 
we had changed it). Sure, we have tried to build pure D part and 
it didnt work.

 the linker is going wrong?


of course the linker was working properly.
Nov 01 2017
prev sibling next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
wrote:

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?


`-betterC` does not add any emulation of missing platform 
features — on the contrary, it *removes* language runtime 
functionality! Thus, if TLS doesn't work for you (IIRC, emulated 
TLS should work on Android following Joakim's work!), adding 
`-betterC` won't improve the situation.

Could you please open an ticket on the LDC GitHub tracker with 
details on the issue?

  — David
Nov 01 2017
parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 17:46:29 UTC, David Nadlinger 
wrote:

 On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
 wrote:

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?


 `-betterC` does not add any emulation of missing platform 
 features — on the contrary, it *removes* language runtime 
 functionality! Thus, if TLS doesn't work for you (IIRC, 
 emulated TLS should work on Android following Joakim's work!), 
 adding `-betterC` won't improve the situation.


I undrastand that.

 Could you please open an ticket on the LDC GitHub tracker with 
 details on the issue?


I'd try with help.


  — David
Nov 01 2017
prev sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
wrote:

 We solved the subject with modifying druntime source related 
 with tls. Imaging, we have lost a lot of D's features.
 As far as I know DMD or GDC are not available for ARM 
 architecture. So we need LDC.
 A short story: we have big C/C++ project that links D (LDC) 
 code for different platforms.

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?

 - IS


If you're having problems with the emulated TLS I put together 
for Android, it is most likely because I didn't document well 
what needs to be done when linking for Android.  Specifically, 
there are three rules that _must_ be followed:

1. You must use the ld.bfd linker, ld.gold won't do.
2. You must have a D main function, even for a shared library 
(which can be put next to android_main, if you're using the 
default Android wrapper from my D android library).
3. The ELF object with the D main function must be passed to the 
linker first.

If you look at my examples on the wiki, you'll see that they all 
follow these rules:

https://wiki.dlang.org/Build_D_for_Android

I should have called these rules out separately though, like I'm 
doing here, a documentation oversight.
Nov 01 2017
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 01 Nov 2017 18:06:29 +0000
schrieb Joakim <dlang joakim.fea.st>:


 On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
 wrote:

 We solved the subject with modifying druntime source related 
 with tls. Imaging, we have lost a lot of D's features.
 As far as I know DMD or GDC are not available for ARM 
 architecture. So we need LDC.
 A short story: we have big C/C++ project that links D (LDC) 
 code for different platforms.

 Does new "-betterC" mean we may use parallelism with using 
 separate linker?

 - IS  


 
 If you're having problems with the emulated TLS I put together 
 for Android, it is most likely because I didn't document well 
 what needs to be done when linking for Android.  Specifically, 
 there are three rules that _must_ be followed:
 
 1. You must use the ld.bfd linker, ld.gold won't do.
 2. You must have a D main function, even for a shared library 
 (which can be put next to android_main, if you're using the 
 default Android wrapper from my D android library).
 3. The ELF object with the D main function must be passed to the 
 linker first.
 
 If you look at my examples on the wiki, you'll see that they all 
 follow these rules:
 
 https://wiki.dlang.org/Build_D_for_Android
 
 I should have called these rules out separately though, like I'm 
 doing here, a documentation oversight.


Also when mixing D and C code, you can't access extern TLS variables
across the language boundary. Maybe OP tries to do that as he mixes D/C
code?

-- Johannes
Nov 01 2017
parent Igor Shirkalin <mathsoft inbox.ru> writes:
On Wednesday, 1 November 2017 at 18:30:28 UTC, Johannes Pfau 
wrote:

 Am Wed, 01 Nov 2017 18:06:29 +0000
 schrieb Joakim <dlang joakim.fea.st>:


 On Wednesday, 1 November 2017 at 17:24:32 UTC, Igor Shirkalin 
 wrote:

 [...]


 
 If you're having problems with the emulated TLS I put together 
 for Android, it is most likely because I didn't document well 
 what needs to be done when linking for Android.  Specifically, 
 there are three rules that _must_ be followed:
 
 1. You must use the ld.bfd linker, ld.gold won't do.
 2. You must have a D main function, even for a shared library
 (which can be put next to android_main, if you're using the
 default Android wrapper from my D android library).
 3. The ELF object with the D main function must be passed to 
 the
 linker first.
 
 If you look at my examples on the wiki, you'll see that they 
 all follow these rules:
 
 https://wiki.dlang.org/Build_D_for_Android
 
 I should have called these rules out separately though, like 
 I'm doing here, a documentation oversight.


 Also when mixing D and C code, you can't access extern TLS 
 variables across the language boundary. Maybe OP tries to do 
 that as he mixes D/C code?

 -- Johannes


Everything was beautiful for Win/Linux/iOS/Android.
Some day we had to use LDC (for some obvious reasons). I marked 
every function with  nogc and pure (it helped me to optimize the 
code). The problem is TLS. Android doesn't support it. At all. If 
you understand I talk about -betterC feature.

- Igor
Nov 01 2017
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 1 November 2017 at 18:06:29 UTC, Joakim wrote:

 If you're having problems with the emulated TLS I put together 
 for Android, it is most likely because I didn't document well 
 what needs to be done when linking for Android.  Specifically, 
 there are three rules that _must_ be followed:

 1. You must use the ld.bfd linker, ld.gold won't do.
 2. You must have a D main function, even for a shared library 
 (which can be put next to android_main, if you're using the 
 default Android wrapper from my D android library).
 3. The ELF object with the D main function must be passed to 
 the linker first.

 If you look at my examples on the wiki, you'll see that they 
 all follow these rules:

 https://wiki.dlang.org/Build_D_for_Android

 I should have called these rules out separately though, like 
 I'm doing here, a documentation oversight.


I've added a new section to the Android wiki page with this info, 
and some more relevant details:

https://wiki.dlang.org/Build_D_for_Android#Changes_for_Android
Nov 01 2017