www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - JTransc + Android/iOS status support

reply soywiz <soywiz gmail.com> writes:
I am working on a software that allows to transform JVM bytecode 
into something else. In last version I have done a D target: 
https://github.com/jtransc/jtransc
D target already passes my test suite and works just fine (next 
version will come with some optimizations regarding to code 
generation that will improve generation).
At this point I'm successfully generating windows/linux and mac 
native executables from Java/Kotlin code.
It would be awesome to be able to support Android and iOS. I have 
seen that there was Android/iOS alpha support in the past. What 
is the current status of this?
Nov 28 2016
next sibling parent Chris <wendlec tcd.ie> writes:
On Monday, 28 November 2016 at 14:28:54 UTC, soywiz wrote:

 I am working on a software that allows to transform JVM 
 bytecode into something else. In last version I have done a D 
 target: https://github.com/jtransc/jtransc
 D target already passes my test suite and works just fine (next 
 version will come with some optimizations regarding to code 
 generation that will improve generation).
 At this point I'm successfully generating windows/linux and mac 
 native executables from Java/Kotlin code.
 It would be awesome to be able to support Android and iOS. I 
 have seen that there was Android/iOS alpha support in the past. 
 What is the current status of this?


Here's a thread that deals with it:

http://forum.dlang.org/thread/fglyajphcqvwfyuklohr forum.dlang.org?page=1
Nov 28 2016
prev sibling next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Monday, 28 November 2016 at 14:28:54 UTC, soywiz wrote:

 I am working on a software that allows to transform JVM 
 bytecode into something else. In last version I have done a D 
 target: https://github.com/jtransc/jtransc
 D target already passes my test suite and works just fine (next 
 version will come with some optimizations regarding to code 
 generation that will improve generation).
 At this point I'm successfully generating windows/linux and mac 
 native executables from Java/Kotlin code.


Interesting, might provide a better bridge for Java compatibility 
on Android.


 It would be awesome to be able to support Android and iOS. I 
 have seen that there was Android/iOS alpha support in the past. 
 What is the current status of this?


Android support is in beta, latest cross-compiler and native 
builds here:

https://github.com/joakim-noah/android/releases

I've been seeing random crashes for apk builds recently, trying 
to track down why.  I've ported the Teapot OpenGL ES 2.0/C++ 
sample app from the NDK, but it's got some bugs, debugging that 
next.  Once both issues are fixed, I'll announce the beta in a 
new thread.

As for iOS, Dan has been busy with other stuff, may get back to 
it sometime this winter.  All mobile builds are linked from the 
main download page, under ldc:

http://dlang.org/download
Nov 28 2016
parent reply soywiz <soywiz gmail.com> writes:
On Monday, 28 November 2016 at 17:53:30 UTC, Joakim wrote:

 On Monday, 28 November 2016 at 14:28:54 UTC, soywiz wrote:

 I am working on a software that allows to transform JVM 
 bytecode into something else. In last version I have done a D 
 target: https://github.com/jtransc/jtransc
 D target already passes my test suite and works just fine 
 (next version will come with some optimizations regarding to 
 code generation that will improve generation).
 At this point I'm successfully generating windows/linux and 
 mac native executables from Java/Kotlin code.


 Interesting, might provide a better bridge for Java 
 compatibility on Android.


 It would be awesome to be able to support Android and iOS. I 
 have seen that there was Android/iOS alpha support in the 
 past. What is the current status of this?


 Android support is in beta, latest cross-compiler and native 
 builds here:

 https://github.com/joakim-noah/android/releases

 I've been seeing random crashes for apk builds recently, trying 
 to track down why.  I've ported the Teapot OpenGL ES 2.0/C++ 
 sample app from the NDK, but it's got some bugs, debugging that 
 next.  Once both issues are fixed, I'll announce the beta in a 
 new thread.

 As for iOS, Dan has been busy with other stuff, may get back to 
 it sometime this winter.  All mobile builds are linked from the 
 main download page, under ldc:

 http://dlang.org/download


Thanks you both for the info!

I have seen those links already. I was a bit concerned about the 
support, since it is outside the main development. It is planned 
to be merged in the main projects? Or those were just 
experiments? It would be great to have full support from the 
community that seems to be pretty active lately at github so it 
doesn't end being outdated.

When considering a new target for jtransc i was searching for a 
language like c++ that allowed to generate native executables, 
with gc already included, with safety in mind, and full oop 
included while allowing to go low level when required. And that 
was exactly D. So I can mix both of the two worlds and mix code 
as required (performant parts in D, and higher level/productivity 
in kotlin).

In the case D targets every major popular platforms (including 
mobile) it would be great, since i can generate javascript for 
the web already and d for desktop+mobile covering all my current 
needs. Instead of having to implement a gc for c++. Also D has 
faster compilation times than it which is great.

I will try both projects asap. If it works I will try to port 
libgdx to jtransc+d+android/ios.

Even when in Android I can run Java directly. I think there is 
still a point for it. Android doesn't support java8 directly and 
it can be running either dalvik or art. Also it is easily 
decompilable. So a native with unknown optimizations depending on 
the versio  AOT could make performance much more predictable 
among versions, obfuscate it, allow to use stuff like simd, or 
support higher java versions.

And for iOS it just supports an AOT approach except for 
javascript. So it is great too.

Thank you for the great work on those forks!
Nov 28 2016
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Tuesday, 29 November 2016 at 01:09:19 UTC, soywiz wrote:

 On Monday, 28 November 2016 at 17:53:30 UTC, Joakim wrote:

 On Monday, 28 November 2016 at 14:28:54 UTC, soywiz wrote:

 I am working on a software that allows to transform JVM 
 bytecode into something else. In last version I have done a D 
 target: https://github.com/jtransc/jtransc
 D target already passes my test suite and works just fine 
 (next version will come with some optimizations regarding to 
 code generation that will improve generation).
 At this point I'm successfully generating windows/linux and 
 mac native executables from Java/Kotlin code.


 Interesting, might provide a better bridge for Java 
 compatibility on Android.


 It would be awesome to be able to support Android and iOS. I 
 have seen that there was Android/iOS alpha support in the 
 past. What is the current status of this?


 Android support is in beta, latest cross-compiler and native 
 builds here:

 https://github.com/joakim-noah/android/releases

 I've been seeing random crashes for apk builds recently, 
 trying to track down why.  I've ported the Teapot OpenGL ES 
 2.0/C++ sample app from the NDK, but it's got some bugs, 
 debugging that next.  Once both issues are fixed, I'll 
 announce the beta in a new thread.

 As for iOS, Dan has been busy with other stuff, may get back 
 to it sometime this winter.  All mobile builds are linked from 
 the main download page, under ldc:

 http://dlang.org/download


 Thanks you both for the info!

 I have seen those links already. I was a bit concerned about 
 the support, since it is outside the main development. It is 
 planned to be merged in the main projects? Or those were just 
 experiments? It would be great to have full support from the 
 community that seems to be pretty active lately at github so it 
 doesn't end being outdated.


Dan, who is behind the iOS/watchOS/tvOS port, and I, Android 
port, are both committers to the ldc project, which has an ARM 
backend unlike dmd, and most of our mobile ports have been merged 
upstream:

https://github.com/ldc-developers/ldc/commits?author=smolt
https://github.com/ldc-developers/ldc/commits?author=joakim-noah

There are a few remaining issues that need to be polished up 
before the ports are fully merged, but the goal is for all mobile 
development to be in the main ldc repo, with official ldc builds 
for mobile released eventually.

Of course, how fast that happens also depend on how many users 
and other ldc developers get involved in the mobile effort, 
something we cannot control.


 When considering a new target for jtransc i was searching for a 
 language like c++ that allowed to generate native executables, 
 with gc already included, with safety in mind, and full oop 
 included while allowing to go low level when required. And that 
 was exactly D. So I can mix both of the two worlds and mix code 
 as required (performant parts in D, and higher 
 level/productivity in kotlin).

 In the case D targets every major popular platforms (including 
 mobile) it would be great, since i can generate javascript for 
 the web already and d for desktop+mobile covering all my 
 current needs. Instead of having to implement a gc for c++. 
 Also D has faster compilation times than it which is great.


You're in the right place. :)


 I will try both projects asap. If it works I will try to port 
 libgdx to jtransc+d+android/ios.


Let me know if you have any problems.  I think the random apk 
issues I'm seeing could be a regression or new incompatibility in 
the ld.bfd linker, but I need to track it down.


 Even when in Android I can run Java directly. I think there is 
 still a point for it. Android doesn't support java8 directly 
 and it can be running either dalvik or art. Also it is easily 
 decompilable. So a native with unknown optimizations depending 
 on the versio  AOT could make performance much more predictable 
 among versions, obfuscate it, allow to use stuff like simd, or 
 support higher java versions.


Yes, ART is AoT-compiled also, so Android has also moved there 
since 5.0.
Nov 28 2016
parent reply soywiz <soywiz gmail.com> writes:
 Jacob:

You can just download this:
https://github.com/jtransc/jtransc-examples/tree/master/hello-world
execute `./gradlew runD` or `./gradlew distD` and check the 
`build/jtransc-d/program.d` file (about 500kb program.d file with 
treeshaking enabled).

Right now it generates classes like `class java_lang_String`, and 
method names like `toString__Ljava_lang_String_` (that's because 
at the JVM level you can have name overloading with all the same 
argument types but a different return type, which is not 
supported by D). I could get rid of the names in most cases, but 
not optimized yet.
Also it uses gotos, except for functions with exception handlers 
where I'm using while+switch to be able to goto try blocks easily.
Also parameter and local variable names are lost and there are 
way too many casts, which makes performance be pretty bad right 
now.
My objective was to create a working target right now. And later 
optimize.
JTransc is divided into two parts: `JVM byte code -> AST` and 
`AST -> Other targets`. So when optimizing the first one, all 
other targets will benefit.

This will happen before jtransc 1.0 (right now I'm on 0.5.0):
I have to create a SSA form to remove casts. And I can use debug 
information to restore local and parameter names. With relooper I 
can convert those gotos into proper structures, though for C++ 
and D targets it is not so important because they have goto and 
performance is good enough.

You can convert between `java.lang.String` and D's `wstring` with 
`N.str` and `N.istr`.


---------

 Joakim:

Regarding to D. Yep I think I'm in the right place :) I started 
using D1 something almost 10 years ago and ended doing this: 
https://github.com/soywiz/pspemu
My main concerns by then were: lack of IDE support for major 
refactorings and productivity + compilation times (because I was 
doing a lot of metaprogramming that affected every compilation). 
But I liked D a lot and knew about its strong points.

programming in Kotlin, which is the most productive language I 
have ever used (because of the sugar + the awesome IDE support) :)

-

Yeah, I know that ART is an AOT. But Android is so fragmented 
that there are still a lot of people with android 4.x using 
Dalvik. Also not sure if ART supports simd optimizations. So 
probably you cannot leverage that without going native.

It is great to know that some effort has already been merged. 
Keep up the good work!

-

I will try it expecting random crashes and will stay tuned for 
updates. But as long as not crashes always, I can still validate 
it. Thanks!
Nov 29 2016
parent reply Joakim <dlang joakim.fea.st> writes:
On Tuesday, 29 November 2016 at 08:49:22 UTC, soywiz wrote:

  Joakim:

 Regarding to D. Yep I think I'm in the right place :) I started 
 using D1 something almost 10 years ago and ended doing this: 
 https://github.com/soywiz/pspemu
 My main concerns by then were: lack of IDE support for major 
 refactorings and productivity + compilation times (because I 
 was doing a lot of metaprogramming that affected every 
 compilation). But I liked D a lot and knew about its strong 
 points.

 mostly programming in Kotlin, which is the most productive 
 language I have ever used (because of the sugar + the awesome 
 IDE support) :)


Oh, nice to see an old user coming back. :)


 Yeah, I know that ART is an AOT. But Android is so fragmented 
 that there are still a lot of people with android 4.x using 
 Dalvik. Also not sure if ART supports simd optimizations. So 
 probably you cannot leverage that without going native.

 It is great to know that some effort has already been merged. 
 Keep up the good work!


Most of my Android work has been merged, especially in 
druntime/phobos upstream.  Last remaining portions are getting 
ldc to use llvm's emulated TLS by registering that data properly 
with the GC and finishing up Kai and Kinke's work to get ldc to 
cross-compile reals properly 
(https://github.com/ldc-developers/ldc/pull/1317).  Once those 
two are done, it should be possible to provide official Android 
builds from ldc master, both native and cross-compiling.


 I will try it expecting random crashes and will stay tuned for 
 updates. But as long as not crashes always, I can still 
 validate it. Thanks!


No crashes if running on the command-line, which is easy to do 
with the Termux app on any Android device:

https://play.google.com/store/apps/details?id=com.termux&hl=en

I'm seeing random crashes of apk builds, ie actual Android GUI 
apps, even with source that hasn't changed at all and used to 
work fine before.  That means there's been a regression somewhere 
in the toolchain, got to figure out where.
Nov 29 2016
parent reply soywiz <soywiz gmail.com> writes:
On Tuesday, 29 November 2016 at 09:09:40 UTC, Jacob Carlborg 
wrote:

 On 2016-11-29 02:09, soywiz wrote:


 And for iOS it just supports an AOT approach except for 
 javascript. So
 it is great too.


 BTW, it's possible to interface Objective-C from D [1].

 [1] http://dlang.org/spec/objc_interface.html


 Jacob
Cool. So I can expect it to work on iOS Obj-C too right? Someone 
tried a wrapper using RAII to emulate partially (except for weak 
references) ARC?

On Tuesday, 29 November 2016 at 09:53:21 UTC, Joakim wrote:

 On Tuesday, 29 November 2016 at 08:49:22 UTC, soywiz wrote:

  Joakim:

 Regarding to D. Yep I think I'm in the right place :) I 
 started using D1 something almost 10 years ago and ended doing 
 this: https://github.com/soywiz/pspemu
 My main concerns by then were: lack of IDE support for major 
 refactorings and productivity + compilation times (because I 
 was doing a lot of metaprogramming that affected every 
 compilation). But I liked D a lot and knew about its strong 
 points.

 mostly programming in Kotlin, which is the most productive 
 language I have ever used (because of the sugar + the awesome 
 IDE support) :)


 Oh, nice to see an old user coming back. :)


 Yeah, I know that ART is an AOT. But Android is so fragmented 
 that there are still a lot of people with android 4.x using 
 Dalvik. Also not sure if ART supports simd optimizations. So 
 probably you cannot leverage that without going native.

 It is great to know that some effort has already been merged. 
 Keep up the good work!


 Most of my Android work has been merged, especially in 
 druntime/phobos upstream.  Last remaining portions are getting 
 ldc to use llvm's emulated TLS by registering that data 
 properly with the GC and finishing up Kai and Kinke's work to 
 get ldc to cross-compile reals properly 
 (https://github.com/ldc-developers/ldc/pull/1317).  Once those 
 two are done, it should be possible to provide official Android 
 builds from ldc master, both native and cross-compiling.


 I will try it expecting random crashes and will stay tuned for 
 updates. But as long as not crashes always, I can still 
 validate it. Thanks!


 No crashes if running on the command-line, which is easy to do 
 with the Termux app on any Android device:

 https://play.google.com/store/apps/details?id=com.termux&hl=en

 I'm seeing random crashes of apk builds, ie actual Android GUI 
 apps, even with source that hasn't changed at all and used to 
 work fine before.  That means there's been a regression 
 somewhere in the toolchain, got to figure out where.


Amazing! Right now all the generated code uses __gshared. In Java 
you have to use 
https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html to get
TLS. And I didn't implemented it yet for multithreading. Because the initial
jtransc design was to support all targets including javascript that doesn't
support the traditional threading model, but one based on workers without
sharing state at all. But I will try to implement multithreading soon for
supported targets.

I used termux, but they do not support java/openjdk yet; so no 
gradle. Even when I'm able to transcompile jtransc itself so I 
could convert it to javascript and run it on node converting 
.class and .jar files into something else. But it is ideally used 
from gradle and compiling java/kt files into .class in the 
process. Also gradle really needs dynamic loading, so no AOT 
possible. It would be amazing if termux people support java.

What I can do is to generate the program.d file in my desktop and 
try D compiler on it.
But I have to investigate it a bit more.
Nov 29 2016
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-11-29 11:37, soywiz wrote:


  Jacob
 Cool. So I can expect it to work on iOS Obj-C too right?


Yes, using LDC.


 Someone tried a wrapper using RAII to emulate partially (except for weak
references) ARC?


Not that I know of.

-- 
/Jacob Carlborg
Nov 29 2016
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Tuesday, 29 November 2016 at 10:37:53 UTC, soywiz wrote:

 Amazing! Right now all the generated code uses __gshared. In 
 Java you have to use 
 https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html to get
TLS. And I didn't implemented it yet for multithreading. Because the initial
jtransc design was to support all targets including javascript that doesn't
support the traditional threading model, but one based on workers without
sharing state at all. But I will try to implement multithreading soon for
supported targets.


OK, so the way TLS is done doesn't matter for you.  There is 
another form of emulated TLS right now, that requires a small 
patch to llvm, so it is there already if you want to try it.


 I used termux, but they do not support java/openjdk yet; so no 
 gradle. Even when I'm able to transcompile jtransc itself so I 
 could convert it to javascript and run it on node converting 
 .class and .jar files into something else. But it is ideally 
 used from gradle and compiling java/kt files into .class in the 
 process. Also gradle really needs dynamic loading, so no AOT 
 possible. It would be amazing if termux people support java.


They do, after all Android runs Java.  For example, jack, the new 
Java compiler from google that is now the default in the Android 
SDK and is itself written in Java, is available as a package in 
Termux.  I use it to compile the Java portions of the sample apps 
from the NDK, seems to work fine.  There are many other Java 
utilities available in Termux, though not gradle.


 What I can do is to generate the program.d file in my desktop 
 and try D compiler on it.
 But I have to investigate it a bit more.


Yes, this is what I had in mind, in any case.  Generate the D 
code elsewhere by using your tool, then cross-compile using the 
ldc cross-compiler for Android linked above.
Nov 29 2016
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-11-29 02:09, soywiz wrote:


 And for iOS it just supports an AOT approach except for javascript. So
 it is great too.


BTW, it's possible to interface Objective-C from D [1].

[1] http://dlang.org/spec/objc_interface.html

-- 
/Jacob Carlborg
Nov 29 2016
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-11-28 15:28, soywiz wrote:

 I am working on a software that allows to transform JVM bytecode into
 something else. In last version I have done a D target:
 https://github.com/jtransc/jtransc
 D target already passes my test suite and works just fine (next version
 will come with some optimizations regarding to code generation that will
 improve generation).
 At this point I'm successfully generating windows/linux and mac native
 executables from Java/Kotlin code.
 It would be awesome to be able to support Android and iOS. I have seen
 that there was Android/iOS alpha support in the past. What is the
 current status of this?


Hmm, interesting. Is there any examples of how the generated code looks 
like?

-- 
/Jacob Carlborg
Nov 28 2016