www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - D for sciencetific scripting / rapid protoryping

reply Prokop Hapala <prokophapala gmail.com> writes:
I'm examining the possibility to move from Python+C/C++ to D or 
Python+D. I read 
(https://wiki.dlang.org/Programming_in_D_for_Python_Programmers) 
and
(https://jackstouffer.com/blog/nd_slice.html), where is mentioned 
PyD, Mir-algorithm, all seems very promising. But I did not test 
it yet.

The main motivation why I like D over Julia 
(https://julialang.org/) is that D has smaller footprint (Julia 
basically needs kind of managed environment (like java virtual 
machine), it does not really compile to standalone native 
executable or library) and compilation of D seems to be 
paradoxically faster than initialization of Julia (despite Julia 
is JIT, in practice many operations (like recompiling, 
installing, importing libraries) are quite slow) - waiting 1 
minute for that considerably hamper fast development cycle.

Nevertheless I have several concerns now:

1) rdmd - it seems like exactly what I want (to use D as a script 
language, with super fast compilation). However, I cannot figure 
out how to make it work with libraries (e.g. If I use 
mir-algorithm, derelict-gl3, derelict-sdl). It does not seem to 
work together with dub and ignores dependencies from dub.json.

2) D footprint - the size of binary produced by D is kind of 
large. This simple project 
https://github.com/ProkopHapala/SimpleSimulationEngine/tree/master/Dlang/TEMP/fi
stTriangle-derelict produce 2.9MB with dmd and 3.5MB with ldc. This is because
libraries are lineked statically, I guess. There should be a way how to link
them dynamically (https://forum.dlang.org/thread/eimyvbojdmanbbshwv
c forum.dlang.org), at least with ldc, but I cannot figure out how. (sorry for
cross posting, but I got no answer to the previous post, I guess because it is
>1year old). Still I would like dynamic linking with dmd, since it has faster
compilation time than ldc.

I'm concerned about binary size because I want to generate many 
different small dynamic libraries (.so) on-the-fly. With 3MB per 
library I would easily waste Gigabytes of disk space that way. 
Basically I want to achieve functionality a bit like python weave 
(https://docs.scipy.org/doc/scipy-0.18.1/reference/tutorial/weave.html).

3) Dynamic loading of D-libraries with ctypes - from what I see 
in PyD examples 
(https://github.com/ariovistus/pyd/tree/master/examples) it all 
uses distutils/setup.py. That is perhaps nice for installation by 
end user, but not so convenient for rapid prototyping and 
development. There is this special construction:

extern(C) void PydMain() {
     def!(hello)();
     module_init();
}

I'm not sure what def! exactly does and how limiting it is? I 
would prefer just to generate standard .so library which exposes 
standard extern(C) functions. And than to load it dynamically 
with ctypes like any other c-library. Is that possible? (I did 
not yet tried to investicate it in detail). The reason is that 
while I often use python for prototyping, I'd like to produce .so 
libraries which I can than load from anywhere else (not just from 
python, but e.g. from C/C++) without changes. And it seems there 
is some problem loading D-library functions from C 
(https://dlang.org/spec/betterc.html).
Oct 21 2019
next sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Tue, Oct 22, 2019 at 8:00 AM Prokop Hapala via Digitalmars-d-learn
<digitalmars-d-learn puremagic.com> wrote:

 I'm examining the possibility to move from Python+C/C++ to D or
 Python+D. I read
 (https://wiki.dlang.org/Programming_in_D_for_Python_Programmers)
 and
 (https://jackstouffer.com/blog/nd_slice.html), where is mentioned
 PyD, Mir-algorithm, all seems very promising. But I did not test
 it yet.
...


You should try to use https://github.com/BindBC/bindbc-opengl and
https://github.com/BindBC/bindbc-sdl. There seems to be an issue with
derelict packages (mainly with the gl3 one)
And as far as I know derelict should be replaced by bindbc anyway in future.

And if you plan to have *.so libs you should add "targetType" :
"dynamicLibrary", to you dub.json
Oct 22 2019
parent reply Prokop Hapala <prokophapala gmail.com> writes:
On Tuesday, 22 October 2019 at 07:23:46 UTC, Daniel Kozak wrote:

 On Tue, Oct 22, 2019 at 8:00 AM Prokop Hapala via 
 Digitalmars-d-learn <digitalmars-d-learn puremagic.com> wrote:

 I'm examining the possibility to move from Python+C/C++ to D or
 Python+D. I read
 (https://wiki.dlang.org/Programming_in_D_for_Python_Programmers)
 and
 (https://jackstouffer.com/blog/nd_slice.html), where is 
 mentioned
 PyD, Mir-algorithm, all seems very promising. But I did not 
 test
 it yet.


  >...

 You should try to use https://github.com/BindBC/bindbc-opengl 
 and
 https://github.com/BindBC/bindbc-sdl. There seems to be an 
 issue with
 derelict packages (mainly with the gl3 one)
 And as far as I know derelict should be replaced by bindbc 
 anyway in future.

 And if you plan to have *.so libs you should add "targetType" : 
 "dynamicLibrary", to you dub.json


OK, thanks. That is useful to know. But just to not turn the 
topic elsewhere I should make clear that:

1) I'm not speaking about OpenGL and SDL specifically (that was 
just small example which I tried first)

2) I'm more concerned about how to D compiler links dependencies 
when it compiles simple .d program (with lot of dependencies).

I think if I can make it link everything dynamically, It would 
considerably reduce both size of binary target (whether it is 
executable or .so) and compilation speed (since it would not 
re-compile dependencies).

What I want is to recompile and run quite large programs/projects 
composed composed of many little sub-programs/sub-libraries from 
Python+D in fast cycles (<< 1 second), because that would make 
debugging workflow much more pleasant and efficient (in 
comparison to Python+C/C++ or Julia).
Oct 22 2019
next sibling parent reply Arun Chandrasekaran <aruncxy gmail.com> writes:
On Tuesday, 22 October 2019 at 07:40:01 UTC, Prokop Hapala wrote:

 On Tuesday, 22 October 2019 at 07:23:46 UTC, Daniel Kozak wrote:

 On Tue, Oct 22, 2019 at 8:00 AM Prokop Hapala via 
 Digitalmars-d-learn <digitalmars-d-learn puremagic.com> wrote:

 I'm examining the possibility to move from Python+C/C++ to D 
 or
 Python+D. I read
 (https://wiki.dlang.org/Programming_in_D_for_Python_Programmers)
 and
 (https://jackstouffer.com/blog/nd_slice.html), where is 
 mentioned
 PyD, Mir-algorithm, all seems very promising. But I did not 
 test
 it yet.


  >...

 You should try to use https://github.com/BindBC/bindbc-opengl 
 and
 https://github.com/BindBC/bindbc-sdl. There seems to be an 
 issue with
 derelict packages (mainly with the gl3 one)
 And as far as I know derelict should be replaced by bindbc 
 anyway in future.

 And if you plan to have *.so libs you should add "targetType" 
 : "dynamicLibrary", to you dub.json


 OK, thanks. That is useful to know. But just to not turn the 
 topic elsewhere I should make clear that:

 1) I'm not speaking about OpenGL and SDL specifically (that was 
 just small example which I tried first)

 2) I'm more concerned about how to D compiler links 
 dependencies when it compiles simple .d program (with lot of 
 dependencies).

 I think if I can make it link everything dynamically, It would 
 considerably reduce both size of binary target (whether it is 
 executable or .so) and compilation speed (since it would not 
 re-compile dependencies).

 What I want is to recompile and run quite large 
 programs/projects composed composed of many little 
 sub-programs/sub-libraries from Python+D in fast cycles (<< 1 
 second), because that would make debugging workflow much more 
 pleasant and efficient (in comparison to Python+C/C++ or Julia).


If you are building individual files, use ldc2 with 
--link-defaultlib-shared flag:

arun home-pc:/tmp$ cat a.d
void main() { import std; writeln("Hai"); }
arun home-pc:/tmp$ ldc2 a.d
arun home-pc:/tmp$ ldd a
         linux-vdso.so.1 (0x00007fff6395b000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
(0x00007f1ec91ea000)
         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007f1ec91c7000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
(0x00007f1ec9078000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
(0x00007f1ec905e000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
(0x00007f1ec8e6d000)
         /lib64/ld-linux-x86-64.so.2 (0x00007f1ec92c4000)
arun home-pc:/tmp$ ldc2 a.d --link-defaultlib-shared
arun home-pc:/tmp$ ldd a
         linux-vdso.so.1 (0x00007ffcbda7f000)
         libphobos2-ldc-shared.so.88 => 
/home/arun/.bin/ldc2-1.18.0-linux-x86_64/bin/../lib/libphobos2-ldc-shared.so.88
(0x00007f1b57be8000)
         libdruntime-ldc-shared.so.88 => 
/home/arun/.bin/ldc2-1.18.0-linux-x86_64/bin/../lib/libdrunt
me-ldc-shared.so.88 (0x00007f1b57abc000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
(0x00007f1b57a84000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
(0x00007f1b57893000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
(0x00007f1b57744000)
         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007f1b57721000)
         librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 
(0x00007f1b57714000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
(0x00007f1b5770e000)
         /lib64/ld-linux-x86-64.so.2 (0x00007f1b58067000)
arun home-pc:/tmp$
Oct 22 2019
parent reply Arun Chandrasekaran <aruncxy gmail.com> writes:
On Tuesday, 22 October 2019 at 07:51:16 UTC, Arun Chandrasekaran 
wrote:

 On Tuesday, 22 October 2019 at 07:40:01 UTC, Prokop Hapala 
 wrote:

 [...]


 If you are building individual files, use ldc2 with 
 --link-defaultlib-shared flag:

 arun home-pc:/tmp$ cat a.d
 void main() { import std; writeln("Hai"); }
 arun home-pc:/tmp$ ldc2 a.d
 arun home-pc:/tmp$ ldd a
         linux-vdso.so.1 (0x00007fff6395b000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
 (0x00007f1ec91ea000)
         libpthread.so.0 => 
 /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1ec91c7000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
 (0x00007f1ec9078000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
 (0x00007f1ec905e000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
 (0x00007f1ec8e6d000)
         /lib64/ld-linux-x86-64.so.2 (0x00007f1ec92c4000)
 arun home-pc:/tmp$ ldc2 a.d --link-defaultlib-shared
 arun home-pc:/tmp$ ldd a
         linux-vdso.so.1 (0x00007ffcbda7f000)
         libphobos2-ldc-shared.so.88 => 
 /home/arun/.bin/ldc2-1.18.0-linux-x86_64/bin/../lib/libphob
s2-ldc-shared.so.88 (0x00007f1b57be8000)
         libdruntime-ldc-shared.so.88 => 
 /home/arun/.bin/ldc2-1.18.0-linux-x86_64/bin/../lib/libdrunt
me-ldc-shared.so.88 (0x00007f1b57abc000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
 (0x00007f1b57a84000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
 (0x00007f1b57893000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
 (0x00007f1b57744000)
         libpthread.so.0 => 
 /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1b57721000)
         librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 
 (0x00007f1b57714000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
 (0x00007f1b5770e000)
         /lib64/ld-linux-x86-64.so.2 (0x00007f1b58067000)
 arun home-pc:/tmp$


If you want similar behavior with dub, here is a sample dub.json

arun home-pc:/tmp/test$ cat dub.json
{
     "authors": [
         "Arun"
     ],
     "copyright": "Copyright © 2019, Arun",
     "description": "A minimal D application.",
     "license": "proprietary",
     "dflags-ldc": ["-link-defaultlib-shared"],
     "name": "test"
}
arun home-pc:/tmp/test$

Dub settings can be at times intimidating, but still give it a 
read https://dub.pm/package-format-json#build-settings
Oct 22 2019
parent reply Prokop Hapala <prokophapala gmail.com> writes:
On Tuesday, 22 October 2019 at 08:04:32 UTC, Arun Chandrasekaran 
wrote:

 On Tuesday, 22 October 2019 at 07:51:16 UTC, Arun 
 Chandrasekaran wrote:

 On Tuesday, 22 October 2019 at 07:40:01 UTC, Prokop Hapala 
 wrote:

 [...]


 If you are building individual files, use ldc2 with 
 --link-defaultlib-shared flag:

 arun home-pc:/tmp$ cat a.d
 void main() { import std; writeln("Hai"); }
 arun home-pc:/tmp$ ldc2 a.d
 arun home-pc:/tmp$ ldd a
         linux-vdso.so.1 (0x00007fff6395b000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
 (0x00007f1ec91ea000)
         libpthread.so.0 => 
 /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1ec91c7000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
 (0x00007f1ec9078000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
 (0x00007f1ec905e000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
 (0x00007f1ec8e6d000)
         /lib64/ld-linux-x86-64.so.2 (0x00007f1ec92c4000)
 arun home-pc:/tmp$ ldc2 a.d --link-defaultlib-shared
 arun home-pc:/tmp$ ldd a
         linux-vdso.so.1 (0x00007ffcbda7f000)
         libphobos2-ldc-shared.so.88 => 
 /home/arun/.bin/ldc2-1.18.0-linux-x86_64/bin/../lib/libphob
s2-ldc-shared.so.88 (0x00007f1b57be8000)
         libdruntime-ldc-shared.so.88 => 
 /home/arun/.bin/ldc2-1.18.0-linux-x86_64/bin/../lib/libdrunt
me-ldc-shared.so.88 (0x00007f1b57abc000)
         libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
 (0x00007f1b57a84000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 
 (0x00007f1b57893000)
         libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 
 (0x00007f1b57744000)
         libpthread.so.0 => 
 /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1b57721000)
         librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 
 (0x00007f1b57714000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 
 (0x00007f1b5770e000)
         /lib64/ld-linux-x86-64.so.2 (0x00007f1b58067000)
 arun home-pc:/tmp$


 If you want similar behavior with dub, here is a sample dub.json

 arun home-pc:/tmp/test$ cat dub.json
 {
     "authors": [
         "Arun"
     ],
     "copyright": "Copyright © 2019, Arun",
     "description": "A minimal D application.",
     "license": "proprietary",
     "dflags-ldc": ["-link-defaultlib-shared"],
     "name": "test"
 }
 arun home-pc:/tmp/test$

 Dub settings can be at times intimidating, but still give it a 
 read https://dub.pm/package-format-json#build-settings


Thank you Arun, that is very helpful!

Do you have also some example which use some other than default 
library ?

I see there is quite comprehensive description how to use dmd 
with dynamic libraries
https://dlang.org/articles/dll-linux.html

But there is nothing how to integrate it with dub and deb.json (I 
really like DUB, it's one of the best improvements over C/C++ 
enviroment I noticed)

Also where is RDMD in the equation? I really like the idea run 
binary programs like:


import std.stdio;
void main(){
     writeln("Hello, world!");
}

But I cannot find any documentation how to use rdmd with any 
libraries/dependencies and dub.json ? Not even statically linked, 
not to say dynamic.
Oct 22 2019
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Tue, Oct 22, 2019 at 10:25 AM Prokop Hapala via Digitalmars-d-learn
<digitalmars-d-learn puremagic.com> wrote:

 ...

 Also where is RDMD in the equation? I really like the idea run
 binary programs like:


 import std.stdio;
 void main(){
      writeln("Hello, world!");
 }

 But I cannot find any documentation how to use rdmd with any
 libraries/dependencies and dub.json ? Not even statically linked,
 not to say dynamic.


You can't use rdmd with dub, but for simple scripts you can use dub directly:

https://dub.pm/advanced_usage.html
Oct 22 2019
parent Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 22 October 2019 at 08:30:36 UTC, Daniel Kozak wrote:

 On Tue, Oct 22, 2019 at 10:25 AM Prokop Hapala via 
 Digitalmars-d-learn <digitalmars-d-learn puremagic.com> wrote:

 ...

 Also where is RDMD in the equation? I really like the idea run 
 binary programs like:


 import std.stdio;
 void main(){
      writeln("Hello, world!");
 }

 But I cannot find any documentation how to use rdmd with any 
 libraries/dependencies and dub.json ? Not even statically 
 linked, not to say dynamic.


 You can't use rdmd with dub, but for simple scripts you can use 
 dub directly:

 https://dub.pm/advanced_usage.html


Here an example, save it as example.d:

```d


/+ dub.json:
{
     "name":"example-s3",
     "dependencies":{
         "aws-sdk:core": "1.0.0",
         "aws-sdk:s3": "1.0.0"
     }
}
+/

import aws.sdk.core, aws.sdk.s3;

void main()
{
     auto client = new AwsClient();
     auto s3 = new S3Service(client);

     ...
}
```

On linux you could start this file using command 
```./example.d```.
Or with command ```dub example.d```.

Kind regards
André
Oct 22 2019
prev sibling parent Mike Parker <aldacron gmail.com> writes:
On Tuesday, 22 October 2019 at 07:40:01 UTC, Prokop Hapala wrote:


 1) I'm not speaking about OpenGL and SDL specifically (that was 
 just small example which I tried first)


FYI, the BindBC bindings can be configured as dynamic (in which 
all the of C library functions are declared as function pointers 
and the C shared library must be loaded manually at runtime) or 
static (in which the C library functions become normal function 
declarations with no implementation and there is a link-time 
dependency on the C static or shared library). The exception is 
OpenGL, for which there is no static binding.

The dynamic bindings add the overhead of the function pointer 
declarations when linked. It's insignificant for most of them. 
The biggest bindbc-opengl, depending on how many extensions you 
configure (it's extremely high for DerelictGL3 most of the time, 
as it tends to pull in more extensions).


 2) I'm more concerned about how to D compiler links 
 dependencies when it compiles simple .d program (with lot of 
 dependencies).


When using dub, it depends on the package configuration of each 
dependency. All of the Derelict and BindBC packages, for example, 
are configured to build as static libraries. It didn't (and 
doesn't) make sense to me to compile a binding as a shared 
library. Dub packages can also be configured as shared libraries, 
or simply as "library" (I haven't investigated what that means). 
Some might provide custom subconfigurations to choose static or 
shared.

The biggest overhead with DMD output tends to come from the 
standard library. On Windows, the static library the only option. 
IIRC on Linux it's necessary to pass -defaultlib=libphobos2.so to 
DMD to link the shared lib. I don't know how LDC handles it.
Oct 22 2019
prev sibling parent Laeeth Isharc <laeeth kaleidic.io> writes:
On Tuesday, 22 October 2019 at 05:58:50 UTC, Prokop Hapala wrote:

 I'm examining the possibility to move from Python+C/C++ to D or 
 Python+D. I read 
 (https://wiki.dlang.org/Programming_in_D_for_Python_Programmers) and
 (https://jackstouffer.com/blog/nd_slice.html), where is 
 mentioned PyD, Mir-algorithm, all seems very promising. But I 
 did not test it yet.

 [...]


See autowrap.  PyNih might eventually be a bit nicer than pyd but 
it's not yet there.

We did some very early work on Julia integration and probably 
will finish when time.

You can call R libraries from D too.

If you do use pyd then ppyd might make it a bit more pleasant.

Some rough edges around pyd but it's okay if you don't mind 
figuring things out.
Oct 22 2019