www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Is it feasible to slowly rewrite a C++ codebase in D?

reply Jordi =?UTF-8?B?R3V0acOpcnJleg==?= Hermoso <jordigh octave.org> writes:
I'm specifically thinking of the GNU Octave codebase:

http://hg.savannah.gnu.org/hgweb/octave/file/ 

It's a fairly old and complicated C++ codebase. I would like to 
see if I could slowly introduce some D in it, anywhere.

Now, as I understand it, I would need to begin with making `main` 
a D function, because D needs to initialise the runtime. Is this 
correct?

Another possibility might be in dlopen'able functions. Currently 
Octave uses so-called oct functions, which are nothing more than 
C++ object code that is dynamically loaded by the interpreter at 
runtime. They are compiled to the Octave C++ API, but we also 
have a Matlab-compatible C API that perhaps could be more 
amenable for D-ification.

What are your ideas?
Jun 20 2018
next sibling parent bachmeier <no spam.net> writes:
On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
Hermoso wrote:
 I'm specifically thinking of the GNU Octave codebase:

 http://hg.savannah.gnu.org/hgweb/octave/file/ 

 It's a fairly old and complicated C++ codebase. I would like to 
 see if I could slowly introduce some D in it, anywhere.

 Now, as I understand it, I would need to begin with making 
 `main` a D function, because D needs to initialise the runtime. 
 Is this correct?

 Another possibility might be in dlopen'able functions. 
 Currently Octave uses so-called oct functions, which are 
 nothing more than C++ object code that is dynamically loaded by 
 the interpreter at runtime. They are compiled to the Octave C++ 
 API, but we also have a Matlab-compatible C API that perhaps 
 could be more amenable for D-ification.

 What are your ideas?
I've looked into this a bit, but haven't had time to do anything with it. My opinion is that the starting point is to add functionality using Oct-Files. The reason is that it is the simplest way to do so in a way that you can share your work with others. That allows you to call into the Octave API and reuse that code. Given the recent work on C++ interoperability, I think this strategy is the clear winner.
Jun 20 2018
prev sibling next sibling parent bachmeier <no spam.net> writes:
On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
Hermoso wrote:
 I'm specifically thinking of the GNU Octave codebase:

 http://hg.savannah.gnu.org/hgweb/octave/file/ 

 It's a fairly old and complicated C++ codebase. I would like to 
 see if I could slowly introduce some D in it, anywhere.

 Now, as I understand it, I would need to begin with making 
 `main` a D function, because D needs to initialise the runtime. 
 Is this correct?
With respect to this part of your question, you might want to check out https://dlang.org/spec/betterc.html and especially the limitations https://dlang.org/spec/betterc.html#consequences It shouldn't take much to do this, but the limitations of betterC are not minor.
Jun 20 2018
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
Hermoso wrote:
 I'm specifically thinking of the GNU Octave codebase:

 http://hg.savannah.gnu.org/hgweb/octave/file/ 

 It's a fairly old and complicated C++ codebase. I would like to 
 see if I could slowly introduce some D in it, anywhere.

 Now, as I understand it, I would need to begin with making 
 `main` a D function, because D needs to initialise the runtime. 
 Is this correct?

 Another possibility might be in dlopen'able functions. 
 Currently Octave uses so-called oct functions, which are 
 nothing more than C++ object code that is dynamically loaded by 
 the interpreter at runtime. They are compiled to the Octave C++ 
 API, but we also have a Matlab-compatible C API that perhaps 
 could be more amenable for D-ification.

 What are your ideas?
I'm a little confused...you mean like embedr [1, 2] and pyd [3], but for Octave and Matlab? The idea would that you could write functions in D and call them in Matlab/Octave, and vice-versa. I suppose that. I'm not sure adding D to the GNU Octave code base is necessarily the biggest value add... There was an old forum post years ago on the subject [4], but the link to the repository is dead. [1] https://bitbucket.org/bachmeil/embedr [2] https://github.com/bdilday/embedr [3] https://code.dlang.org/packages/pyd [4] https://forum.dlang.org/thread/op.vhjavyoc3ncmek enigma.fem.tu-ilmenau.de
Jun 20 2018
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 20 June 2018 at 19:57:55 UTC, jmh530 wrote:
 
 I suppose that. [snip]
I suppose that would be good.
Jun 20 2018
prev sibling parent Jordi =?UTF-8?B?R3V0acOpcnJleg==?= Hermoso <jordigh octave.org> writes:
On Wednesday, 20 June 2018 at 19:57:55 UTC, jmh530 wrote:
 I'm not sure adding D to the GNU Octave code base is 
 necessarily the biggest value add...
I'm daydreaming of being able to rewrite all of Octave in D. I just was trying to think of where to start.
Jul 10 2018
prev sibling next sibling parent user1234 <user1234 12.nl> writes:
On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
Hermoso wrote:
 I'm specifically thinking of the GNU Octave codebase:

 http://hg.savannah.gnu.org/hgweb/octave/file/ 

 It's a fairly old and complicated C++ codebase. I would like to 
 see if I could slowly introduce some D in it, anywhere.
- This can be a problem if the project is very active. This issue was noticed by the people who converted DMD compiler from C++ to D and finally a dedicated tool was used, improved until some CI tests passed.
 Now, as I understand it, I would need to begin with making 
 `main` a D function, because D needs to initialise the runtime. 
 Is this correct?
- The runtime can be initialized by hand if it's required, see core.runtime.Runtime.initialize()
Jun 20 2018
prev sibling next sibling parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
Hermoso wrote:
 I'm specifically thinking of the GNU Octave codebase:

 http://hg.savannah.gnu.org/hgweb/octave/file/ 

 It's a fairly old and complicated C++ codebase. I would like to 
 see if I could slowly introduce some D in it, anywhere.

 Now, as I understand it, I would need to begin with making 
 `main` a D function, because D needs to initialise the runtime. 
 Is this correct?

 Another possibility might be in dlopen'able functions. 
 Currently Octave uses so-called oct functions, which are 
 nothing more than C++ object code that is dynamically loaded by 
 the interpreter at runtime. They are compiled to the Octave C++ 
 API, but we also have a Matlab-compatible C API that perhaps 
 could be more amenable for D-ification.

 What are your ideas?
Maybe looking at the recent DMD Backend to D conversion PRs (https://github.com/dlang/dmd/pulls?utf8=%E2%9C%93&q=is%3Apr+label%3A 22D+Conversion%22+) helps? Here -betterC is used. The frontend was automatically converted to D a few years ago (https://github.com/dlang/dmd/tree/last-cdmd/src/magicport), though magicport was a bit specific to DMD's codebase.
Jul 10 2018
parent reply Jordi =?UTF-8?B?R3V0acOpcnJleg==?= Hermoso <jordigh octave.org> writes:
On Tuesday, 10 July 2018 at 20:28:00 UTC, Seb wrote:

 Maybe looking at the recent DMD Backend to D conversion PRs 
 (https://github.com/dlang/dmd/pulls?utf8=%E2%9C%93&q=is%3Apr+label%3A
22D+Conversion%22+) helps?
 Here -betterC is used.
Octave is so far from -betterC, though. It's very C++-heavy, with C++11 features being used since the last couple of years. It's an old codebase that started circa 1992. Just getting it into -betterC territory seems like a very daunting task.
Jul 11 2018
parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 11 July 2018 at 19:41:37 UTC, Jordi Gutiérrez 
Hermoso wrote:
 Just getting it into -betterC territory seems like a very 
 daunting task.
You do not need -betterC anymore. At least the LDC frontend will only add linking hooks for what you use, -betterC or no. No need build a stub runtime anymore or give a switch to the compiler to not use the default one. I know because I compile to JavaScript: first to LLVM bitcode, then manual link (with llvm-link), then to JavaScript using Emscripten. I only have to compile those parts of DRuntime and Phobos I use. Its unlikely I could even have a stub runtime to work, so this is the only reason I can use D in my web page to any real degree.
Jul 11 2018
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 11 July 2018 at 20:38:13 UTC, Dukc wrote:
 On Wednesday, 11 July 2018 at 19:41:37 UTC, Jordi Gutiérrez 
 Hermoso wrote:
 Just getting it into -betterC territory seems like a very 
 daunting task.
You do not need -betterC anymore. At least the LDC frontend will only add linking hooks for what you use, -betterC or no. No need build a stub runtime anymore or give a switch to the compiler to not use the default one. I know because I compile to JavaScript: first to LLVM bitcode, then manual link (with llvm-link), then to JavaScript using Emscripten. I only have to compile those parts of DRuntime and Phobos I use. Its unlikely I could even have a stub runtime to work, so this is the only reason I can use D in my web page to any real degree.
Also since 2.079 the GC (and many parts of Phobos) get lazily initialized, so while I think there are still some features that might require initialization, it might also be feasible to simply use normal D. Have you already tried this?
Jul 12 2018
parent reply Dukc <ajieskola gmail.com> writes:
On Thursday, 12 July 2018 at 15:12:51 UTC, Seb wrote:
 it might also be feasible to simply use normal D.

 Have you already tried this?
There's no strict distinction between using D normally and in systems programming fashion for me, because my main function isn't written in D. But in practice it's systems-style code. I mostly use stdc suff which can conveniently link into libraries coming with Emscripten, and std.range and std.algorithm, which do not require instantiations. There are a few instantiated thing from DRuntime and Phobos too, at least slice copying and stuff from std.random. I do not use the GC, nor anything else which has required to call a global initializator. I know about Vladimir's d-scripten tools library which would help, but it's based on Alawains library copyleft library, which makes also Vladimir's work copyleft, so I won't use it.
Jul 13 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 13 July 2018 at 07:51:51 UTC, Dukc wrote:
 I know about Vladimir's d-scripten tools library which would 
 help, but it's based on Alawains library copyleft library, 
 which makes also Vladimir's work copyleft, so I won't use it.
Hmm, I wasn't aware of this. I wonder if the decision to make the redistributable parts of dscripten was intentional (maybe the intention was to apply the license only to the scripts which build the toolchain). I'll follow up with Alawain. Regardless, dscripten-tools borrows very little from the redistributable parts of dscripten - mostly the "minimalistic runtime", which I think was itself borrowed from somewhere else.
Jul 13 2018
parent reply Dukc <ajieskola gmail.com> writes:
On Saturday, 14 July 2018 at 03:08:50 UTC, Vladimir Panteleev 
wrote:
 I'll follow up with Alawain. Regardless, dscripten-tools 
 borrows very little from the redistributable parts of dscripten 
 - mostly the "minimalistic runtime", which I think was itself 
 borrowed from somewhere else.
Indeed, it appears Alawain himself just changed the runtime part to be boost! If he is reading this, thank you!
Jul 16 2018
parent reply Seb <seb wilzba.ch> writes:
On Monday, 16 July 2018 at 11:12:20 UTC, Dukc wrote:
 On Saturday, 14 July 2018 at 03:08:50 UTC, Vladimir Panteleev 
 wrote:
 I'll follow up with Alawain. Regardless, dscripten-tools 
 borrows very little from the redistributable parts of 
 dscripten - mostly the "minimalistic runtime", which I think 
 was itself borrowed from somewhere else.
Indeed, it appears Alawain himself just changed the runtime part to be boost! If he is reading this, thank you!
https://github.com/Ace17/dscriptenQ/issues/4
Jul 16 2018
parent reply Timoses <timosesu gmail.com> writes:
On Monday, 16 July 2018 at 11:31:32 UTC, Seb wrote:
 On Monday, 16 July 2018 at 11:12:20 UTC, Dukc wrote:
 On Saturday, 14 July 2018 at 03:08:50 UTC, Vladimir Panteleev 
 wrote:
 I'll follow up with Alawain. Regardless, dscripten-tools 
 borrows very little from the redistributable parts of 
 dscripten - mostly the "minimalistic runtime", which I think 
 was itself borrowed from somewhere else.
Indeed, it appears Alawain himself just changed the runtime part to be boost! If he is reading this, thank you!
https://github.com/Ace17/dscriptenQ/issues/4
Superfluous Q attack. Working link: https://github.com/Ace17/dscripten/issues/4
Jul 16 2018
parent Seb <seb wilzba.ch> writes:
On Monday, 16 July 2018 at 13:15:56 UTC, Timoses wrote:
 On Monday, 16 July 2018 at 11:31:32 UTC, Seb wrote:
 On Monday, 16 July 2018 at 11:12:20 UTC, Dukc wrote:
 On Saturday, 14 July 2018 at 03:08:50 UTC, Vladimir Panteleev 
 wrote:
 I'll follow up with Alawain. Regardless, dscripten-tools 
 borrows very little from the redistributable parts of 
 dscripten - mostly the "minimalistic runtime", which I think 
 was itself borrowed from somewhere else.
Indeed, it appears Alawain himself just changed the runtime part to be boost! If he is reading this, thank you!
https://github.com/Ace17/dscriptenQ/issues/4
Superfluous Q attack. Working link: https://github.com/Ace17/dscripten/issues/4
Sorry for the typo (not sure how that happened) and thanks for correcting!
Jul 16 2018
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
Hermoso wrote:
 Now, as I understand it, I would need to begin with making 
 `main` a D function, because D needs to initialise the runtime. 
 Is this correct?
No. Some initialization is required if you use the GC, as I understand it. But there's no rule it has to be done in a D main function, you can initialize it from anywhere you like. And if you avoid using stuff that requires the runtime (I'm not sure what it constitutes in addition to GC. Perhaps classes, perhaps typeid, perhaps static constructors and destructors) you do not need to initialize it. -betterC style stuff can be done without it, and as I said it does not need the -betterC switch anymore, at least in LDC.
Jul 13 2018
prev sibling next sibling parent reply Laeeth Isharc <laeeth kaleidic.io> writes:
On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
Hermoso wrote:
 I'm specifically thinking of the GNU Octave codebase:

 http://hg.savannah.gnu.org/hgweb/octave/file/ 

 It's a fairly old and complicated C++ codebase. I would like to 
 see if I could slowly introduce some D in it, anywhere.

 Now, as I understand it, I would need to begin with making 
 `main` a D function, because D needs to initialise the runtime. 
 Is this correct?

 Another possibility might be in dlopen'able functions. 
 Currently Octave uses so-called oct functions, which are 
 nothing more than C++ object code that is dynamically loaded by 
 the interpreter at runtime. They are compiled to the Octave C++ 
 API, but we also have a Matlab-compatible C API that perhaps 
 could be more amenable for D-ification.

 What are your ideas?
If you would like to expose C function and type declarations to D, you could take a look at DPP, which allows you to just #include a C header. If you encounter a bug, please file an issue and in time we will fix it. Does not yet work for C++ except in some cases. https://github.com/atilaneves/dpp
Jul 13 2018
parent reply bachmeier <no spam.net> writes:
On Friday, 13 July 2018 at 19:53:45 UTC, Laeeth Isharc wrote:
 On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
 Hermoso wrote:
 What are your ideas?
If you would like to expose C function and type declarations to D, you could take a look at DPP, which allows you to just #include a C header. If you encounter a bug, please file an issue and in time we will fix it. Does not yet work for C++ except in some cases. https://github.com/atilaneves/dpp
I would like to give this a try, but I don't see any instructions for installation on the Github page. I tried dub build dpp, but I got a bunch of error messages.
Jul 16 2018
parent reply drug <drug2004 bk.ru> writes:
16.07.2018 17:32, bachmeier пишет:
 On Friday, 13 July 2018 at 19:53:45 UTC, Laeeth Isharc wrote:
 On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez Hermoso 
 wrote:
 What are your ideas?
If you would like to expose C function and type declarations to D, you could take a look at DPP, which allows you to just #include a C header.  If you encounter a bug, please file an issue and in time we will fix it. Does not yet work for C++ except in some cases. https://github.com/atilaneves/dpp
I would like to give this a try, but I don't see any instructions for installation on the Github page. I tried dub build dpp, but I got a bunch of error messages.
I just build it using dub then do something like that(from memory): `d++ some_c_header.h --include-path path/to/other/c/header/files --keep-d-files` and it generates d file for the corresponding c header. Although dpp is intended to be used directly I use it for generating intermediate files. First of all because when I tried to import nuklear.h dpp generated nuklear.d with simple error - due to bug it included diagnostic message of libclang into d file and it was simpler and faster to edit intermediate file manually than wait for bug fixing.
Jul 16 2018
parent reply bachmeier <no spam.net> writes:
On Tuesday, 17 July 2018 at 06:57:37 UTC, drug wrote:
 I just build it using dub then do something like that(from 
 memory):
 `d++ some_c_header.h --include-path 
 path/to/other/c/header/files --keep-d-files` and it generates d 
 file for the corresponding c header. Although dpp is intended 
 to be used directly I use it for generating intermediate files. 
 First of all because when I tried to import nuklear.h dpp 
 generated nuklear.d with simple error - due to bug it included 
 diagnostic message of libclang into d file and it was simpler 
 and faster to edit intermediate file manually than wait for bug 
 fixing.
I'm going to create an issue on Github. This is the output I get: $ dub build dpp Building package dpp in /home/office/.dub/packages/dpp-0.0.1/dpp/ WARNING: A deprecated branch based version specification is used for the dependency libclang. Please use numbered versions instead. Also note that you can still use the dub.selections.json file to override a certain dependency to use a branch instead. Performing "debug" build using /usr/bin/dmd for x86_64. libclang ~master: target for configuration "library" is up to date. dpp 0.0.1: building configuration "executable"... .dub/packages/dpp-0.0.1/dpp/source/dpp/runtime/app.d(106,16): Warning: C preprocessor directive #define is not supported .dub/packages/dpp-0.0.1/dpp/source/dpp/cursor/typedef_.d(89,9): Error: nogc function dpp.cursor.typedef_.isSomeFunction cannot call non- nogc function clang.Type.pointee .dub/packages/dpp-0.0.1/dpp/source/dpp/type/package.d(176,8): Error: incompatible types for (type.pointee()) is (null): Type and typeof(null) .dub/packages/dpp-0.0.1/dpp/source/dpp/type/package.d(177,12): Error: incompatible types for (type.pointee()) !is (null): Type and typeof(null) .dub/packages/dpp-0.0.1/dpp/source/dpp/type/package.d(184,40): Error: can only * a pointer, not a Type .dub/packages/dpp-0.0.1/dpp/source/dpp/type/package.d(192,21): Error: can only * a pointer, not a Type .dub/packages/dpp-0.0.1/dpp/source/dpp/type/package.d(202,22): Error: cannot implicitly convert expression (*ptr).pointee() of type Type to const(Type)* .dub/packages/dpp-0.0.1/dpp/source/dpp/type/package.d(246,31): Error: can only * a pointer, not a Type /usr/bin/dmd failed with exit code 1.
Jul 17 2018
parent reply bachmeier <no spam.net> writes:
On Tuesday, 17 July 2018 at 15:55:03 UTC, bachmeier wrote:
 On Tuesday, 17 July 2018 at 06:57:37 UTC, drug wrote:
 [...]
I'm going to create an issue on Github. This is the output I get: [...]
I solved that problem but now I have others. dpp is a good thing on paper but maybe not yet in practice.
Jul 17 2018
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 17 July 2018 at 16:39:48 UTC, bachmeier wrote:
 On Tuesday, 17 July 2018 at 15:55:03 UTC, bachmeier wrote:
 On Tuesday, 17 July 2018 at 06:57:37 UTC, drug wrote:
 [...]
I'm going to create an issue on Github. This is the output I get: [...]
I solved that problem but now I have others. dpp is a good thing on paper but maybe not yet in practice.
Echoing what Andrei and Walter say all the time, it's not going to get fixed without bug reports...
Jul 17 2018
parent bachmeier <no spam.net> writes:
On Tuesday, 17 July 2018 at 22:10:52 UTC, jmh530 wrote:
 On Tuesday, 17 July 2018 at 16:39:48 UTC, bachmeier wrote:
 On Tuesday, 17 July 2018 at 15:55:03 UTC, bachmeier wrote:
 On Tuesday, 17 July 2018 at 06:57:37 UTC, drug wrote:
 [...]
I'm going to create an issue on Github. This is the output I get: [...]
I solved that problem but now I have others. dpp is a good thing on paper but maybe not yet in practice.
Echoing what Andrei and Walter say all the time, it's not going to get fixed without bug reports...
I plan to report this...eventually.
Jul 18 2018
prev sibling parent reply bachmeier <no spam.net> writes:
On Wednesday, 20 June 2018 at 18:47:10 UTC, Jordi Gutiérrez 
Hermoso wrote:

 Another possibility might be in dlopen'able functions. 
 Currently Octave uses so-called oct functions, which are 
 nothing more than C++ object code that is dynamically loaded by 
 the interpreter at runtime. They are compiled to the Octave C++ 
 API, but we also have a Matlab-compatible C API that perhaps 
 could be more amenable for D-ification.
Can confirm that dpp works to create mex files. Rewriting the mypow2.c example in D: File mypow3.dpp #include "mex.h" import std.conv, std.stdio; extern(C) void mexFunction(int nlhs, mxArray** plhs, int nrhs, const mxArray** prhs) { mwSize n; mwIndex i; double * vri; double * vro; if (nrhs != 1 || !mxIsDouble(prhs[0])) mexErrMsgTxt ("ARG1 must be a double matrix"); n = mxGetNumberOfElements(prhs[0]).to!int; plhs[0] = mxCreateNumericArray (mxGetNumberOfDimensions (prhs[0]), mxGetDimensions (prhs[0]), mxGetClassID (prhs[0]), mxIsComplex(prhs[0]).to!mxComplexity); vri = mxGetPr (prhs[0]); vro = mxGetPr (plhs[0]); if (mxIsComplex (prhs[0])) { double* vii, vio; vii = mxGetPi (prhs[0]); vio = mxGetPi (plhs[0]); for (i = 0; i < n; i++) { vro[i] = vri[i] * vri[i] - vii[i] * vii[i]; vio[i] = 2 * vri[i] * vii[i]; } } else { for (i = 0; i < n; i++) vro[i] = vri[i] * vri[i]; } } Create the shared library using the information provided by mkoctfile: d++ -c -fPIC --include-path /usr/include/octave-4.2.2/octave/.. \ --include-path /usr/include/octave-4.2.2/octave \ --include-path /usr/include/hdf5/serial mypow3.dpp dmd -shared -defaultlib=libphobos2.so -of=mypow3.mex mypow3.o \ -L/usr/lib/x86_64-linux-gnu/liboctinterp.so \ -L/usr/lib/x86_64-linux-gnu/liboctave.so In Octave:
 b = randn (4,1);
 mypow3(b)
ans = 0.45984 3.54732 0.30075 2.09107
Jul 20 2018
parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 20 July 2018 at 15:20:43 UTC, bachmeier wrote:
 

 Can confirm that dpp works to create mex files. Rewriting the 
 mypow2.c example in D:

 [snip]
This sounds really great. I assume if it works with Octave it is very likely to work with Matlab's mex.h without issues (I might get dragged in to using Matlab again here soon).
Jul 20 2018