www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - llvm-d

reply "Moritz Maxeiner" <moritz ucworks.org> writes:
Hi, I would like to announce llvm-d, which provides LLVM bindings 
for D.

It loads LLVM from a dynamic library (so/dylib/dll) and has 
support
for LLVM versions 3.1, 3.2 and 3.3 (current svn trunk).
It has as of now been tested on 64bit versions of Archlinux, OS X 
Mountain Lion and Windows 7.

The support for multiple versions is achieved via conditional 
compilation (As the LLVM C API sometimes has changes such as 
insertion of new enum items
in the middle of an enum, changing the values for all succeeding 
enum items), specifically the version system: LLVM version X.Y 
translates to the version identifier "LLVM_X_Y" (without the 
quotes).

More information (and an example) can be found at llvm-d's 
location:

https://github.com/Calrama/llvm-d

and also has a DUB registry entry at

http://registry.vibed.org/packages/llvm-d
Mar 15 2013
next sibling parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 Hi, I would like to announce llvm-d, which provides LLVM bindings
 for D.
 
 It loads LLVM from a dynamic library (so/dylib/dll) and has support
 for LLVM versions 3.1, 3.2 and 3.3 (current svn trunk).
 It has as of now been tested on 64bit versions of Archlinux, OS X
 Mountain Lion and Windows 7.
 
 The support for multiple versions is achieved via conditional
 compilation (As the LLVM C API sometimes has changes such as
 insertion of new enum items
 in the middle of an enum, changing the values for all succeeding
 enum items), specifically the version system: LLVM version X.Y
 translates to the version identifier "LLVM_X_Y" (without the
 quotes).
 
 More information (and an example) can be found at llvm-d's location:
 
 https://github.com/Calrama/llvm-d
 
 and also has a DUB registry entry at
 
 http://registry.vibed.org/packages/llvm-d


Nice.
Can you make it compatible with Deimos
https://github.com/D-Programming-Deimos/?

Jens
Mar 15 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Friday, 15 March 2013 at 21:18:09 UTC, Jens Mueller wrote:

 Nice.
 Can you make it compatible with Deimos
 https://github.com/D-Programming-Deimos/?

 Jens


If by compatible you mean "can be used with":

I don't see anything that would prevent you from using llvm-d 
together
with deimos bindings for other libraries.

If by compatible you mean "can be integrated into":

Afaik Deimos projects are linked with the libraries, instead of 
loading
them at runtime. This is something I do not want because of the 
coff/omf
hassle with optlink (on win32), which I can avoid by sidestepping 
the linker.

The next problem would be that afaik deimos is for pure C 
bindings only, but as mentioned under "Planned Features" on the 
github page I do plan on more or less recreating LLVM's OOP 
structure (as much as the C API exposes, anyway) in D around the 
C API as close to the original as sensible and feasible. Keeping 
two seperate projects for that seemed inefficient to me, even 
more so when one of them would only change for bug fixes (if any) 
and for new LLVM versions.

Another pro for using runtime loading is that I can move the 
names of the libraries into llvm-d (which is what I did), so 
someone using llvm-d does not have to worry about library names, 
prefixes and suffixes. The most they have to do it to give -ldl 
to the linker on posix.

These are the main reasons as to why I decided against making 
another deimos project in the first place and I can't think of a 
compelling reason for deviating from that decision right now, 
sorry. Correct me if I'm wrong, though, there's much about D I 
don't know yet.

If, however, there are enough people that *really* want a deimos 
version I could fork the project, strip out the shared lib stuff 
and change a few calls in the code generation to create extern 
functions instead of extern function pointers (The process is 
automated via CTFE, a mixin version of the "map" function and 
associative array enums) and that would be it, although I don't 
really see any real benefit to that (other than it being in 
deimos, which I don't think is a key factor now that we have dub 
with its package registry around).
Mar 15 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Friday, 15 March 2013 at 21:18:09 UTC, Jens Mueller wrote:

Nice.
Can you make it compatible with Deimos
https://github.com/D-Programming-Deimos/?

Jens


 
 If by compatible you mean "can be used with":
 
 I don't see anything that would prevent you from using llvm-d
 together
 with deimos bindings for other libraries.
 
 If by compatible you mean "can be integrated into":
 
 Afaik Deimos projects are linked with the libraries, instead of
 loading
 them at runtime. This is something I do not want because of the
 coff/omf
 hassle with optlink (on win32), which I can avoid by sidestepping
 the linker.


Deimos just provides a plain D interface for C libraries.
What do you mean?
You can do static or dynamic linking.
With dynamic linking it will be loaded at startup time by the loader.
But you can also load it at runtime using some system API which I assume
you do. I started ddl (http://jkm.github.com/ddl/ddl.html) which does
work with Deimos to some extent. A limiting factor is compilation speed.


 The next problem would be that afaik deimos is for pure C bindings
 only, but as mentioned under "Planned Features" on the github page I
 do plan on more or less recreating LLVM's OOP structure (as much as
 the C API exposes, anyway) in D around the C API as close to the
 original as sensible and feasible. Keeping two seperate projects for
 that seemed inefficient to me, even more so when one of them would
 only change for bug fixes (if any) and for new LLVM versions.


Even though I understand your arguments. Technically, they are different
projects. As you say built on LLVM's C API. One could use git submodules
to ease building the higher level API.


 Another pro for using runtime loading is that I can move the names
 of the libraries into llvm-d (which is what I did), so someone using
 llvm-d does not have to worry about library names, prefixes and
 suffixes. The most they have to do it to give -ldl to the linker on
 posix.


Very sensible. Yet this is something you want in general. ddl is able to
do this.


 These are the main reasons as to why I decided against making
 another deimos project in the first place and I can't think of a
 compelling reason for deviating from that decision right now, sorry.
 Correct me if I'm wrong, though, there's much about D I don't know
 yet.


In sum you want to load libraries at runtime and have a higher level
API. Both are fine. But I question that it is a good decision to do the
loading at runtime for each C library.


 If, however, there are enough people that *really* want a deimos
 version I could fork the project, strip out the shared lib stuff and
 change a few calls in the code generation to create extern functions
 instead of extern function pointers (The process is automated via
 CTFE, a mixin version of the "map" function and associative array
 enums) and that would be it, although I don't really see any real
 benefit to that (other than it being in deimos, which I don't think
 is a key factor now that we have dub with its package registry
 around).


I would use Deimos bindings. I even started one myself.
https://github.com/jkm/deimos-llvm/
But I cannot devote much time on it.

Jens
Mar 17 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Sunday, 17 March 2013 at 20:05:09 UTC, Jens Mueller wrote:

 Deimos just provides a plain D interface for C libraries.
 What do you mean?
 You can do static or dynamic linking.
 With dynamic linking it will be loaded at startup time by the 
 loader.
 But you can also load it at runtime using some system API which 
 I assume
 you do. I started ddl (http://jkm.github.com/ddl/ddl.html) 
 which does
 work with Deimos to some extent. A limiting factor is 
 compilation speed.


Technically deimos provides extern(C) functions, which by 
themselves have - as  you said - to be linked 
statically/dynamically and cannot be loaded at runtime.
Normally to load at runtime you need a function pointer to give 
the OS functions to load from the shared lib and afaict you're 
using ctfe to extract the functions from deimos and create those 
functions pointers from them to circumvent that problem - kudos 
by the way, never even thought about that possibility.
Now to the real problem: When you're creating a shared lib on 
windows (32bit) it's going to be COFF, normally. But optlink 
can't link with that, only with OMF - which is one of the reason, 
I believe, why support for the MSLinker was added to windows 
64bit versions of dmd. You can use import libs to make dmd on 
win32 work with COFF shared libs, but again, that is extra work 
on the user of llvm-d, which I do not want. The solution for me 
was simply to cut out the middle man reponsible for that hassle 
(optlink) from the process.

On another note:
- Afaict ddl doesn't have OS X support, so I can't use it as a 
replacement for what I already have. If you want to add that to 
ddl you could just look at llvm/util/shlib.d in llvm-d, it's done 
there, missing support for one of the three majors is a big no-no 
for me.
- Another problem would be that I want to keep things compatible 
with dub, which would become very difficult seing as ddl doesn't 
have support for dub at present.
- Moreover, ddl doesn't seem to have support for loading from a 
specific directory (such as "lib"), or at least I can't find it
- There are functions in the LLVM C API, which are not available 
in every LLVM shared lib: The target specific ones. That means 
that - depending on which targets where selected when compiling 
th shared lib - these functions may or may not be in the shared 
lib and trying to load them from the shared lib MUST NOT throw an 
exception. If loading fails it simply means the shared lib 
doesn't have support for that specific target; ddl however, 
always throws an exception when it can't load a symbol.

Only with those issues resolved in ddl would it be even feasible 
for me to do a split into say llvm-d + (deimos)llvm and have 
llvm-d depend on ddl.
And even then llvm-d would depend on 2 projects (ddl and 
(deimos)llvm) instead of 0 like now and more dependencies for 
very little to no functionality increase is a very big argument 
against doing so for me.


 Even though I understand your arguments. Technically, they are 
 different
 projects. As you say built on LLVM's C API. One could use git 
 submodules
 to ease building the higher level API.


By that same argument, however, you could also claim that the D 
runtime core should be split into more projects - one who only 
imports the necessary C functions from the libc, and the other 
who does all the D stuff.
Not the best example, I know, but the point still stands: When 
the D OOP API is done, the llvm-d user should have no more need 
to use the C functions by themselves. Also LLVM A API functions 
often do strange things you wouldn't expect and the documentation 
is sometimes just simply wrong; using the LLVM C API from D with 
D's GC without something in between that takes care of the 
GC/NonGC gap is dangerous at best, I believe.


 In sum you want to load libraries at runtime and have a higher 
 level
 API. Both are fine. But I question that it is a good decision 
 to do the
 loading at runtime for each C library.


Not that I advocate doing it for every C library, but if one 
really wants to what would be the real downside? The program 
startup time would suffer, of course, but with current computer 
architectures would it really be by an amount that actually 
hurts? And would the comfort of not having to worry about 
different library formats and linkers  on different OS's outweigh 
that? It's not like it is actually calculation-heavy stuff, only 
loading symbols. You could even parallelise it if it's more than 
a couple of hundred.


 I would use Deimos bindings. I even started one myself.
 https://github.com/jkm/deimos-llvm/
 But I cannot devote much time on it.


That would make it one for and one somewhat against^^
In all seriousness, though: Even if I were to split llvm-d into 
the two parts, I can't do it right now, as only the C API stuff 
exists at present, so splitting would leave me with one empty 
github repository right now.
What I could do, however, is copying the C parts from llvm-d 
instead of splitting them. If I were to do that, I'd need to know 
how to get it in the deimos "D programming" group, though.

To sum it up: I'd much rather make a copy of the C stuff from 
llvm-d, make it a deimos project, but leave llvm-d untouched then 
to do a split and deal with the consequences. It might seem 
contraproductive, but the actual effort to support two very 
similar projects with updates is less than to deal with problems 
arising from the split.

- Moritz
Mar 18 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Sunday, 17 March 2013 at 20:05:09 UTC, Jens Mueller wrote:

Deimos just provides a plain D interface for C libraries.
What do you mean?
You can do static or dynamic linking.
With dynamic linking it will be loaded at startup time by the
loader.
But you can also load it at runtime using some system API which I
assume
you do. I started ddl (http://jkm.github.com/ddl/ddl.html) which
does
work with Deimos to some extent. A limiting factor is compilation
speed.


 
 Technically deimos provides extern(C) functions, which by themselves
 have - as  you said - to be linked statically/dynamically and cannot
 be loaded at runtime.
 Normally to load at runtime you need a function pointer to give the
 OS functions to load from the shared lib and afaict you're using
 ctfe to extract the functions from deimos and create those functions
 pointers from them to circumvent that problem - kudos by the way,
 never even thought about that possibility.
 Now to the real problem: When you're creating a shared lib on
 windows (32bit) it's going to be COFF, normally. But optlink can't
 link with that, only with OMF - which is one of the reason, I
 believe, why support for the MSLinker was added to windows 64bit
 versions of dmd. You can use import libs to make dmd on win32 work
 with COFF shared libs, but again, that is extra work on the user of
 llvm-d, which I do not want. The solution for me was simply to cut
 out the middle man reponsible for that hassle (optlink) from the
 process.


This makes perfect sense. I understand why that was your solution. But
it is limiting for non-Windows users.


 On another note:
 - Afaict ddl doesn't have OS X support, so I can't use it as a
 replacement for what I already have. If you want to add that to ddl
 you could just look at llvm/util/shlib.d in llvm-d, it's done there,
 missing support for one of the three majors is a big no-no for me.


I assumed OS X is Posix compliant. I never checked that though. Thanks
for the pointer.


 - Another problem would be that I want to keep things compatible
 with dub, which would become very difficult seing as ddl doesn't
 have support for dub at present.


No problem. I can add this. It is easy to do. I didn't do it because I
wasn't sure which package manager will be the "official" one. This
hasn't been settled as far as I know.


 - Moreover, ddl doesn't seem to have support for loading from a
 specific directory (such as "lib"), or at least I can't find it


I don't know how this is done on Windows. On Linux you just set the
LD_LIBRARY_PATH.


 - There are functions in the LLVM C API, which are not available in
 every LLVM shared lib: The target specific ones. That means that -
 depending on which targets where selected when compiling th shared
 lib - these functions may or may not be in the shared lib and trying
 to load them from the shared lib MUST NOT throw an exception. If
 loading fails it simply means the shared lib doesn't have support
 for that specific target; ddl however, always throws an exception
 when it can't load a symbol.


Uhh. I didn't know. Unfortunately I have very little experience with
the LLVM C API. But it is a very important use case for ddl and Deimos
in general.


 Only with those issues resolved in ddl would it be even feasible for
 me to do a split into say llvm-d + (deimos)llvm and have llvm-d
 depend on ddl.
 And even then llvm-d would depend on 2 projects (ddl and
 (deimos)llvm) instead of 0 like now and more dependencies for very
 little to no functionality increase is a very big argument against
 doing so for me.


I see. It's up to you. You know my preference.


Even though I understand your arguments. Technically, they are
different
projects. As you say built on LLVM's C API. One could use git
submodules
to ease building the higher level API.


 
 By that same argument, however, you could also claim that the D
 runtime core should be split into more projects - one who only
 imports the necessary C functions from the libc, and the other who
 does all the D stuff.
 Not the best example, I know, but the point still stands: When the D
 OOP API is done, the llvm-d user should have no more need to use the
 C functions by themselves. Also LLVM A API functions often do
 strange things you wouldn't expect and the documentation is
 sometimes just simply wrong; using the LLVM C API from D with D's GC
 without something in between that takes care of the GC/NonGC gap is
 dangerous at best, I believe.


Probably. There are very few users for such a use case.
But if you would use LLVM's plain C interface in D users who have used
the API in C will feel home. I mean why expose the LLVM C API as an
addition. It certainly does no harm.


In sum you want to load libraries at runtime and have a higher
level
API. Both are fine. But I question that it is a good decision to
do the
loading at runtime for each C library.


 
 Not that I advocate doing it for every C library, but if one really
 wants to what would be the real downside? The program startup time
 would suffer, of course, but with current computer architectures
 would it really be by an amount that actually hurts? And would the
 comfort of not having to worry about different library formats and
 linkers  on different OS's outweigh that? It's not like it is
 actually calculation-heavy stuff, only loading symbols. You could
 even parallelise it if it's more than a couple of hundred.


Usually, I use dynamic linking. I believe you would do the same if
optlink worked.


I would use Deimos bindings. I even started one myself.
https://github.com/jkm/deimos-llvm/
But I cannot devote much time on it.


 
 That would make it one for and one somewhat against^^
 In all seriousness, though: Even if I were to split llvm-d into the
 two parts, I can't do it right now, as only the C API stuff exists
 at present, so splitting would leave me with one empty github
 repository right now.
 What I could do, however, is copying the C parts from llvm-d instead
 of splitting them. If I were to do that, I'd need to know how to get
 it in the deimos "D programming" group, though.


The process is usually pretty simple. But the documentation is lacking.
There are several pieces floating around. I can try writing one document
and try to push it to the official web site.


 To sum it up: I'd much rather make a copy of the C stuff from
 llvm-d, make it a deimos project, but leave llvm-d untouched then to
 do a split and deal with the consequences. It might seem
 contraproductive, but the actual effort to support two very similar
 projects with updates is less than to deal with problems arising
 from the split.


Do as you like it. I do not think this is a bad approach.

Jens
Mar 18 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:

 I assumed OS X is Posix compliant. I never checked that though. 
 Thanks
 for the pointer.


No problem, OSX seems mostly Posix compliant. The shared 
libraries are suffixed with ".dylib", though, instead of ".so". 
There may be other differences you need to deal with, but that's 
the only one I ran into in my minimalistic approach.


 No problem. I can add this. It is easy to do. I didn't do it 
 because I
 wasn't sure which package manager will be the "official" one. 
 This
 hasn't been settled as far as I know.


You're right, of course, but at present time dub has a package 
registry and is easy to use, so I went with that. If, however, 
there is going to be an official package manager at some point in 
the future I'll change to that, then.


 - Moreover, ddl doesn't seem to have support for loading from a
 specific directory (such as "lib"), or at least I can't find it


 I don't know how this is done on Windows. On Linux you just set 
 the
 LD_LIBRARY_PATH.


My problem with that is that it's a) not encapsulated inside the 
program and b) OS dependent.


 Probably. There are very few users for such a use case.
 But if you would use LLVM's plain C interface in D users who 
 have used
 the API in C will feel home. I mean why expose the LLVM C API 
 as an
 addition. It certainly does no harm.


The C API will keep being exposed, of course; all I meant was 
that splitting them off might not be the best idea when they 
mostly exist only to be used by the D API and the C API should 
not be used unless you really want/have to.


 Usually, I use dynamic linking. I believe you would do the same 
 if
 optlink worked.


Maybe if optlink had worked with COFF when I started using D 
(back in D1 times), but right now, even if optlink would start 
working on win32 with COFF I'd still go with runtime loading as 
often as possible (Not counting D libraries with D classes in 
them, of course, only C API libraries). Mostly because this way 
you can start your programs with "rdmd program.d". No need to 
care at all about libraries: Getting a working compilation should 
imho be as simple as invoking the compiler with the source file 
that contains your main (or the root file of the library to 
create). Extra argument are necessary for specific configuration, 
but compiling should work with the least amount of extra argument 
as possible.


 The process is usually pretty simple. But the documentation is 
 lacking.
 There are several pieces floating around. I can try writing one 
 document
 and try to push it to the official web site.


That'd be great, thank you very much.


 Do as you like it. I do not think this is a bad approach.


I'll to that, then. I'm currently working on the D API, but I'll 
try to get the copied deimos compatible project started in the 
next couple of days.

- Moritz
Mar 18 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:

I don't know how this is done on Windows. On Linux you just set
the
LD_LIBRARY_PATH.


 
 My problem with that is that it's a) not encapsulated inside the
 program and b) OS dependent.


Very true. Do you have a solution for this? I can add it to ddl, if that
is the right place to put it.


Probably. There are very few users for such a use case.
But if you would use LLVM's plain C interface in D users who have
used
the API in C will feel home. I mean why expose the LLVM C API as
an
addition. It certainly does no harm.


 
 The C API will keep being exposed, of course; all I meant was that
 splitting them off might not be the best idea when they mostly exist
 only to be used by the D API and the C API should not be used unless
 you really want/have to.


And by splitting you mean in different files, right?


Usually, I use dynamic linking. I believe you would do the same if
optlink worked.


 
 Maybe if optlink had worked with COFF when I started using D (back
 in D1 times), but right now, even if optlink would start working on
 win32 with COFF I'd still go with runtime loading as often as
 possible (Not counting D libraries with D classes in them, of
 course, only C API libraries). Mostly because this way you can start
 your programs with "rdmd program.d". No need to care at all about
 libraries: Getting a working compilation should imho be as simple as
 invoking the compiler with the source file that contains your main
 (or the root file of the library to create). Extra argument are
 necessary for specific configuration, but compiling should work with
 the least amount of extra argument as possible.


I see.


The process is usually pretty simple. But the documentation is
lacking.
There are several pieces floating around. I can try writing one
document
and try to push it to the official web site.


 
 That'd be great, thank you very much.
 

Do as you like it. I do not think this is a bad approach.


 
 I'll to that, then. I'm currently working on the D API, but I'll try
 to get the copied deimos compatible project started in the next
 couple of days.


Sounds good. I'm going to write the documentation this evening.

Jens
Mar 18 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Monday, 18 March 2013 at 14:51:00 UTC, Jens Mueller wrote:

 Moritz Maxeiner wrote:

 On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:

I don't know how this is done on Windows. On Linux you just 
set
the
LD_LIBRARY_PATH.


 
 My problem with that is that it's a) not encapsulated inside 
 the
 program and b) OS dependent.


 Very true. Do you have a solution for this? I can add it to 
 ddl, if that
 is the right place to put it.


Well, it's more of a workaround than a solution:
The loading function (D) takes two arguments: The path and the 
file(name)
of the shared lib. If no filename is given, a ctfe generated one 
is used. The shared contructors calls this function with a path 
of "". That's about all I need for the LLVM shared lib; but if 
you're going to load a shared lib from a certain path which have 
dependencies in that certain path you'll need to additionally use 
SetDLLDirectoryA on windows (posix automatically looks in the 
directory a shared lib is loaded from afaik).
For source code you could look here:
https://github.com/Calrama/llvm-d/blob/master/llvm/c/all.d


Probably. There are very few users for such a use case.
But if you would use LLVM's plain C interface in D users who 
have
used
the API in C will feel home. I mean why expose the LLVM C API 
as
an
addition. It certainly does no harm.


 
 The C API will keep being exposed, of course; all I meant was 
 that
 splitting them off might not be the best idea when they mostly 
 exist
 only to be used by the D API and the C API should not be used 
 unless
 you really want/have to.


 And by splitting you mean in different files, right?


No, the C API will rest in llvm/c/... (as it does right now) 
while the D API will rest in llvm/d/... (so the C API will be 
usable without the D API and can be compiled and linked in 
without it).
With "splitting" I meant putting them in another project entirely 
like llvm-d and (deimos)llvm.


The process is usually pretty simple. But the documentation is
lacking.
There are several pieces floating around. I can try writing 
one
document
and try to push it to the official web site.


 
 That'd be great, thank you very much.
 

Do as you like it. I do not think this is a bad approach.


 
 I'll to that, then. I'm currently working on the D API, but 
 I'll try
 to get the copied deimos compatible project started in the next
 couple of days.


 Sounds good. I'm going to write the documentation this evening.


Awesome, thanks.
Mar 18 2013
next sibling parent Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Monday, 18 March 2013 at 14:51:00 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:

I'll to that, then. I'm currently working on the D API, but I'll
try
to get the copied deimos compatible project started in the next
couple of days.


Sounds good. I'm going to write the documentation this evening.


 
 Awesome, thanks.


Took longer than expected.
http://jkm.github.com/phobos/deimos.html
I will post this on digitalmars.D whether it is possible to add it to
the official site and to collect feedback.

Jens
Mar 20 2013
prev sibling parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Jens Mueller wrote:

 Moritz Maxeiner wrote:

 On Monday, 18 March 2013 at 14:51:00 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:

I'll to that, then. I'm currently working on the D API, but I'll
try
to get the copied deimos compatible project started in the next
couple of days.


Sounds good. I'm going to write the documentation this evening.


 
 Awesome, thanks.


 
 Took longer than expected.
 http://jkm.github.com/phobos/deimos.html
 I will post this on digitalmars.D whether it is possible to add it to
 the official site and to collect feedback.


Updated documentation
http://jkm.github.com/d-programming-language.org/deimos.html

Jens
Mar 22 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Friday, 22 March 2013 at 08:34:11 UTC, Jens Mueller wrote:

 Updated documentation
 http://jkm.github.com/d-programming-language.org/deimos.html


NIice, but conforming to the following would create too much work 
for me:

"For each file a proper module declaration has to be provided"

Since files in llvm-c appear and dissappear across different LLVM 
versions, you would eventually have to keep files around that are 
many LLVM versions old.

In llvm-d I use three files for the three main purposes: types, 
constants and functions. With ctfe having compatibility is 
trivial in that setup, with many more files that would be 
increasingly more work. Especially so since there are llvm-c 
headers with only few lines of code and creating an entire D 
module for that purpose seems wasteful and hurts maintainability 
or me.

"So far, each C header file was renamed to a D module. Next the 
contents of each module will be adjusted. In general following 
the advices from interfacing to c is recommended.

The D files should try to do as least modifications as possible 
to simplify updates of the C headers. This includes leaving 
comments intact."

Again, by cutting out everything but the actual code dealing with 
updates/new versions of LLVM becomes a lot easier. Dealing with 
the copyright issue can be done in compliance with LLVM's license 
by including LLVM's license note and copyright note in the 
documentation, in this case the README.md.

If I can make a deimos-llvm project the way I described it 
previously (by simply stripping out the shared lib code) I'll do 
it, because the changes would be few and maintaining both 
projects would be trivial; but if all deimos projects *have* to 
conform to these - in my case harmful - conventions, you might be 
better off with asking someone else to take over your deimos-llvm 
project.
Mar 22 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Friday, 22 March 2013 at 08:34:11 UTC, Jens Mueller wrote:

Updated documentation
http://jkm.github.com/d-programming-language.org/deimos.html


 
 NIice, but conforming to the following would create too much work
 for me:
 
 "For each file a proper module declaration has to be provided"
 
 Since files in llvm-c appear and dissappear across different LLVM
 versions, you would eventually have to keep files around that are
 many LLVM versions old.


Really. I thought the C API was expected to be more stable.
Also it's not yet clear how to manage multiple versions of the same
library in Deimos. But your feedback helps sharpening which issues
should be addressed by Deimos. Thanks.


 In llvm-d I use three files for the three main purposes: types,
 constants and functions. With ctfe having compatibility is trivial
 in that setup, with many more files that would be increasingly more
 work. Especially so since there are llvm-c headers with only few
 lines of code and creating an entire D module for that purpose seems
 wasteful and hurts maintainability or me.


I just did LLVM 3.1 and I get
$ find llvm -name '*.d' | xargs wc -l                                          
                           0
    45 llvm/c/initialization.d
  2607 llvm/c/core.d
    97 llvm/c/object.d
    63 llvm/c/linktimeoptimizer.d
   523 llvm/c/enhanceddisassembly.d
    56 llvm/c/analysis.d
   136 llvm/c/executionengine.d
    38 llvm/c/transforms/vectorize.d
    72 llvm/c/transforms/ipo.d
    82 llvm/c/transforms/passmanagerbuilder.d
   125 llvm/c/transforms/scalar.d
   294 llvm/c/lto.d
    47 llvm/c/bitwriter.d
   166 llvm/c/disassembler.d
   115 llvm/c/targetmachine.d
    67 llvm/c/bitreader.d
   286 llvm/c/target.d
  4819 total

I assume you are referring to those files that are under 100 lines.


 "So far, each C header file was renamed to a D module. Next the
 contents of each module will be adjusted. In general following the
 advices from interfacing to c is recommended.
 
 The D files should try to do as least modifications as possible to
 simplify updates of the C headers. This includes leaving comments
 intact."
 
 Again, by cutting out everything but the actual code dealing with
 updates/new versions of LLVM becomes a lot easier. Dealing with the
 copyright issue can be done in compliance with LLVM's license by
 including LLVM's license note and copyright note in the
 documentation, in this case the README.md.
 
 If I can make a deimos-llvm project the way I described it
 previously (by simply stripping out the shared lib code) I'll do it,
 because the changes would be few and maintaining both projects would
 be trivial; but if all deimos projects *have* to conform to these -
 in my case harmful - conventions, you might be better off with
 asking someone else to take over your deimos-llvm project.


I'm not the one to decide this. Probably this is the reason why I'm
writing the documentation because there is de facto no clear decision on
what needs to be done to warrant Deimos inclusion.
Assuming I can spend some time on deimos-llvm would you switch? I mean I
have 3.1. Adding 3.2 is possible. But I would need you to use it and
check that it works for your higher level API.

Jens
Mar 22 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Friday, 22 March 2013 at 17:16:26 UTC, Jens Mueller wrote:

 Moritz Maxeiner wrote:

 On Friday, 22 March 2013 at 08:34:11 UTC, Jens Mueller wrote:

Updated documentation
http://jkm.github.com/d-programming-language.org/deimos.html


 
 NIice, but conforming to the following would create too much 
 work
 for me:
 
 "For each file a proper module declaration has to be provided"
 
 Since files in llvm-c appear and dissappear across different 
 LLVM
 versions, you would eventually have to keep files around that 
 are
 many LLVM versions old.


 Really. I thought the C API was expected to be more stable.
 Also it's not yet clear how to manage multiple versions of the 
 same
 library in Deimos. But your feedback helps sharpening which 
 issues
 should be addressed by Deimos. Thanks.


Afaict in the rang from 3.1 to 3.3svn it has happened two times:
- The "Enhanced Disassembly" header exists in 3.2, but has 
vanished in the trunk.
- The "Linker" header has been added in 3.2 and it is also an 
example of the short files. It contains only a single function 
"LLVMLinkModules" and a single enum for one of the that 
function's parameters.


 Assuming I can spend some time on deimos-llvm would you switch? 
 I mean I
 have 3.1. Adding 3.2 is possible. But I would need you to use 
 it and
 check that it works for your higher level API.


I could add support for it via a version flag. E.g. set 
-version=DEIMOS_LLVM or something similar. That way someone using 
llvm-d can choose to either use the included C bindings or use 
your deimos compatible ones. My only concern would be that the 
LLVM C function signatures would have to be translated in the 
same, or at least a compatible way in both my C bindings and the 
deimos ones, so using either of them would just be a matter of 
which import statement to use.
Since I've translated them pretty much strictly according to the 
"interfacing with c" guideline that's probably the case already, 
but I'd have to try it out first.
Mar 22 2013
next sibling parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
A couple of more things I forgot to mention:

- You will need to additionally add bindings for all the 
LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmP
inter,Disassembler} 
functions. They reside inside the target libraries and not inside 
llvm-c, which is why only translating the C API won't get you 
them. The LLVMInitializeNativeTarget function , which is 
necessary to use LLVM for jitting - otherwise you'll only have 
access to interpreting, which is horribly slow - uses them, but 
it is a static inline function and as such does not get exposed 
because LLVM builds with the option to hide all inline functions. 
That is not a problem for C, as this function gets defined in the 
header, but for D it is a big problem, because you cannot access 
LLVMInitializeNativeTarget. You need to recreate it in D by using 
all the functions it references. Of course, you'll also have to 
accomodate the fact that these referenced functions may or may 
not be compiled in depending on which target where selected when 
compiling LLVM.
I have done so in llvm.c.functions of llvm-d so if you want you 
could use that as a reference.
- When supporting multiple LLVM versions, we need to use the same 
method. I currently use a version flag, specifically 
"-version=LLVM_X_Y" for LLVM X.Y release (including the trunk) 
and set two variables, a string "LLVM_VersionString" (for some C 
bindings internal stuff) and a float "LLVM_Version" (for static 
ifs). The relevant module is llvm.c.versions.
- It would be nice if deimos-llvm had a module that publically 
imports all other modules, e.g. "all" or "llvm".
Mar 22 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 A couple of more things I forgot to mention:
 
 - You will need to additionally add bindings for all the
LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmPrinter,Disassembler}
 functions. They reside inside the target libraries and not inside
 llvm-c, which is why only translating the C API won't get you them.
 The LLVMInitializeNativeTarget function , which is necessary to use
 LLVM for jitting - otherwise you'll only have access to
 interpreting, which is horribly slow - uses them, but it is a static
 inline function and as such does not get exposed because LLVM builds
 with the option to hide all inline functions. That is not a problem
 for C, as this function gets defined in the header, but for D it is
 a big problem, because you cannot access LLVMInitializeNativeTarget.
 You need to recreate it in D by using all the functions it
 references. Of course, you'll also have to accomodate the fact that
 these referenced functions may or may not be compiled in depending
 on which target where selected when compiling LLVM.


Do you happen to know why these functions are not declared in the header
files. I mean when using the C interface in C I wouldn't even know that
these functions exist.


 I have done so in llvm.c.functions of llvm-d so if you want you
 could use that as a reference.
 - When supporting multiple LLVM versions, we need to use the same
 method. I currently use a version flag, specifically


Yes. Handling different version of the same library hasn't been
addressed by Deimos.


 "-version=LLVM_X_Y" for LLVM X.Y release (including the trunk) and
 set two variables, a string "LLVM_VersionString" (for some C
 bindings internal stuff) and a float "LLVM_Version" (for static
 ifs). The relevant module is llvm.c.versions.


That's an option.


 - It would be nice if deimos-llvm had a module that publically
 imports all other modules, e.g. "all" or "llvm".


I think I leave this up to a higher layer API.
Deimos should provide an API that resembles the C api whenever possible.

Jens
Mar 23 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Saturday, 23 March 2013 at 09:30:07 UTC, Jens Mueller wrote:

 Moritz Maxeiner wrote:

 A couple of more things I forgot to mention:
 
 - You will need to additionally add bindings for all the 
 LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmPrinter,Disassembler}
 functions. They reside inside the target libraries and not 
 inside
 llvm-c, which is why only translating the C API won't get you 
 them.
 The LLVMInitializeNativeTarget function , which is necessary 
 to use
 LLVM for jitting - otherwise you'll only have access to
 interpreting, which is horribly slow - uses them, but it is a 
 static
 inline function and as such does not get exposed because LLVM 
 builds
 with the option to hide all inline functions. That is not a 
 problem
 for C, as this function gets defined in the header, but for D 
 it is
 a big problem, because you cannot access 
 LLVMInitializeNativeTarget.
 You need to recreate it in D by using all the functions it
 references. Of course, you'll also have to accomodate the fact 
 that
 these referenced functions may or may not be compiled in 
 depending
 on which target where selected when compiling LLVM.


 Do you happen to know why these functions are not declared in 
 the header
 files. I mean when using the C interface in C I wouldn't even 
 know that
 these functions exist.


I suspect it is because of LLVM's internal structure. Each 
supported target is presented by its own directory and they are 
afaict mutually independent from one another, "so their 
initialiation routines should be part of them themselves" I would 
guess, but I don't know, sorry.
Also, when using the C interface in C there is little need to 
know them because they get encapsulated in 
LLVMInitializeAllTargets and LLVMInitializeNativeTarget.


 I think I leave this up to a higher layer API.
 Deimos should provide an API that resembles the C api whenever 
 possible.


Okay, I'll put one into llvm-d one then.

-- Moritz
Mar 23 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Saturday, 23 March 2013 at 09:30:07 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

A couple of more things I forgot to mention:

- You will need to additionally add bindings for all the
LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmPrinter,Disassembler}
functions. They reside inside the target libraries and not
inside
llvm-c, which is why only translating the C API won't get you
them.
The LLVMInitializeNativeTarget function , which is necessary to
use
LLVM for jitting - otherwise you'll only have access to
interpreting, which is horribly slow - uses them, but it is a
static
inline function and as such does not get exposed because LLVM
builds
with the option to hide all inline functions. That is not a
problem
for C, as this function gets defined in the header, but for D it
is
a big problem, because you cannot access
LLVMInitializeNativeTarget.
You need to recreate it in D by using all the functions it
references. Of course, you'll also have to accomodate the fact
that
these referenced functions may or may not be compiled in
depending
on which target where selected when compiling LLVM.


Do you happen to know why these functions are not declared in the
header
files. I mean when using the C interface in C I wouldn't even know
that
these functions exist.


 
 I suspect it is because of LLVM's internal structure. Each supported
 target is presented by its own directory and they are afaict
 mutually independent from one another, "so their initialiation
 routines should be part of them themselves" I would guess, but I
 don't know, sorry.
 Also, when using the C interface in C there is little need to know
 them because they get encapsulated in LLVMInitializeAllTargets and
 LLVMInitializeNativeTarget.


Not sure whether this fixes the problem completely. Can you check
https://github.com/jkm/deimos-llvm/blob/master/deimos/llvm/c/target.d#L157
I came up with this solution to replace the macros in the C binding.

Jens
Mar 23 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Saturday, 23 March 2013 at 10:31:30 UTC, Jens Mueller wrote:

 Moritz Maxeiner wrote:

 On Saturday, 23 March 2013 at 09:30:07 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

A couple of more things I forgot to mention:

- You will need to additionally add bindings for all the 
LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmPrinter,Disassembler}
functions. They reside inside the target libraries and not
inside
llvm-c, which is why only translating the C API won't get you
them.
The LLVMInitializeNativeTarget function , which is necessary 
to
use
LLVM for jitting - otherwise you'll only have access to
interpreting, which is horribly slow - uses them, but it is a
static
inline function and as such does not get exposed because LLVM
builds
with the option to hide all inline functions. That is not a
problem
for C, as this function gets defined in the header, but for 
D it
is
a big problem, because you cannot access
LLVMInitializeNativeTarget.
You need to recreate it in D by using all the functions it
references. Of course, you'll also have to accomodate the 
fact
that
these referenced functions may or may not be compiled in
depending
on which target where selected when compiling LLVM.


Do you happen to know why these functions are not declared in 
the
header
files. I mean when using the C interface in C I wouldn't even 
know
that
these functions exist.


 
 I suspect it is because of LLVM's internal structure. Each 
 supported
 target is presented by its own directory and they are afaict
 mutually independent from one another, "so their initialiation
 routines should be part of them themselves" I would guess, but 
 I
 don't know, sorry.
 Also, when using the C interface in C there is little need to 
 know
 them because they get encapsulated in LLVMInitializeAllTargets 
 and
 LLVMInitializeNativeTarget.


 Not sure whether this fixes the problem completely. Can you 
 check
 https://github.com/jkm/deimos-llvm/blob/master/deimos/llvm/c/target.d#L157
 I came up with this solution to replace the macros in the C 
 binding.


It looks mostly okay to me with one problem: Afaict the code 
enforces the presence of the initialization routines of all 
targets, e.g. if one target is missing LLVMInitializeAllTargets 
will not link, as there are undefined references for that missing 
target, but LLVM may or may not be compiled with that target so 
you cannot enforce its presence in the bindings. For runtime 
loading the solution I used was to check the function pointer for 
null; for linking you have this problem: When using linking, 
knowing which targets are available happens at link time (when 
the LLVM libraries are linked in), which means you cannot use any 
compile time tricks for automatic detection of which targets are 
available.
The only solution for that problem I can think of would be to use 
runtime reflection and check at runtime for each initialiation 
routine if it is a callable function, but afaik D only has 
compile time reflection.

- Moritz

PS: All your recent posts seem to create new threads, I don't 
know why, though, just mentioning it because some people not 
interested in this might get annoyed.
Mar 23 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Saturday, 23 March 2013 at 10:31:30 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Saturday, 23 March 2013 at 09:30:07 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

A couple of more things I forgot to mention:

- You will need to additionally add bindings for all the
>>LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmPrinter,Disassembler}
functions. They reside inside the target libraries and not
inside
llvm-c, which is why only translating the C API won't get you
them.
The LLVMInitializeNativeTarget function , which is necessary
to
use
LLVM for jitting - otherwise you'll only have access to
interpreting, which is horribly slow - uses them, but it is a
static
inline function and as such does not get exposed because LLVM
builds
with the option to hide all inline functions. That is not a
problem
for C, as this function gets defined in the header, but for
D it
is
a big problem, because you cannot access
LLVMInitializeNativeTarget.
You need to recreate it in D by using all the functions it
references. Of course, you'll also have to accomodate the
fact
that
these referenced functions may or may not be compiled in
depending
on which target where selected when compiling LLVM.


Do you happen to know why these functions are not declared in
the
header
files. I mean when using the C interface in C I wouldn't even
know
that
these functions exist.


I suspect it is because of LLVM's internal structure. Each
supported
target is presented by its own directory and they are afaict
mutually independent from one another, "so their initialiation
routines should be part of them themselves" I would guess, but I
don't know, sorry.
Also, when using the C interface in C there is little need to
know
them because they get encapsulated in LLVMInitializeAllTargets
and
LLVMInitializeNativeTarget.


Not sure whether this fixes the problem completely. Can you check
https://github.com/jkm/deimos-llvm/blob/master/deimos/llvm/c/target.d#L157
I came up with this solution to replace the macros in the C
binding.


 
 It looks mostly okay to me with one problem: Afaict the code
 enforces the presence of the initialization routines of all targets,
 e.g. if one target is missing LLVMInitializeAllTargets will not
 link, as there are undefined references for that missing target, but
 LLVM may or may not be compiled with that target so you cannot
 enforce its presence in the bindings. For runtime loading the
 solution I used was to check the function pointer for null; for
 linking you have this problem: When using linking, knowing which
 targets are available happens at link time (when the LLVM libraries
 are linked in), which means you cannot use any compile time tricks
 for automatic detection of which targets are available.
 The only solution for that problem I can think of would be to use
 runtime reflection and check at runtime for each initialiation
 routine if it is a callable function, but afaik D only has compile
 time reflection.


I wonder how they do it in C. Don't you have to set a macro? I could add
some version identifiers.


 PS: All your recent posts seem to create new threads, I don't know
 why, though, just mentioning it because some people not interested
 in this might get annoyed.


Don't know either. I'm using the mail interface.

Jens
Mar 23 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Saturday, 23 March 2013 at 16:37:35 UTC, Jens Mueller wrote:

 Moritz Maxeiner wrote:

 On Saturday, 23 March 2013 at 10:31:30 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Saturday, 23 March 2013 at 09:30:07 UTC, Jens Mueller 
wrote:

Moritz Maxeiner wrote:

A couple of more things I forgot to mention:

- You will need to additionally add bindings for all the

LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmPrinter,Disassembler}




functions. They reside inside the target libraries and not
inside
llvm-c, which is why only translating the C API won't get 
you
them.
The LLVMInitializeNativeTarget function , which is 
necessary
to
use
LLVM for jitting - otherwise you'll only have access to
interpreting, which is horribly slow - uses them, but it 
is a
static
inline function and as such does not get exposed because 
LLVM
builds
with the option to hide all inline functions. That is not a
problem
for C, as this function gets defined in the header, but for
D it
is
a big problem, because you cannot access
LLVMInitializeNativeTarget.
You need to recreate it in D by using all the functions it
references. Of course, you'll also have to accomodate the
fact
that
these referenced functions may or may not be compiled in
depending
on which target where selected when compiling LLVM.


Do you happen to know why these functions are not declared 
in
the
header
files. I mean when using the C interface in C I wouldn't 
even
know
that
these functions exist.


I suspect it is because of LLVM's internal structure. Each
supported
target is presented by its own directory and they are afaict
mutually independent from one another, "so their 
initialiation
routines should be part of them themselves" I would guess, 
but I
don't know, sorry.
Also, when using the C interface in C there is little need to
know
them because they get encapsulated in 
LLVMInitializeAllTargets
and
LLVMInitializeNativeTarget.


Not sure whether this fixes the problem completely. Can you 
check
https://github.com/jkm/deimos-llvm/blob/master/deimos/llvm/c/target.d#L157
I came up with this solution to replace the macros in the C
binding.


 
 It looks mostly okay to me with one problem: Afaict the code
 enforces the presence of the initialization routines of all 
 targets,
 e.g. if one target is missing LLVMInitializeAllTargets will not
 link, as there are undefined references for that missing 
 target, but
 LLVM may or may not be compiled with that target so you cannot
 enforce its presence in the bindings. For runtime loading the
 solution I used was to check the function pointer for null; for
 linking you have this problem: When using linking, knowing 
 which
 targets are available happens at link time (when the LLVM 
 libraries
 are linked in), which means you cannot use any compile time 
 tricks
 for automatic detection of which targets are available.
 The only solution for that problem I can think of would be to 
 use
 runtime reflection and check at runtime for each initialiation
 routine if it is a callable function, but afaik D only has 
 compile
 time reflection.


 I wonder how they do it in C. Don't you have to set a macro?


Afaict they rely on the fact that when you install llvm on your 
system you get the {/usr/include/}llvm/Config/Targets.def file, 
in which is set what targets LLVM was compiled with (it gets 
generated at LLVM compile-time). Then the Target.h, in which the 
LLVMInitializeAllTargets function rests, includes that file and 
does some of that macro-voodoo that makes C/C++ so "lovable" to 
only create calls for the targets enabled in the Targets.def file 
when LLVMInitalizeAllTargets gets inlined. Of course, that 
solution isn't viable because afaik you cannot access filesystem 
IO functions in D CTFE, meaning the Targets.def file is useless 
to you. <rant>Hooray for the C/C++ preprocessor, may it die, die, 
die!</rant>


  I could add
 some version identifiers.


That would be one solution, but you'd have to create one version 
identifier for each target.

- Moritz
Mar 23 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Saturday, 23 March 2013 at 16:37:35 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Saturday, 23 March 2013 at 10:31:30 UTC, Jens Mueller wrote:

It looks mostly okay to me with one problem: Afaict the code
enforces the presence of the initialization routines of all
targets,
e.g. if one target is missing LLVMInitializeAllTargets will not
link, as there are undefined references for that missing target,
but
LLVM may or may not be compiled with that target so you cannot
enforce its presence in the bindings. For runtime loading the
solution I used was to check the function pointer for null; for
linking you have this problem: When using linking, knowing which
targets are available happens at link time (when the LLVM
libraries
are linked in), which means you cannot use any compile time
tricks
for automatic detection of which targets are available.
The only solution for that problem I can think of would be to
use
runtime reflection and check at runtime for each initialiation
routine if it is a callable function, but afaik D only has
compile
time reflection.


I wonder how they do it in C. Don't you have to set a macro?


 
 Afaict they rely on the fact that when you install llvm on your
 system you get the {/usr/include/}llvm/Config/Targets.def file, in
 which is set what targets LLVM was compiled with (it gets generated
 at LLVM compile-time). Then the Target.h, in which the
 LLVMInitializeAllTargets function rests, includes that file and does
 some of that macro-voodoo that makes C/C++ so "lovable" to only
 create calls for the targets enabled in the Targets.def file when
 LLVMInitalizeAllTargets gets inlined. Of course, that solution isn't
 viable because afaik you cannot access filesystem IO functions in D
 CTFE, meaning the Targets.def file is useless to you. <rant>Hooray
 for the C/C++ preprocessor, may it die, die, die!</rant>


If I knew the path to the Targets.def file at compile I could load its
contents and use it.
http://dlang.org/expression.html#ImportExpression
Still have to find out how to get the path.

Jens
Mar 23 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Saturday, 23 March 2013 at 21:24:50 UTC, Jens Mueller wrote:

 Moritz Maxeiner wrote:

 On Saturday, 23 March 2013 at 16:37:35 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Saturday, 23 March 2013 at 10:31:30 UTC, Jens Mueller 
wrote:

It looks mostly okay to me with one problem: Afaict the code
enforces the presence of the initialization routines of all
targets,
e.g. if one target is missing LLVMInitializeAllTargets will 
not
link, as there are undefined references for that missing 
target,
but
LLVM may or may not be compiled with that target so you 
cannot
enforce its presence in the bindings. For runtime loading the
solution I used was to check the function pointer for null; 
for
linking you have this problem: When using linking, knowing 
which
targets are available happens at link time (when the LLVM
libraries
are linked in), which means you cannot use any compile time
tricks
for automatic detection of which targets are available.
The only solution for that problem I can think of would be to
use
runtime reflection and check at runtime for each 
initialiation
routine if it is a callable function, but afaik D only has
compile
time reflection.


I wonder how they do it in C. Don't you have to set a macro?


 
 Afaict they rely on the fact that when you install llvm on your
 system you get the {/usr/include/}llvm/Config/Targets.def 
 file, in
 which is set what targets LLVM was compiled with (it gets 
 generated
 at LLVM compile-time). Then the Target.h, in which the
 LLVMInitializeAllTargets function rests, includes that file 
 and does
 some of that macro-voodoo that makes C/C++ so "lovable" to only
 create calls for the targets enabled in the Targets.def file 
 when
 LLVMInitalizeAllTargets gets inlined. Of course, that solution 
 isn't
 viable because afaik you cannot access filesystem IO functions 
 in D
 CTFE, meaning the Targets.def file is useless to you. 
 <rant>Hooray
 for the C/C++ preprocessor, may it die, die, die!</rant>


 If I knew the path to the Targets.def file at compile I could 
 load its
 contents and use it.
 http://dlang.org/expression.html#ImportExpression
 Still have to find out how to get the path.

 Jens


Oh, I did not know that was possible, thanks for the link!
Can't really help you with the path, though. On Linux it's 
probably either /usr/include/llvm/Config/Targets.def or 
/usr/local/include/llvm/Config/Targets.def, but some people might 
install it on /opt/llvm or other custom paths. You also have a 
problem when there's only a shared lib available for LLVM and no 
headers.. No idea about OSX though. And on Windows you'll 
probably only deal with a DLL and not have the headers available 
anyway - at least I do (I cross compile the LLVM dll on linux 
with a mingw64 gcc toolchain, so no headers on windows for me).

Moritz
Mar 23 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Saturday, 23 March 2013 at 21:24:50 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Saturday, 23 March 2013 at 16:37:35 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Saturday, 23 March 2013 at 10:31:30 UTC, Jens Mueller
wrote:

It looks mostly okay to me with one problem: Afaict the code
enforces the presence of the initialization routines of all
targets,
e.g. if one target is missing LLVMInitializeAllTargets will
not
link, as there are undefined references for that missing
target,
but
LLVM may or may not be compiled with that target so you
cannot
enforce its presence in the bindings. For runtime loading the
solution I used was to check the function pointer for null;
for
linking you have this problem: When using linking, knowing
which
targets are available happens at link time (when the LLVM
libraries
are linked in), which means you cannot use any compile time
tricks
for automatic detection of which targets are available.
The only solution for that problem I can think of would be to
use
runtime reflection and check at runtime for each
initialiation
routine if it is a callable function, but afaik D only has
compile
time reflection.


I wonder how they do it in C. Don't you have to set a macro?


Afaict they rely on the fact that when you install llvm on your
system you get the {/usr/include/}llvm/Config/Targets.def file,
in
which is set what targets LLVM was compiled with (it gets
generated
at LLVM compile-time). Then the Target.h, in which the
LLVMInitializeAllTargets function rests, includes that file and
does
some of that macro-voodoo that makes C/C++ so "lovable" to only
create calls for the targets enabled in the Targets.def file
when
LLVMInitalizeAllTargets gets inlined. Of course, that solution
isn't
viable because afaik you cannot access filesystem IO functions
in D
CTFE, meaning the Targets.def file is useless to you.
<rant>Hooray
for the C/C++ preprocessor, may it die, die, die!</rant>


If I knew the path to the Targets.def file at compile I could load
its
contents and use it.
http://dlang.org/expression.html#ImportExpression
Still have to find out how to get the path.

Jens


 
 Oh, I did not know that was possible, thanks for the link!
 Can't really help you with the path, though. On Linux it's probably
 either /usr/include/llvm/Config/Targets.def or
 /usr/local/include/llvm/Config/Targets.def, but some people might
 install it on /opt/llvm or other custom paths. You also have a
 problem when there's only a shared lib available for LLVM and no
 headers.. No idea about OSX though. And on Windows you'll probably
 only deal with a DLL and not have the headers available anyway - at
 least I do (I cross compile the LLVM dll on linux with a mingw64 gcc
 toolchain, so no headers on windows for me).


I think passing the path -J should work. But I don't know what to do
when there is no Targets.def.

Jens
Mar 23 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
 I think passing the path -J should work. But I don't know what 
 to do
 when there is no Targets.def.

 Jens


Off topic, but can you please ensure not to split the thread like 
this ?
Mar 23 2013
parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
deadalnix wrote:

I think passing the path -J should work. But I don't know what to
do
when there is no Targets.def.

Jens


 
 Off topic, but can you please ensure not to split the thread like
 this ?


How do I ensure this?
I'm using the mailman interface.
There are no such problems with digitalsmars.D using the same setup.

Jens
Mar 25 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 25 March 2013 at 08:54:04 UTC, Jens Mueller wrote:

 deadalnix wrote:

I think passing the path -J should work. But I don't know 
what to
do
when there is no Targets.def.

Jens


 
 Off topic, but can you please ensure not to split the thread 
 like
 this ?


 How do I ensure this?
 I'm using the mailman interface.
 There are no such problems with digitalsmars.D using the same 
 setup.

 Jens


I don't know. I used to use thunderbird, which worked fine, and 
now mostly the web forum ( http://forum.dlang.org/ which does 
work great as well ).
Mar 27 2013
prev sibling parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Moritz Maxeiner wrote:

 On Friday, 22 March 2013 at 17:16:26 UTC, Jens Mueller wrote:

Moritz Maxeiner wrote:

On Friday, 22 March 2013 at 08:34:11 UTC, Jens Mueller wrote:

Updated documentation
http://jkm.github.com/d-programming-language.org/deimos.html


NIice, but conforming to the following would create too much
work
for me:

"For each file a proper module declaration has to be provided"

Since files in llvm-c appear and dissappear across different
LLVM
versions, you would eventually have to keep files around that
are
many LLVM versions old.


Really. I thought the C API was expected to be more stable.
Also it's not yet clear how to manage multiple versions of the
same
library in Deimos. But your feedback helps sharpening which issues
should be addressed by Deimos. Thanks.


 
 Afaict in the rang from 3.1 to 3.3svn it has happened two times:
 - The "Enhanced Disassembly" header exists in 3.2, but has vanished
 in the trunk.
 - The "Linker" header has been added in 3.2 and it is also an
 example of the short files. It contains only a single function
 "LLVMLinkModules" and a single enum for one of the that function's
 parameters.


I see.
I need to think about to handle the different versions.


Assuming I can spend some time on deimos-llvm would you switch? I
mean I
have 3.1. Adding 3.2 is possible. But I would need you to use it
and
check that it works for your higher level API.


 
 I could add support for it via a version flag. E.g. set
 -version=DEIMOS_LLVM or something similar. That way someone using
 llvm-d can choose to either use the included C bindings or use your
 deimos compatible ones. My only concern would be that the LLVM C
 function signatures would have to be translated in the same, or at
 least a compatible way in both my C bindings and the deimos ones, so
 using either of them would just be a matter of which import
 statement to use.
 Since I've translated them pretty much strictly according to the
 "interfacing with c" guideline that's probably the case already, but
 I'd have to try it out first.


I would like that. Probably the signatures are already the same. There
is not much freedom when translating them.

Jens
Mar 23 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Saturday, 23 March 2013 at 08:53:47 UTC, Jens Mueller wrote:

 Moritz Maxeiner wrote:

 I could add support for it via a version flag. E.g. set
 -version=DEIMOS_LLVM or something similar. That way someone 
 using
 llvm-d can choose to either use the included C bindings or use 
 your
 deimos compatible ones. My only concern would be that the LLVM 
 C
 function signatures would have to be translated in the same, 
 or at
 least a compatible way in both my C bindings and the deimos 
 ones, so
 using either of them would just be a matter of which import
 statement to use.
 Since I've translated them pretty much strictly according to 
 the
 "interfacing with c" guideline that's probably the case 
 already, but
 I'd have to try it out first.


 I would like that. Probably the signatures are already the 
 same. There
 is not much freedom when translating them.


I've updated llvm-d to do just that. When compiling with 
-version=DEIMOS_LLVM the D API will import the deimos-llvm 
bindings instead of llvm-d's own ones. I've not yet tested it 
fully, but it should work. One thing, though: Your
https://github.com/jkm/deimos-llvm/blob/master/deimos/llvm/c/executionengine.d
should also have executionengine as the module name, not 
enhanceddisassembly.

-Moritz
Mar 23 2013
parent "Moritz Maxeiner" <moritz ucworks.org> writes:
Also, you named the enums, meaing "LLVMCCallConv", like in C, 
doesn't work and one has to use "LLVMCallConv.LLVMCCallConv". 
Same for the other enums.
Mar 23 2013
prev sibling parent reply "Chris Cain" <clcain uncg.edu> writes:
On Friday, 15 March 2013 at 17:40:36 UTC, Moritz Maxeiner wrote:

 Hi, I would like to announce llvm-d, which provides LLVM 
 bindings for D.


Greetings,

I hope you don't mind a bit of a support request, but I'm having 
difficulties getting it to work:

https://gist.github.com/Zshazz/c7dbd6eee0b6b242252b

The code is from 
http://npcontemplation.blogspot.com/2008/06/secret-of-llvm-c-bindings.html 
but with slight updates (and obviously fixing syntax to work in 
D).

This is running on Manjaro Linux (a derivative of Archlinux) and 
using DMD v2.062.

Thanks for your time.
Mar 22 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Saturday, 23 March 2013 at 04:34:53 UTC, Chris Cain wrote:

 On Friday, 15 March 2013 at 17:40:36 UTC, Moritz Maxeiner wrote:

 Hi, I would like to announce llvm-d, which provides LLVM 
 bindings for D.


 Greetings,

 I hope you don't mind a bit of a support request, but I'm 
 having difficulties getting it to work:

 https://gist.github.com/Zshazz/c7dbd6eee0b6b242252b

 The code is from 
 http://npcontemplation.blogspot.com/2008/06/secret-of-llvm-c-bindings.html 
 but with slight updates (and obviously fixing syntax to work in 
 D).

 This is running on Manjaro Linux (a derivative of Archlinux) 
 and using DMD v2.062.

 Thanks for your time.


No problem at all. There is an example quoted in the README and 
how to compile it, so without further information form your side 
I don't know what the problem is. To get the example from the 
README working:
- Download/Clone the github repo into a folder (let's say llvm-d)
- cd into that folder
- execute rdmd -L-ldl sample/multithreaded.d

The need for the -dl flag is only for POSIX platforms and will be 
gone with the next commit as I have included a lib pragma.

If the above doesn't help could you please tell me what exactly 
you have done and at which point you have the problems?

-- Moritz
Mar 23 2013
next sibling parent reply 1100110 <0b1100110 gmail.com> writes:
On 03/23/2013 05:01 AM, Moritz Maxeiner wrote:
[snip]

 No problem at all. There is an example quoted in the README and how to
 compile it, so without further information form your side I don't know
 what the problem is. To get the example from the README working:
 - Download/Clone the github repo into a folder (let's say llvm-d)
 - cd into that folder
 - execute rdmd -L-ldl sample/multithreaded.d

 The need for the -dl flag is only for POSIX platforms and will be gone
 with the next commit as I have included a lib pragma.

 If the above doesn't help could you please tell me what exactly you have
 done and at which point you have the problems?

 -- Moritz


...And its fixed.
TY
Mar 23 2013
parent "Moritz Maxeiner" <moritz ucworks.org> writes:
On Saturday, 23 March 2013 at 14:42:06 UTC, 1100110 wrote:

 On 03/23/2013 05:01 AM, Moritz Maxeiner wrote:
 [snip]

 No problem at all. There is an example quoted in the README 
 and how to
 compile it, so without further information form your side I 
 don't know
 what the problem is. To get the example from the README 
 working:
 - Download/Clone the github repo into a folder (let's say 
 llvm-d)
 - cd into that folder
 - execute rdmd -L-ldl sample/multithreaded.d

 The need for the -dl flag is only for POSIX platforms and will 
 be gone
 with the next commit as I have included a lib pragma.

 If the above doesn't help could you please tell me what 
 exactly you have
 done and at which point you have the problems?

 -- Moritz


 ...And its fixed.
 TY


No problem, I discovered that using "struct XYZ;" instead of 
"struct XYZ {};" will lead to forward referencing errors (which I 
did not expect) and changed all the structs accordingly. I'll 
have another sample up later today to show a more complex 
use-case.
Mar 23 2013
prev sibling parent reply "Chris Cain" <clcain uncg.edu> writes:
On Saturday, 23 March 2013 at 10:01:19 UTC, Moritz Maxeiner wrote:

 No problem at all. There is an example quoted in the README and 
 how to compile it, so without further information form your 
 side I don't know what the problem is. To get the example from 
 the README working:
 - Download/Clone the github repo into a folder (let's say 
 llvm-d)
 - cd into that folder
 - execute rdmd -L-ldl sample/multithreaded.d

 The need for the -dl flag is only for POSIX platforms and will 
 be gone with the next commit as I have included a lib pragma.

 If the above doesn't help could you please tell me what exactly 
 you have done and at which point you have the problems?

 -- Moritz


Thanks so much.

I had dub upgrade the llvm-d package and now I have different 
problems. That said, I do have a working example (as long as rdmd 
is used).

I have updated the gist (with repro steps): 
https://gist.github.com/Zshazz/c7dbd6eee0b6b242252b

I'm running LLVM 3.2, so the example given in the README doesn't 
work. But the code in the gist works as long as you run 'dub 
--rdmd', so that's some progress! Not really sure about the 
linking problems without rdmd. This is sufficient to start some 
work with it to really try some things out, though.

I'll keep messing around with it. It seems like you already know 
that JIT doesn't work because we need InitializeNativeTarget.

That all said, the tutorials, instructions, and documentation for 
the C API of LLVM is pretty sparse. The best I've seen to figure 
out how things work is to go through 
http://llvm.org/doxygen/modules.html and piece together (very 
slowly) how something might work. I'm also having to cross 
reference tutorials made for the C++ API and figure out how to 
make it work in C by reading the raw source code to get the 
equivalent calls. Fortunately I did find that one post that gave 
a close-to-complete example in C. Is there any better way to do 
this at this point?

Thanks again.
Mar 23 2013
next sibling parent "Moritz Maxeiner" <moritz ucworks.org> writes:
On Saturday, 23 March 2013 at 19:57:52 UTC, Chris Cain wrote:

 On Saturday, 23 March 2013 at 10:01:19 UTC, Moritz Maxeiner 
 wrote:

 No problem at all. There is an example quoted in the README 
 and how to compile it, so without further information form 
 your side I don't know what the problem is. To get the example 
 from the README working:
 - Download/Clone the github repo into a folder (let's say 
 llvm-d)
 - cd into that folder
 - execute rdmd -L-ldl sample/multithreaded.d

 The need for the -dl flag is only for POSIX platforms and will 
 be gone with the next commit as I have included a lib pragma.

 If the above doesn't help could you please tell me what 
 exactly you have done and at which point you have the problems?

 -- Moritz


 Thanks so much.

 I had dub upgrade the llvm-d package and now I have different 
 problems. That said, I do have a working example (as long as 
 rdmd is used).

 I have updated the gist (with repro steps): 
 https://gist.github.com/Zshazz/c7dbd6eee0b6b242252b


Regarding the two warnings:
The second one for line 346 has been fixed.
I have no idea how you could get the first one as the statement 
is - at least according to my logic - reachable: It should be 
reached if the "isSizedDerivedType" is used on a StructType 
instance. I'll look into that shortly.

Other than that, the link time errors appear because the dl 
library (which allows for loading of dynamic libraries at 
runtime) did not get linked in. That should not be possible as 
llvm.util.shlib contains a lib pragma that ensures the dl library 
does get linked in.

Also, you're giving d string to c function which expect c 
strings. Not a good idea (c strings are \0 terminated, d strings 
aren't). Use std.string.toStringz (which allocates GC memory, 
though and the c strings may dissappear while LLVM is using them 
as D collects them, because the GC doesn't know about any use on 
the C side) or use llvm.util.memory.toCString (which does the 
same thing as toStringz, only it doesn't allocate GC memory, but 
"unmanaged" memory).

And you don't need the two lines with the lib directory if you 
have LLVM installed. That exists in the README sample only to 
show how it is done for people who wish to ship the LLLVM library 
together with their program inside a subfolder, I've removed them 
from the README example since it seems to cause confusion and 
also added the fibonacci example to the README to showcase 
something more complex.


 I'm running LLVM 3.2, so the example given in the README 
 doesn't work. But the code in the gist works as long as you run 
 'dub --rdmd', so that's some progress! Not really sure about 
 the linking problems without rdmd. This is sufficient to start 
 some work with it to really try some things out, though.


There another example now available under samples/fibonacci.d 
that shows a complex example that works with LLVM 3.2.


 I'll keep messing around with it. It seems like you already 
 know that JIT doesn't work because we need 
 InitializeNativeTarget.


It does work, that is precisely what I reimplemented 
LLVMInitializeNativeTarget and LLVMInitializeAllTargets in D for. 
See llvm.c.functions.


 That all said, the tutorials, instructions, and documentation 
 for the C API of LLVM is pretty sparse. The best I've seen to 
 figure out how things work is to go through 
 http://llvm.org/doxygen/modules.html and piece together (very 
 slowly) how something might work. I'm also having to cross 
 reference tutorials made for the C++ API and figure out how to 
 make it work in C by reading the raw source code to get the 
 equivalent calls. Fortunately I did find that one post that 
 gave a close-to-complete example in C. Is there any better way 
 to do this at this point?


You just pretty much described what I've been doing myself, I 
don't know of a better way at present, sorry. I'm trying to 
advance the D API to a point where it's usable and then I'll add 
decent documentation to the D API, but until then you'll have to 
suffer through the doxygen "documentation" the same way I do^^

- Moritz
Mar 23 2013
prev sibling parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
The second warning has been dealt with now, as well (even though 
the program semantic has not changed at all, somethings wrong 
with the unreachable statement detenction >.>).

Also, readded the -ldl flag to lflags-posix in the package.json 
which solves the problem with the dl library not being linked in 
by dub. Apparently dub does not honor lib pragmas, you might want 
to report that (as you found it).

TLDR: Your example should now work, provided you fix what I 
previously mentioned. You can also look at sample/fibonacci.d 
which I used instead of your fac to confirm that you gist now 
works.

- Moritz
Mar 23 2013
parent reply "Chris Cain" <clcain uncg.edu> writes:
On Saturday, 23 March 2013 at 21:19:14 UTC, Moritz Maxeiner wrote:

 TLDR: Your example should now work, provided you fix what I 
 previously mentioned. You can also look at sample/fibonacci.d 
 which I used instead of your fac to confirm that you gist now 
 works.

 - Moritz


Awesome. Indeed, it now fully works (and JIT does work after all! 
Thanks for showing me how to use that). Thanks for the more 
interesting example in the README, it's extremely helpful. And 
also thank you for taking some time to help with the issues I was 
having.
Mar 23 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Sunday, 24 March 2013 at 01:35:28 UTC, Chris Cain wrote:

 On Saturday, 23 March 2013 at 21:19:14 UTC, Moritz Maxeiner 
 wrote:

 TLDR: Your example should now work, provided you fix what I 
 previously mentioned. You can also look at sample/fibonacci.d 
 which I used instead of your fac to confirm that you gist now 
 works.

 - Moritz


 Awesome. Indeed, it now fully works (and JIT does work after 
 all! Thanks for showing me how to use that). Thanks for the 
 more interesting example in the README, it's extremely helpful. 
 And also thank you for taking some time to help with the issues 
 I was having.


No problem, writing that fibonacci example forced me to read up 
Stuff about LLVM I need to know anyway (for making the D API)^^
Just one thing I forgot to mention: When you're using 
llvm.util.memory.toCString you'll need to take care of the 
allocated memory yourself, or you'll get memory leaks. The 
example is a special case as all the c strings need to be kept 
aroound until program termination, anyway (since LLVM's global 
context exists until then and all the c strings used by LLVM 
internally), but that's not the case with all LLVM C functions 
with c string args.

- Moritz
Mar 23 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 24 March 2013 at 02:37:52 UTC, Moritz Maxeiner wrote:

 On Sunday, 24 March 2013 at 01:35:28 UTC, Chris Cain wrote:

 On Saturday, 23 March 2013 at 21:19:14 UTC, Moritz Maxeiner 
 wrote:

 TLDR: Your example should now work, provided you fix what I 
 previously mentioned. You can also look at sample/fibonacci.d 
 which I used instead of your fac to confirm that you gist now 
 works.

 - Moritz


 Awesome. Indeed, it now fully works (and JIT does work after 
 all! Thanks for showing me how to use that). Thanks for the 
 more interesting example in the README, it's extremely 
 helpful. And also thank you for taking some time to help with 
 the issues I was having.


 No problem, writing that fibonacci example forced me to read up 
 Stuff about LLVM I need to know anyway (for making the D API)^^
 Just one thing I forgot to mention: When you're using 
 llvm.util.memory.toCString you'll need to take care of the 
 allocated memory yourself, or you'll get memory leaks. The 
 example is a special case as all the c strings need to be kept 
 aroound until program termination, anyway (since LLVM's global 
 context exists until then and all the c strings used by LLVM 
 internally), but that's not the case with all LLVM C functions 
 with c string args.


Question : why did you reorganize the modules ? They don't match 
LLVM's .h filenames.

Is that intended , if yes, why ?
Mar 27 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Wednesday, 27 March 2013 at 15:29:19 UTC, deadalnix wrote:

 On Sunday, 24 March 2013 at 02:37:52 UTC, Moritz Maxeiner wrote:

 On Sunday, 24 March 2013 at 01:35:28 UTC, Chris Cain wrote:

 On Saturday, 23 March 2013 at 21:19:14 UTC, Moritz Maxeiner 
 wrote:

 TLDR: Your example should now work, provided you fix what I 
 previously mentioned. You can also look at 
 sample/fibonacci.d which I used instead of your fac to 
 confirm that you gist now works.

 - Moritz


 Awesome. Indeed, it now fully works (and JIT does work after 
 all! Thanks for showing me how to use that). Thanks for the 
 more interesting example in the README, it's extremely 
 helpful. And also thank you for taking some time to help with 
 the issues I was having.


 No problem, writing that fibonacci example forced me to read 
 up Stuff about LLVM I need to know anyway (for making the D 
 API)^^
 Just one thing I forgot to mention: When you're using 
 llvm.util.memory.toCString you'll need to take care of the 
 allocated memory yourself, or you'll get memory leaks. The 
 example is a special case as all the c strings need to be kept 
 aroound until program termination, anyway (since LLVM's global 
 context exists until then and all the c strings used by LLVM 
 internally), but that's not the case with all LLVM C functions 
 with c string args.


 Question : why did you reorganize the modules ? They don't 
 match LLVM's .h filenames.

 Is that intended , if yes, why ?


Yes it is intended to be that way. Regarding the reasons for that 
decision:

Short answer:

Because in the case of the LLVM C API it makes things a lot 
easier and simpler to maintain, as well as trivial to add new 
parts for new LLVM versions.

Long answer:

1) The LLVM C API has headers which have very few lines of actual 
code (e.g. Linker.h which only contains one enum and one 
functions), making those into D modules seems wasteful.

2) Over versions of the LLVM C API headers appear and dissappear 
(e.g. Linker.h exists since 3.2, EnhancedDissassembly.h has been 
removed in trunk 3.3) and having them all around as D modules 
makes maintenence a lot more complicated.

3) Having all functions in a single compile time enum makes the 
runtime loading process a lot easier as you can generate all 
three steps ( 1. function pointer alias for every C function 2. 
function pointer variable for every function pointer alias 3. 
loading of the shared lib symbol into the function pointer for 
every function pointer) with but a few lines of codes, instead of 
having to write the names of all C functions three times. And 
since you can encode more information in that enum (an 
associative array in this case) adding more function is trivial: 
Simply add the function to the array in this manner: "NAME" : 
["SIGNATURE", "+", "VERSION"] and that's it (and if the function 
has been removed in said VERSION, change "+" into "-"). Since the 
MixinMap template (a CTFE version of the map function designed to 
get an array as its list and create D code for each of the items 
of said array based on a delegate function f) is used for the 
three steps described above no further code is needed.

TLDR: Supporting different versions of the LLVM C API takes 
considerably less effort this way and if there is a disadvantage 
to this approach big enough to outweigh that I can't see it.
Mar 27 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 27 March 2013 at 21:28:42 UTC, Moritz Maxeiner 
wrote:

 Yes it is intended to be that way. Regarding the reasons for 
 that decision:

 Short answer:

 Because in the case of the LLVM C API it makes things a lot 
 easier and simpler to maintain, as well as trivial to add new 
 parts for new LLVM versions.

 Long answer:

 1) The LLVM C API has headers which have very few lines of 
 actual code (e.g. Linker.h which only contains one enum and one 
 functions), making those into D modules seems wasteful.

 2) Over versions of the LLVM C API headers appear and 
 dissappear (e.g. Linker.h exists since 3.2, 
 EnhancedDissassembly.h has been removed in trunk 3.3) and 
 having them all around as D modules makes maintenence a lot 
 more complicated.

 3) Having all functions in a single compile time enum makes the 
 runtime loading process a lot easier as you can generate all 
 three steps ( 1. function pointer alias for every C function 2. 
 function pointer variable for every function pointer alias 3. 
 loading of the shared lib symbol into the function pointer for 
 every function pointer) with but a few lines of codes, instead 
 of having to write the names of all C functions three times. 
 And since you can encode more information in that enum (an 
 associative array in this case) adding more function is 
 trivial: Simply add the function to the array in this manner: 
 "NAME" : ["SIGNATURE", "+", "VERSION"] and that's it (and if 
 the function has been removed in said VERSION, change "+" into 
 "-"). Since the MixinMap template (a CTFE version of the map 
 function designed to get an array as its list and create D code 
 for each of the items of said array based on a delegate 
 function f) is used for the three steps described above no 
 further code is needed.

 TLDR: Supporting different versions of the LLVM C API takes 
 considerably less effort this way and if there is a 
 disadvantage to this approach big enough to outweigh that I 
 can't see it.


Ok, that sound reasonable.

Second question, why do you do stuff like :

enum Foo {
    FooA,
    FooB,
    FooC,
}

when you would do

enum Foo {
     A,
     B,
     C,
}

?

D enums introduce a scope, when C's don't. So C need to prefix 
enums entries manually, when in D it isn't required. The C to D 
translation goes as follow : FooA => Foo.A instead of Foo.FooA .

If the goal isn't to follow LLVM's source as closely as possible, 
I think this is the way to go.
Mar 27 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Thursday, 28 March 2013 at 05:04:42 UTC, deadalnix wrote:

 Ok, that sound reasonable.

 Second question, why do you do stuff like :

 enum Foo {
    FooA,
    FooB,
    FooC,
 }

 when you would do

 enum Foo {
     A,
     B,
     C,
 }

 ?


I don't, I do:

alias uint Foo;
enum : Foo
{
   FooA,
   FooB,
   FooC
}

which is a direct translation of the C enums. It gives the enum 
items no named scope, only creates an (alias) type for them. See
https://github.com/Calrama/llvm-d/blob/master/llvm/c/constants.d
and
https://github.com/Calrama/llvm-d/blob/master/llvm/c/types.d


 D enums introduce a scope, when C's don't. So C need to prefix 
 enums entries manually, when in D it isn't required. The C to D 
 translation goes as follow : FooA => Foo.A instead of Foo.FooA .


That is not the D equivalence of the C code. That is how D 
improves upon C enums. The equivalence and thus the translation 
is to not use a named scope as seen above.


 If the goal isn't to follow LLVM's source as closely as 
 possible, I think this is the way to go.


It does follow the LLVM C API source code as closely as possible. 
It just uses a module structure different from LLVM's C API 
header structure for the reasons stated above, which does not 
change how to use the C API (As you should do "import llvm.c.all" 
either way). This is also not a new approach, Derelict (one of 
the major providers for D bindings to C libraries other than 
deimos) has been doing it for a long time.
Mar 28 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 28 March 2013 at 09:23:27 UTC, Moritz Maxeiner wrote:

 I don't, I do:

 alias uint Foo;
 enum : Foo
 {
   FooA,
   FooB,
   FooC
 }

 which is a direct translation of the C enums. It gives the enum 
 items no named scope, only creates an (alias) type for them. See
 https://github.com/Calrama/llvm-d/blob/master/llvm/c/constants.d
 and
 https://github.com/Calrama/llvm-d/blob/master/llvm/c/types.d


Ho sorry, I missed that. That is still inferior as it doesn't 
introduce its own type now.


 D enums introduce a scope, when C's don't. So C need to prefix 
 enums entries manually, when in D it isn't required. The C to 
 D translation goes as follow : FooA => Foo.A instead of 
 Foo.FooA .


 That is not the D equivalence of the C code. That is how D 
 improves upon C enums. The equivalence and thus the translation 
 is to not use a named scope as seen above.


This is why I used the word translation. A good translation 
doesn't repeat stupidly, but uses idoms of the target language.


 If the goal isn't to follow LLVM's source as closely as 
 possible, I think this is the way to go.


 It does follow the LLVM C API source code as closely as 
 possible. It just uses a module structure different from LLVM's 
 C API header structure for the reasons stated above, which does 
 not change how to use the C API (As you should do "import 
 llvm.c.all" either way). This is also not a new approach, 
 Derelict (one of the major providers for D bindings to C 
 libraries other than deimos) has been doing it for a long time.


It doesn't follow as closely as possible as the way code is 
structured is part of the code. I also don't see the consistency 
between that choice and the enum one.
Mar 28 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Thursday, 28 March 2013 at 17:30:24 UTC, deadalnix wrote:

 On Thursday, 28 March 2013 at 09:23:27 UTC, Moritz Maxeiner 
 wrote:

 I don't, I do:

 alias uint Foo;
 enum : Foo
 {
  FooA,
  FooB,
  FooC
 }

 which is a direct translation of the C enums. It gives the 
 enum items no named scope, only creates an (alias) type for 
 them. See
 https://github.com/Calrama/llvm-d/blob/master/llvm/c/constants.d
 and
 https://github.com/Calrama/llvm-d/blob/master/llvm/c/types.d


 Ho sorry, I missed that. That is still inferior as it doesn't 
 introduce its own type now.


No problem, if you mean a strict/strong type, iirc (correct me if 
I'm wrong, though) that is the case for C enums as well. I can 
change the alias to typedef, though, if it is really that 
important.


 D enums introduce a scope, when C's don't. So C need to 
 prefix enums entries manually, when in D it isn't required. 
 The C to D translation goes as follow : FooA => Foo.A instead 
 of Foo.FooA .


 That is not the D equivalence of the C code. That is how D 
 improves upon C enums. The equivalence and thus the 
 translation is to not use a named scope as seen above.


 This is why I used the word translation. A good translation 
 doesn't repeat stupidly, but uses idoms of the target language.


Generally I would agree, but I'm translating the declarations of 
an interface here, not porting the implementation. And as such 
the content of the D interface imho should match the content of 
the C interface, meaning a literal translation, even if that 
makes it stupid from the D perspective. The *only* exception to 
that is the file structure as C headers are different things than 
D modules and there's no 100% match in D for C headers. D 
interface files (.di) would probably be technically the closest 
thing.


 If the goal isn't to follow LLVM's source as closely as 
 possible, I think this is the way to go.


 It does follow the LLVM C API source code as closely as 
 possible. It just uses a module structure different from 
 LLVM's C API header structure for the reasons stated above, 
 which does not change how to use the C API (As you should do 
 "import llvm.c.all" either way). This is also not a new 
 approach, Derelict (one of the major providers for D bindings 
 to C libraries other than deimos) has been doing it for a long 
 time.


 It doesn't follow as closely as possible as the way code is 
 structured is part of the code.


The way the code (in C) is structured is preserved in the D 
modules by grouping and documentation tags. The C header file 
structure itself is lost, yes, but not the logical structure it 
represent. As such I don't see any vital information being lost 
compared to translating the C headers as D modules.


 I also don't see the consistency between that choice and the 
 enum one.


It is only consistent under the directive "Follow the original C 
API code as closely as possible while minimizing the amount of 
work needed to maintain the result", not in a general sense, but 
consider this:

Your enum choice would create a break in the documentation, as 
the enum items would not match what is written in the LLVM 
doxygen documentation anymore (which means I would have to 
provide documentation for that difference); that also holds true 
for the file structure, of course, but since you'll get the 
entire C API with one import anyway ("import llvm.c.all") there's 
no additional documentation needed there (it is covered by the 
already existing examples).

If you want proper D enums (with names), though, they'll be 
present in the D API - mapping to the C bindings' enum's items.
Mar 28 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 28 March 2013 at 19:09:47 UTC, Moritz Maxeiner wrote:

 No problem, if you mean a strict/strong type, iirc (correct me 
 if I'm wrong, though) that is the case for C enums as well. I 
 can change the alias to typedef, though, if it is really that 
 important.


Typedef is depreacted. The D way is enum TypeName : superType { 
members }


 Generally I would agree, but I'm translating the declarations 
 of an interface here, not porting the implementation. And as 
 such the content of the D interface imho should match the 
 content of the C interface, meaning a literal translation, even 
 if that makes it stupid from the D perspective. The *only* 
 exception to that is the file structure as C headers are 
 different things than D modules and there's no 100% match in D 
 for C headers. D interface files (.di) would probably be 
 technically the closest thing.


Changing the module structure change the interface. Including all 
isn't a solution to that problem, as this kind of thing will make 
compile time and resource explode, especially since many things 
here are built out of mixins.


 Your enum choice would create a break in the documentation, as 
 the enum items would not match what is written in the LLVM 
 doxygen documentation anymore (which means I would have to 
 provide documentation for that difference); that also holds 
 true for the file structure, of course, but since you'll get 
 the entire C API with one import anyway ("import llvm.c.all") 
 there's no additional documentation needed there (it is covered 
 by the already existing examples).


It wont break LLVM's documentation. See for instance : 
http://llvm.org/docs/doxygen/html/group__LLVMCCoreType.html

LLVMGetTypeKind is clearly documented as returning a 
LLVMTypeKind, not an int.


 If you want proper D enums (with names), though, they'll be 
 present in the D API - mapping to the C bindings' enum's items.


Code duplication is rarely a good idea. Especially since your 
goal mentioned above is to reduce maintenance costs.
Mar 28 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Friday, 29 March 2013 at 03:47:47 UTC, deadalnix wrote:

 On Thursday, 28 March 2013 at 19:09:47 UTC, Moritz Maxeiner 
 wrote:
Including all isn't a solution to that problem, as this kind of 
thing
 will make compile time and resource explode, especially since 
 many things here are built out of mixins.


The problem you describe is that you consider not representing C 
headers as D modules a change in the interface, correct (I 
disagree, but that's another matter)? If noone interfaces with 
the structure at all, e.g. by importing an all module, what you 
feel is a problem won't bother anyone, as noone interacts with 
the structure anymore; thus it is a solution to that problem.

Now regarding the fact that you say that solution will have the 
negative side-effect of making compile time and resource 
"explode": I have tested llvm-d on some of my old machines 
(inluding a first-gen intel atom underclocked to about 800 Mhz) 
and llvm-d compilation was never noticable slow, or used 
considerable memory. But to either prove or contradict whether 
this is a noticable side-effect of that solution we'd need hard 
benchmarks.


 Your enum choice would create a break in the documentation, as 
 the enum items would not match what is written in the LLVM 
 doxygen documentation anymore (which means I would have to 
 provide documentation for that difference); that also holds 
 true for the file structure, of course, but since you'll get 
 the entire C API with one import anyway ("import llvm.c.all") 
 there's no additional documentation needed there (it is 
 covered by the already existing examples).


 It wont break LLVM's documentation. See for instance : 
 http://llvm.org/docs/doxygen/html/group__LLVMCCoreType.html

 LLVMGetTypeKind is clearly documented as returning a 
 LLVMTypeKind, not an int.


I did not talk about functions using the enums at that point, but 
about the enum definition (the enums' items):


 as the enum items would not match what is written in the LLVM 
 doxygen documentation anymore




See here under "Enumerations":
http://llvm.org/docs/doxygen/html/group__LLVMCCoreTypes.html

The enums documentation would not match the enums anymore, as the 
items' names would be different (even if it is just missing the 
part about their enum's name).


 If you want proper D enums (with names), though, they'll be 
 present in the D API - mapping to the C bindings' enum's items.


 Code duplication is rarely a good idea. Especially since your 
 goal mentioned above is to reduce maintenance costs.


The enums in the C bindings exist in a global context, while the 
enums in the D API are often bound to specific classes (as the 
original C++ enums are, to which the C API enums partially map). 
An alias to the C bindings' enums would not suffixe as then the 
global C bindings' enums' items could still be used as arguments. 
And again, my goal was/is to


 "Follow the original C API code as closely as possible while 
 minimizing the amount of work needed to maintain the result"




Not to simply cut down on maintenence costs. Also the maintenence 
cost for that is virtually nonexistent in constrast to the 
header/module issue.

This goal is also followed for the D API reconstruction of the 
C++ class hierarchy, so the enums that are bound to classes in 
the C++ API should be bound to the classes in the D API, unless 
their names already imply where they belong - as is the case with 
TypeID. Literal it should e.g. become in D: "Type.TypeID.Void" 
but here TypeID implies Type so I made it "TypeID.Void".

Another reason why the C bindings' enums and the D bindings' 
enums will stay seperated is for compatibility with deimos-llvm, 
which will be usable as a replacement for the internal C bindings 
for people who e.g. want a c header -> d module translation.
Mar 29 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 29 March 2013 at 12:07:11 UTC, Moritz Maxeiner wrote:

 Now regarding the fact that you say that solution will have the 
 negative side-effect of making compile time and resource 
 "explode": I have tested llvm-d on some of my old machines 
 (inluding a first-gen intel atom underclocked to about 800 Mhz) 
 and llvm-d compilation was never noticable slow, or used 
 considerable memory. But to either prove or contradict whether 
 this is a noticable side-effect of that solution we'd need hard 
 benchmarks.


DMD uses an huge amount of resource for CTFE, and leak a LOT of 
memory. That is already known. Obviously, compiling this lib by 
itself isn't going to trigger such issue.


 See here under "Enumerations":
 http://llvm.org/docs/doxygen/html/group__LLVMCCoreTypes.html

 The enums documentation would not match the enums anymore, as 
 the items' names would be different (even if it is just missing 
 the part about their enum's name).


It would simply transform LLVMXXXYYY into LLVMXXX.YYY, which is 
simply a workaround C limitations in the first place.


 Another reason why the C bindings' enums and the D bindings' 
 enums will stay seperated is for compatibility with 
 deimos-llvm, which will be usable as a replacement for the 
 internal C bindings for people who e.g. want a c header -> d 
 module translation.


You thrown away that with your module refactoring, so it is kind 
of pointless to raise that point now.
Apr 02 2013
parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Tuesday, 2 April 2013 at 11:16:27 UTC, deadalnix wrote:

 On Friday, 29 March 2013 at 12:07:11 UTC, Moritz Maxeiner wrote:

 Now regarding the fact that you say that solution will have 
 the negative side-effect of making compile time and resource 
 "explode": I have tested llvm-d on some of my old machines 
 (inluding a first-gen intel atom underclocked to about 800 
 Mhz) and llvm-d compilation was never noticable slow, or used 
 considerable memory. But to either prove or contradict whether 
 this is a noticable side-effect of that solution we'd need 
 hard benchmarks.


 DMD uses an huge amount of resource for CTFE, and leak a LOT of 
 memory. That is already known. Obviously, compiling this lib by 
 itself isn't going to trigger such issue.


I've not seen any mention about that under the "Documentation" 
category and I've only found posts about that after explicitly 
searching for that problem right now. I'm sorry for not knowing 
about that, but if that is such a critical problem, it may be 
sensible to document it in an article, e.g. "Compile time 
function evaluation", under dlang.org's "Documentation" category 
- I expected dmd to not use more memory for CTFE operations than 
D would for runtime operations.

Regardless of that, the llvm-d's D API will need to include all 
of the C API bindings anyway (however they may be done) and 
llvm-d's internal C bindings exist primarily for the use of the D 
API. If you only want the C bindings anyway, consider using 
deimos-llvm. I've forked it and am going to update it to 3.2 and 
3.3svn: https://github.com/Calrama/deimos-llvm


 Another reason why the C bindings' enums and the D bindings' 
 enums will stay seperated is for compatibility with 
 deimos-llvm, which will be usable as a replacement for the 
 internal C bindings for people who e.g. want a c header -> d 
 module translation.


 You thrown away that with your module refactoring, so it is 
 kind of pointless to raise that point now.


This is the important part of that sentence:

 which will be usable as a replacement for the internal C 
 bindings




I don't use C header -> D module in llvm-d's internal C bindings, 
llvm-d's D API, however, can use either llvm-d's internal C 
bindings or (soon) deimos-llvm. I do not consider it pointless to 
raise the point that if you (the user, not you specifically) want 
to have a C header -> D module translation you can use 
deimos-llvm together with llvm-d's D API without llvm'd internal 
C bindings, that's all.
Apr 02 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 2 April 2013 at 15:30:53 UTC, Moritz Maxeiner wrote:

 I've not seen any mention about that under the "Documentation" 
 category and I've only found posts about that after explicitly 
 searching for that problem right now. I'm sorry for not knowing 
 about that, but if that is such a critical problem, it may be 
 sensible to document it in an article, e.g. "Compile time 
 function evaluation", under dlang.org's "Documentation" 
 category - I expected dmd to not use more memory for CTFE 
 operations than D would for runtime operations.


I'd prefers see the problem fixed than documented. Anyway it is 
real, and should be considered now.


 Regardless of that, the llvm-d's D API will need to include all 
 of the C API bindings anyway (however they may be done) and 
 llvm-d's internal C bindings exist primarily for the use of the 
 D API. If you only want the C bindings anyway, consider using 
 deimos-llvm. I've forked it and am going to update it to 3.2 
 and 3.3svn: https://github.com/Calrama/deimos-llvm


I don't think all modules need to, but you know better than me.


 I don't use C header -> D module in llvm-d's internal C 
 bindings, llvm-d's D API, however, can use either llvm-d's 
 internal C bindings or (soon) deimos-llvm. I do not consider it 
 pointless to raise the point that if you (the user, not you 
 specifically) want to have a C header -> D module translation 
 you can use deimos-llvm together with llvm-d's D API without 
 llvm'd internal C bindings, that's all.


The thing is that any automated tool will suffer from the 
translated module name as it cannot transform #include directives.
Apr 04 2013
parent "Moritz Maxeiner" <moritz ucworks.org> writes:
On Thursday, 4 April 2013 at 18:31:59 UTC, deadalnix wrote:

 On Tuesday, 2 April 2013 at 15:30:53 UTC, Moritz Maxeiner wrote:

 I've not seen any mention about that under the "Documentation" 
 category and I've only found posts about that after explicitly 
 searching for that problem right now. I'm sorry for not 
 knowing about that, but if that is such a critical problem, it 
 may be sensible to document it in an article, e.g. "Compile 
 time function evaluation", under dlang.org's "Documentation" 
 category - I expected dmd to not use more memory for CTFE 
 operations than D would for runtime operations.


 I'd prefers see the problem fixed than documented. Anyway it is 
 real, and should be considered now.


For the internal C bindings I'll stick with the CTFE (although 
I'll try to minimize/optimize its use). For the deimos-llvm pull 
request I used CTFE as well, but Jens already said that he thinks 
it's more natural to use multiple branches for supporting 
different LLVM versions (See here: 
https://github.com/jkm/deimos-llvm/pull/2) and llvm-d now 
supports using deimos-llvm as a replacement for its internal C 
bindings, so if the CTFE is a serious problem for someone, using 
deimos-llvm once he has integrated the split the CTFE into the 
branches would be the best option, I think.


 Regardless of that, the llvm-d's D API will need to include 
 all of the C API bindings anyway (however they may be done) 
 and llvm-d's internal C bindings exist primarily for the use 
 of the D API. If you only want the C bindings anyway, consider 
 using deimos-llvm. I've forked it and am going to update it to 
 3.2 and 3.3svn: https://github.com/Calrama/deimos-llvm


 I don't think all modules need to, but you know better than me.


At present not yet, but since all LLVM C API functions wrap 
around LLVM's C++ class functions (except a few globals, which 
wrap around C++ globals) afaik to mimic as much of the LLVM 
hierarchy as possible, all these wrapper functions will be 
needed. I'll only be able to tell for sure once I'm finished, 
though and judging be the amount of work left that's going to 
take a while.


 I don't use C header -> D module in llvm-d's internal C 
 bindings, llvm-d's D API, however, can use either llvm-d's 
 internal C bindings or (soon) deimos-llvm. I do not consider 
 it pointless to raise the point that if you (the user, not you 
 specifically) want to have a C header -> D module translation 
 you can use deimos-llvm together with llvm-d's D API without 
 llvm'd internal C bindings, that's all.


 The thing is that any automated tool will suffer from the 
 translated module name as it cannot transform #include 
 directives.


I'm not quite sure where you'd use an automated tool here that 
needs to be aware of any translation from C to D but if you do 
why not change the tool to make it aware of the 
constant/type/functions approach - It's not something I invented 
or actually new, see Derelict for example...

In any case, if you really need that, go with the deimos-llvm 
bindings. All you have to do is use "-version=DEIMOS_LLVM" and 
the internal C bindings won't even be imported by the D API and 
subsequently don't have to be compiled.
Apr 04 2013