www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Any interest in libgit bindings?

reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
http://libgit2.github.com/

libgit has things like clone/commit/tag, reading/writing git loose
object files. You can easily deflate object files using std.zlib, but
unfortunately there's no git pack file unpacker in libgit (you can
externally use git's unpack-objects command for that).

I don't know whether dstep[1] can autogenerate the bindings (haven't
tried it myself), any luck there from anyone?

My dgen generator (unreleased) can generate bindings but it's a
C++-based generator so the generated files aren't pure 1to1 bindings
since forwarding functions were automatically generated (this is an
implementation detail of the generator that can and will be fixed).

win32-based bindings are here, along with somewhat ported samples (a
little hardcoded but things generally work):
https://github.com/AndrejMitrovic/dgen_libgit

But if there's any interest I could clean it up and make it a nice
simple C wrapper just like the other deimos projects
(https://github.com/D-Programming-Deimos). If Dstep can do it
automatically even better, but I didn't get to use it yet.

[1] https://github.com/jacob-carlborg/dstep/
Oct 09 2012
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/9/2012 3:46 PM, Andrej Mitrovic wrote:

 But if there's any interest I could clean it up and make it a nice
 simple C wrapper just like the other deimos projects
 (https://github.com/D-Programming-Deimos).


Sure!
Oct 09 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-10 00:46, Andrej Mitrovic wrote:

 http://libgit2.github.com/

 libgit has things like clone/commit/tag, reading/writing git loose
 object files. You can easily deflate object files using std.zlib, but
 unfortunately there's no git pack file unpacker in libgit (you can
 externally use git's unpack-objects command for that).

 I don't know whether dstep[1] can autogenerate the bindings (haven't
 tried it myself), any luck there from anyone?


If there aren't too many macros or bitfields it shouldn't be any 
problem. But the more people use it the better it can be.


 My dgen generator (unreleased) can generate bindings but it's a
 C++-based generator so the generated files aren't pure 1to1 bindings
 since forwarding functions were automatically generated (this is an
 implementation detail of the generator that can and will be fixed).

 win32-based bindings are here, along with somewhat ported samples (a
 little hardcoded but things generally work):
 https://github.com/AndrejMitrovic/dgen_libgit

 But if there's any interest I could clean it up and make it a nice
 simple C wrapper just like the other deimos projects
 (https://github.com/D-Programming-Deimos). If Dstep can do it
 automatically even better, but I didn't get to use it yet.


I would absolutely have interest in this.

-- 
/Jacob Carlborg
Oct 10 2012
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/10/12, Jacob Carlborg <doob me.com> wrote:

 I would absolutely have interest in this.


https://github.com/AndrejMitrovic/dgen_libgit

Well it's mostly done (can't tell if anything is missing, except the
docs). I had to reimplement inline functions in D since they're not
exported on the C side (maybe -fkeep-inline-functions could work on
Posix but it doesn't on Windows). I don't know of a non-intrusive way
to work around this so I copy-pasted the functions to D and made them
compilable.

I'll put an extern(C) at the top later to replace all the extern(C)
declarations, but that's just a styling issue that's easily fixed.

But I can't get it to link with libgit on Linux though. After
compiling and installing libgit (which copied the shared lib to
/usr/local/lib), I've tried:

$ cd samples/diff
$ rdmd --force -L-L/usr/local/lib -L-lgit2 -I../../src diff.d

/tmp/.rdmd-1000/rdmd-diff.d-572543621CF312EC72273F3DB4C6B8BE/diff:
error while loading shared libraries: libgit2.so.0: cannot open shared
object file: No such file or directory

I never have luck with Posix. :)

This is from branch 'development', commit
acd1700630ea1159a55dc5e8cee12e4a725afe18 from
https://github.com/libgit2/libgit2/

Instructions on building libgit2 are here: http://libgit2.github.com/#install
Oct 10 2012
parent reply "Craig Dillabaugh" <cdillaba cg.scs.carleton.ca> writes:
On Thursday, 11 October 2012 at 01:39:46 UTC, Andrej Mitrovic
wrote:

 On 10/10/12, Jacob Carlborg <doob me.com> wrote:

 I would absolutely have interest in this.


 https://github.com/AndrejMitrovic/dgen_libgit

 Well it's mostly done (can't tell if anything is missing, 
 except the
 docs). I had to reimplement inline functions in D since they're 
 not
 exported on the C side (maybe -fkeep-inline-functions could 
 work on
 Posix but it doesn't on Windows). I don't know of a 
 non-intrusive way
 to work around this so I copy-pasted the functions to D and 
 made them
 compilable.

 I'll put an extern(C) at the top later to replace all the 
 extern(C)
 declarations, but that's just a styling issue that's easily 
 fixed.

 But I can't get it to link with libgit on Linux though. After
 compiling and installing libgit (which copied the shared lib to
 /usr/local/lib), I've tried:

 $ cd samples/diff
 $ rdmd --force -L-L/usr/local/lib -L-lgit2 -I../../src diff.d

 /tmp/.rdmd-1000/rdmd-diff.d-572543621CF312EC72273F3DB4C6B8BE/diff:
 error while loading shared libraries: libgit2.so.0: cannot open 
 shared
 object file: No such file or directory

 I never have luck with Posix. :)

 This is from branch 'development', commit
 acd1700630ea1159a55dc5e8cee12e4a725afe18 from
 https://github.com/libgit2/libgit2/

 Instructions on building libgit2 are here: 
 http://libgit2.github.com/#install


Likely you have already tried this but just in case not ...

For Linux is /usr/local/lib in your  /etc/ld.so.conf file?
If not, add it and run ldconfig.

Craig
Oct 10 2012
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/11/12, Craig Dillabaugh <cdillaba cg.scs.carleton.ca> wrote:

 Likely you have already tried this but just in case not ...

 For Linux is /usr/local/lib in your  /etc/ld.so.conf file?
 If not, add it and run ldconfig.


Thanks! That did the trick, it's linking now.

I'll do a few verification tests and then post the bindings to d.announce later.
Oct 10 2012
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/10/12, Jacob Carlborg <doob me.com> wrote:

 I would absolutely have interest in this.


On 10/10/12, Justin Whear <justin economicmodeling.com> wrote:

 I'd be interested.


On 10/11/12, alexhairyman <alexhairyman gmail.com> wrote:

 I think this would be cool :D


For those that missed it, the bindings are now online:

http://forum.dlang.org/thread/mailman.730.1349931087.5162.digitalmars-d-announce puremagic.com
Oct 11 2012
prev sibling next sibling parent reply Justin Whear <justin economicmodeling.com> writes:
On Wed, 10 Oct 2012 00:46:28 +0200, Andrej Mitrovic wrote:


 http://libgit2.github.com/
 
 libgit has things like clone/commit/tag, reading/writing git loose
 object files. You can easily deflate object files using std.zlib, but
 unfortunately there's no git pack file unpacker in libgit (you can
 externally use git's unpack-objects command for that).
 
 I don't know whether dstep[1] can autogenerate the bindings (haven't
 tried it myself), any luck there from anyone?
 
 My dgen generator (unreleased) can generate bindings but it's a
 C++-based generator so the generated files aren't pure 1to1 bindings
 since forwarding functions were automatically generated (this is an
 implementation detail of the generator that can and will be fixed).
 
 win32-based bindings are here, along with somewhat ported samples (a
 little hardcoded but things generally work):
 https://github.com/AndrejMitrovic/dgen_libgit
 
 But if there's any interest I could clean it up and make it a nice
 simple C wrapper just like the other deimos projects
 (https://github.com/D-Programming-Deimos). If Dstep can do it
 automatically even better, but I didn't get to use it yet.
 
 [1] https://github.com/jacob-carlborg/dstep/


I'd be interested.  I hand-wrote bindings to most of libgit2 (though not 
the latest version), but I skipped most of the write-oriented stuff, so a 
complete binding would be welcome.
Oct 10 2012
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/10/12, Justin Whear <justin economicmodeling.com> wrote:

 I'd be interested.  I hand-wrote bindings to most of libgit2 (though not
 the latest version), but I skipped most of the write-oriented stuff, so a
 complete binding would be welcome.


Well the master branch was last updated 8 months ago and the libgit
devs said I should only use the development branch because master
doesn't compile, so yeah the bindings will be based on the latest
version.

It's pure irony that libgit devs don't have a stable main branch. I
mean, what exactly is Git for again? :D
Oct 10 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-10 00:46, Andrej Mitrovic wrote:

 http://libgit2.github.com/

 libgit has things like clone/commit/tag, reading/writing git loose
 object files. You can easily deflate object files using std.zlib, but
 unfortunately there's no git pack file unpacker in libgit (you can
 externally use git's unpack-objects command for that).

 I don't know whether dstep[1] can autogenerate the bindings (haven't
 tried it myself), any luck there from anyone?


This is the output of dstep on all files in libgit2/include:

https://dl.dropbox.com/u/18386187/libgit2.zip

I have no idea if it compiles.

-- 
/Jacob Carlborg
Oct 10 2012
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/10/12, Jacob Carlborg <doob me.com> wrote:

 This is the output of dstep on all files in libgit2/include:
 https://dl.dropbox.com/u/18386187/libgit2.zip

 I have no idea if it compiles.


Yeah, after incorporating some C shortcuts mine is now similar to
yours: https://github.com/AndrejMitrovic/dgen_libgit/tree/master/out/d/git2

Mine compiles but I'll have to reference some inlined functions since
they're not exported in the DLL. Yours looks very similar (you almost
can't go wrong with generating C bindings, it's very easy compared to
C++).

Btw, you should try to rename a module from 'object' to '_object' or
similar, otherwise it might conflict with the implicitly imported
object module in druntime. Walter flagged this as wontfix
(http://d.puremagic.com/issues/show_bug.cgi?id=7651).
Oct 10 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-10 23:04, Andrej Mitrovic wrote:


 Mine compiles but I'll have to reference some inlined functions since
 they're not exported in the DLL. Yours looks very similar (you almost
 can't go wrong with generating C bindings, it's very easy compared to
 C++).


Yeah, I haven't done anything about inline functions yet. Another think 
I haven't been able to figure out yet is to how to properly translate 
include directives. It's not easy to translate to the module system used 
by D.


 Btw, you should try to rename a module from 'object' to '_object' or
 similar, otherwise it might conflict with the implicitly imported
 object module in druntime. Walter flagged this as wontfix
 (http://d.puremagic.com/issues/show_bug.cgi?id=7651).


Right, I missed that. It's easy to forget this kind of symbols, they're 
not really keywords but they are kind of reserved words. Any other 
symbols behaving like this I could have missed?

-- 
/Jacob Carlborg
Oct 10 2012
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
 Another thing I haven't been able to figure out yet is to how to properly
translate
 include directives. It's not easy to translate to the module system used
 by D.


I don't have access to the include directives at all. The way I do it
is to have a "toNativeType" function which translates C/C++ types to D
and at the same time resolves includes. If a type is located in
another module (all non-fundamental types have a fileID field in XML)
then I insert that module's name to the currently generating module's
imports. Once all the module's symbols are generated I generate the
import directives.


 Right, I missed that. It's easy to forget this kind of symbols, they're
 not really keywords but they are kind of reserved words. Any other
 symbols behaving like this I could have missed?


Well if the library defines files which get converted to modules like
'std' you could easily conflict with Phobos. Some words I check are:
std core gc rt object Object string toString toHash mangleof. But
there's probably more, when something clashes I add it to the list.

You could easily get standard aliases (the ones from implicitly
imported object.d) hijacked, e.g. if there was a string typedef in a C
library:

alias const(char)* string;  // hijacks string alias from object.d
Oct 11 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-11 16:34, Andrej Mitrovic wrote:


 I don't have access to the include directives at all. The way I do it
 is to have a "toNativeType" function which translates C/C++ types to D
 and at the same time resolves includes. If a type is located in
 another module (all non-fundamental types have a fileID field in XML)
 then I insert that module's name to the currently generating module's
 imports. Once all the module's symbols are generated I generate the
 import directives.


Actually I already do the same. But that's only for the C standard 
library and a couple of Posix headers, basically all files in core.stdc. 
But the problem is that I get the full path to the source file where the 
symbol is defined. How do I turn that in to an import declaration? 
Something relative to the include paths?

With symbols from the standard library and the Posix headers there's the 
problem with many of the symbols actually are not defined in the header 
the documentation says. Instead they're included using various macro 
conditions from other "private" headers.


 Well if the library defines files which get converted to modules like
 'std' you could easily conflict with Phobos. Some words I check are:
 std core gc rt object Object string toString toHash mangleof. But
 there's probably more, when something clashes I add it to the list.

 You could easily get standard aliases (the ones from implicitly
 imported object.d) hijacked, e.g. if there was a string typedef in a C
 library:

 alias const(char)* string;  // hijacks string alias from object.d


I actually started a new thread for this:

http://forum.dlang.org/thread/k55qpq$20io$1 digitalmars.com

-- 
/Jacob Carlborg
Oct 11 2012
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/11/12, Jacob Carlborg <doob me.com> wrote:

 Actually I already do the same. But that's only for the C standard
 library and a couple of Posix headers, basically all files in core.stdc.
 But the problem is that I get the full path to the source file where the
 symbol is defined. How do I turn that in to an import declaration?
 Something relative to the include paths?


I guess it depends on what you get from clang. The XML I'm working on
has this sort of output:
  <File id="f3" name="git2/submodule.h"/>
  <File id="f4"
name="c:/mingw/bin/../lib/gcc/mingw32/4.6.1/../../../../include/inttypes.h"/>
  <File id="f39" name="c:/mingw/bin/../lib/gcc/mingw32/4.6.1/include/stddef.h"/>

This is translated into:
git2\submodule.d
mingw\include\inttypes.d
mingw\lib\gcc\mingw32\_4\_6\_1\include\stddef.d

The last one could use improvement (_4\_6\_1\ should be _4_6_1). I
call remDrive, chomp(".h"), chomp(".tcc"), translate slashes to dot,
and I make sure module names aren't keywords and it's done.

As for macros, I don't have access to those. If a symbol is
initialized with a macro then I get the post-processed value of the
macro, but I don't have the definition of the macro itself.
Oct 11 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-10-11 21:38, Andrej Mitrovic wrote:


 I guess it depends on what you get from clang. The XML I'm working on
 has this sort of output:
    <File id="f3" name="git2/submodule.h"/>
    <File id="f4"
 name="c:/mingw/bin/../lib/gcc/mingw32/4.6.1/../../../../include/inttypes.h"/>
    <File id="f39" name="c:/mingw/bin/../lib/gcc/mingw32/4.6.1/include/stddef.h"/>


I'm pretty sure I get the full path of every header. That is 
"/usr/home/foo/libgit2/includes/git2/submodule.h", or something similar.


 This is translated into:
 git2\submodule.d
 mingw\include\inttypes.d
 mingw\lib\gcc\mingw32\_4\_6\_1\include\stddef.d

 The last one could use improvement (_4\_6\_1\ should be _4_6_1). I
 call remDrive, chomp(".h"), chomp(".tcc"), translate slashes to dot,
 and I make sure module names aren't keywords and it's done.


Yeah, that looks a bit ugly.


 As for macros, I don't have access to those. If a symbol is
 initialized with a macro then I get the post-processed value of the
 macro, but I don't have the definition of the macro itself.


I haven't tackled macros yet. There are no API for this in libclang yet 
so I would have to create that first.

On the other hand, libclang do have an API for doxygen documentation 
which recently got added. If think that is awesome and it will be in the 
next release of Clang, which they recently announced the release 
schedule for.

-- 
/Jacob Carlborg
Oct 11 2012
prev sibling parent "alexhairyman" <alexhairyman gmail.com> writes:
On Tuesday, 9 October 2012 at 23:10:39 UTC, Andrej Mitrovic wrote:

 http://libgit2.github.com/

 libgit has things like clone/commit/tag, reading/writing git 
 loose
 object files. You can easily deflate object files using 
 std.zlib, but
 unfortunately there's no git pack file unpacker in libgit (you 
 can
 externally use git's unpack-objects command for that).

 I don't know whether dstep[1] can autogenerate the bindings 
 (haven't
 tried it myself), any luck there from anyone?

 My dgen generator (unreleased) can generate bindings but it's a
 C++-based generator so the generated files aren't pure 1to1 
 bindings
 since forwarding functions were automatically generated (this 
 is an
 implementation detail of the generator that can and will be 
 fixed).

 win32-based bindings are here, along with somewhat ported 
 samples (a
 little hardcoded but things generally work):
 https://github.com/AndrejMitrovic/dgen_libgit

 But if there's any interest I could clean it up and make it a 
 nice
 simple C wrapper just like the other deimos projects
 (https://github.com/D-Programming-Deimos). If Dstep can do it
 automatically even better, but I didn't get to use it yet.

 [1] https://github.com/jacob-carlborg/dstep/


I think this would be cool :D
Oct 10 2012