www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The CAPI Manifesto

reply Walter Bright <newshound2 digitalmars.com> writes:
Brad and I were talking about some D code that needed openssl support, when we 
ran into the same old problem:

No D files corresponding to the openssl C .h files.

It's not that these are a big problem to create, it's just that they are not 
done, and it tends to turn off people from using D. D is binary API compatible 
with C, but only with a corresponding D import file. This, out of the box,
makes 
D *harder* to use than C.

Lots of people roll their own, but that work is hard to find and haphazard.

This problem keeps coming up again and again.

So I propose creating, on github.com/D-Programming-Language, a new repository 
called CAPI.

The CAPI Manifesto
------------------

CAPI is a collection of C header files to publicly available C libraries
and their translations to D. The idea is that if, in C, to interface to a
library
one would write:

    #include "foo.h"

then the corresponding D code would look like:

    import foo;

Each C .h file would have a corresponding .d file. Each C directory would
have a corresponding D directory, for example:

    #include "bar/foo.h"   // C

    import bar.foo; // D

The top level directory of each library will have two subdirectories:

    C/
    D/

and there will be a one-to-one correspondence of files and directory structure
between them.

The D import files will be a rote translation of the corresponding C .h file.
No attempt will be made to fix, improve, or extend the C api. No attempt will
be made to duplicate the C documentation, or replace it in any way. There
will be no unittests. Every effort will be made to avoid needing any D specific
binary files.

When an updated version of the C header files becomes available, those will
get checked into the C subdirectory tree, and then the corresponding D files
will get updated.

Version tags used must match the version tags used by the C API files.

The license used for the D versions should match the C ones, as they are a
derived work.
Oct 16 2011
next sibling parent reply Ary Manzana <ary esperanto.org.ar> writes:
On 10/16/11 11:02 PM, Walter Bright wrote:

 Brad and I were talking about some D code that needed openssl support,
 when we ran into the same old problem:

 No D files corresponding to the openssl C .h files.

 It's not that these are a big problem to create, it's just that they are
 not done, and it tends to turn off people from using D. D is binary API
 compatible with C, but only with a corresponding D import file. This,
 out of the box, makes D *harder* to use than C.

 Lots of people roll their own, but that work is hard to find and haphazard.

 This problem keeps coming up again and again.

 So I propose creating, on github.com/D-Programming-Language, a new
 repository called CAPI.


So you would put every interface to every possible C code there?

In Ruby if you want to have very efficient code you'd implement it as C 
extensions. For that, you create wrappers in Ruby for C. Now, big part 
of the standard library has extensions for the most needed things. 
Everything else, like bindings to an efficient xml parser, are made by 
different people that public them as gems. Having a public gem 
repository it's really easy to find bindings for whatever you want. They 
don't need to be part of the standard library. And it wouldn't make 
sense, having so much functionality out there available as C code.

So I'd suggest having D headers for the most common things in phobos and 
focusing on a tool like rubygems. It would give such a big boost to the 
language.

I also can't imagine how that big repository would work. You'd copy the 
remote file locally? What if that file gets fixes? You'd copy it again? 
Or maybe you'd git checkout everything from that repository locally and 
synchronize it from time to time, with the chance of breaking existing 
code...

Having "gems" and versioning them should make all these problems 
disappear. Maybe there is an openssl header in D. The problem is that 
there might be many, and they don't know each other, and google is a 
maze to find such things.
Oct 16 2011
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2011 7:18 PM, Ary Manzana wrote:

 So you would put every interface to every possible C code there?


No. It's simply a collection point for interfaces to publicly available C 
libraries. The place to go to see if someone has already written what you're 
looking for. Its mission is also narrowly focused on simply being a way to 
access existing C libraries from D in the most straightforward way possible.



 In Ruby if you want to have very efficient code you'd implement it as C
 extensions. For that, you create wrappers in Ruby for C. Now, big part of the
 standard library has extensions for the most needed things. Everything else,
 like bindings to an efficient xml parser, are made by different people that
 public them as gems. Having a public gem repository it's really easy to find
 bindings for whatever you want. They don't need to be part of the standard
 library. And it wouldn't make sense, having so much functionality out there
 available as C code.


Right.


 So I'd suggest having D headers for the most common things in phobos


Yes.


 and focusing on a tool like rubygems. It would give such a big boost to the
language.


I think github is adequate for now to be the tool.



 I also can't imagine how that big repository would work. You'd copy the remote
 file locally?


That'll work, or you can use the github 'clone' feature.


 What if that file gets fixes? You'd copy it again? Or maybe you'd
 git checkout everything from that repository locally and synchronize it from
 time to time, with the chance of breaking existing code...
 Having "gems" and versioning them should make all these problems disappear.


Github has extensive versioning and branching support. This shouldn't be an
issue.


 Maybe there is an openssl header in D. The problem is that there might be many,
 and they don't know each other, and google is a maze to find such things.


Exactly. And maybe there are several openssl D headers, each of which is 
half-assed in a different way. With a central library for them, we can improve 
them globally rather than piecemeal and randomly.
Oct 16 2011
parent reply Jude Young <10equals2 gmail.com> writes:
Please excuse my ignorance,
but several types in D do not currently translate well into C.

For example, strings in D are not '\0' terminated, which breaks with C.
This is not usually a problem, and it's easy to wrap the function in the
translated header file to automate that particular process.

It seems that your proposal would disallow this particular example.
The translation code would have to be somewhere, what is the argument
against allowing it?

In my (admittedly ignorant) opinion, it seems that allowing the automatic
translation of D types to C types
would fit very well, otherwise you'll have to add them yourself every time
you wish to call C.

Forgive me If I have misunderstood something.
Oct 16 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2011 9:28 PM, Jude Young wrote:

 Please excuse my ignorance,
 but several types in D do not currently translate well into C.

 For example, strings in D are not '\0' terminated, which breaks with C.


This is incorrect. String literals in D are 0 terminated, as in C. C has a 
convention that strings are 0 terminated, but it isn't part of the core 
language. In D, you can terminate a string with 0 if you want to, or not.


 This is not usually a problem, and it's easy to wrap the function in the
 translated header file to automate that particular process.


This would fall under attempting to fix the api, which would be beyond the
scope 
of the library.



 It seems that your proposal would disallow this particular example.
 The translation code would have to be somewhere, what is the argument against
 allowing it?


D access to the C API should be direct, and not include hidden costs like 
translation layers.



 In my (admittedly ignorant) opinion, it seems that allowing the automatic
 translation of D types to C types
 would fit very well, otherwise you'll have to add them yourself every time you
 wish to call C.


C doesn't actually even have a 'string' type.
Oct 16 2011
parent Jude Young <10equals2 gmail.com> writes:
Fair enough, thanks for the time.
I found a thin D binding to ncurses, and 'fixed' it to work with D2.
If y'all get this up and running, this code will probably be better than
starting from scratch.

In any case, I think that CAPI is exactly the type of thing that D needs
going forward.
I really hope that CAPI gets good support.

On Mon, Oct 17, 2011 at 12:55 AM, Walter Bright
<newshound2 digitalmars.com>wrote:


 On 10/16/2011 9:28 PM, Jude Young wrote:


 Please excuse my ignorance,
 but several types in D do not currently translate well into C.

 For example, strings in D are not '\0' terminated, which breaks with C.


 This is incorrect. String literals in D are 0 terminated, as in C. C has a
 convention that strings are 0 terminated, but it isn't part of the core
 language. In D, you can terminate a string with 0 if you want to, or not.


  This is not usually a problem, and it's easy to wrap the function in the

 translated header file to automate that particular process.


 This would fall under attempting to fix the api, which would be beyond the
 scope of the library.



  It seems that your proposal would disallow this particular example.

 The translation code would have to be somewhere, what is the argument
 against
 allowing it?


 D access to the C API should be direct, and not include hidden costs like
 translation layers.



  In my (admittedly ignorant) opinion, it seems that allowing the automatic

 translation of D types to C types
 would fit very well, otherwise you'll have to add them yourself every time
 you
 wish to call C.


 C doesn't actually even have a 'string' type.
Oct 16 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-17 04:18, Ary Manzana wrote:

 On 10/16/11 11:02 PM, Walter Bright wrote:

 Brad and I were talking about some D code that needed openssl support,
 when we ran into the same old problem:

 No D files corresponding to the openssl C .h files.

 It's not that these are a big problem to create, it's just that they are
 not done, and it tends to turn off people from using D. D is binary API
 compatible with C, but only with a corresponding D import file. This,
 out of the box, makes D *harder* to use than C.

 Lots of people roll their own, but that work is hard to find and
 haphazard.

 This problem keeps coming up again and again.

 So I propose creating, on github.com/D-Programming-Language, a new
 repository called CAPI.


 So you would put every interface to every possible C code there?

 In Ruby if you want to have very efficient code you'd implement it as C
 extensions. For that, you create wrappers in Ruby for C. Now, big part
 of the standard library has extensions for the most needed things.
 Everything else, like bindings to an efficient xml parser, are made by
 different people that public them as gems. Having a public gem
 repository it's really easy to find bindings for whatever you want. They
 don't need to be part of the standard library. And it wouldn't make
 sense, having so much functionality out there available as C code.

 So I'd suggest having D headers for the most common things in phobos and
 focusing on a tool like rubygems. It would give such a big boost to the
 language.

 I also can't imagine how that big repository would work. You'd copy the
 remote file locally? What if that file gets fixes? You'd copy it again?
 Or maybe you'd git checkout everything from that repository locally and
 synchronize it from time to time, with the chance of breaking existing
 code...

 Having "gems" and versioning them should make all these problems
 disappear. Maybe there is an openssl header in D. The problem is that
 there might be many, and they don't know each other, and google is a
 maze to find such things.


Already working on a package manager for D:

https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
https://github.com/jacob-carlborg/orbit/

-- 
/Jacob Carlborg
Oct 17 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/17/2011 12:42 AM, Jacob Carlborg wrote:

 Already working on a package manager for D:

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
 https://github.com/jacob-carlborg/orbit/



Is it possible (and worthwhile) to layer a package manager over a github
repository?
Oct 17 2011
next sibling parent reply Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
Walter Bright wrote:


 On 10/17/2011 12:42 AM, Jacob Carlborg wrote:

 Already working on a package manager for D:

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
 https://github.com/jacob-carlborg/orbit/


 
 
 Is it possible (and worthwhile) to layer a package manager over a github
 repository?


Yes, pip (for python) can do it.
Oct 17 2011
parent Brad Anderson <eco gnuk.net> writes:
On Mon, Oct 17, 2011 at 11:46 PM, Lutger Blijdestijn <
lutger.blijdestijn gmail.com> wrote:


 Walter Bright wrote:


 On 10/17/2011 12:42 AM, Jacob Carlborg wrote:

 Already working on a package manager for D:




 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 https://github.com/jacob-carlborg/orbit/



 Is it possible (and worthwhile) to layer a package manager over a github
 repository?


 Yes, pip (for python) can do it.


Homebrew[1] does too.  It's rather slick actually.  You can type "brew edit
<package>" and it'll open the script it uses to install that package
(typically just a configure and make install) in an editor.  You make your
changes and save and it saves to a local git clone of the "formulas"
(package installation scripts).  If you have a shared development
environment you can share this modified repository among your developers so
everyone is working with the same packages.  It's design is rather clever
and makes full use of DVCSes.

Regards,
Brad Anderson

[1] http://mxcl.github.com/homebrew/
Oct 17 2011
prev sibling next sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 18.10.2011, 07:26 Uhr, schrieb Walter Bright  
<newshound2 digitalmars.com>:


 On 10/17/2011 12:42 AM, Jacob Carlborg wrote:

 Already working on a package manager for D:

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
 https://github.com/jacob-carlborg/orbit/



 Is it possible (and worthwhile) to layer a package manager over a github  
 repository?


All package managers have to store their packages in some sort of  
repository. It's more like it solves Jacob's problem where to upload the  
files now that you proposed CAPI :)
Oct 17 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-18 07:26, Walter Bright wrote:

 On 10/17/2011 12:42 AM, Jacob Carlborg wrote:

 Already working on a package manager for D:

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
 https://github.com/jacob-carlborg/orbit/



 Is it possible (and worthwhile) to layer a package manager over a github
 repository?


I'm planning to support github repositories as packages. Have a look at 
"Source Code Management":

https://github.com/jacob-carlborg/orbit/wiki/integration

-- 
/Jacob Carlborg
Oct 17 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/17/2011 11:40 PM, Jacob Carlborg wrote:

 On 2011-10-18 07:26, Walter Bright wrote:

 Is it possible (and worthwhile) to layer a package manager over a github
 repository?


 I'm planning to support github repositories as packages. Have a look at "Source
 Code Management":

 https://github.com/jacob-carlborg/orbit/wiki/integration



Great!
Oct 18 2011
next sibling parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
The main problem with this approach is how to support different versions =
of a library, or of OS. It quickly becomes difficult to support anything =
but the latest, or a fixed version.
It works beautifully for mature libs.

I still cannot avoid thinking that a C frontend automatically generating =
D modules with the help of recipes would be a better way.
It will need some manual intervention for "difficult" cases, mainly =
giving manual translation of some macros, but it should be small.

One would set if all the files correspond to modules, or there are just =
some "main" directories/files.

Some things are easy:
#define a
enum { a=3Dtrue }
#define b "xyz"
enum { b=3D"xyz" }

one could be tempted to replace=20
	#ifdef x
with
	static if (is(typeof(x)) && x)
and treat other #if in a similar way, but in D a static if must contain =
a full statement, as its content must be syntactically valid, whereas =
the C preprocessor does not have this limitation.
The way to work around this, if we create the headers on demand is =
simple: we already evaluate all #if using the building definitions of =
the associated C compiler (gcc -E -dD for example) and its default =

directives).

real macros are more tricky, for example one could do

#define isSmall(x) (x<2)
isSmall(T)(T x){
	return x<2;
}

#define c(x) { x , #x }
template(alias x){
	{ x, x.stringof }
}

thus c(t) has to become c!(t).

and maybe one has to provide some macros definition by hand, but I guess =
the cases are not so much.

In all this there is still a major pitfall: redefinitions of the same =
macro. It is not common, but it happens, and when it does everything =
breaks.
One could give different names for the clashing symbols, but it remains =
ugly.
Furthermore in D one cannot define the same interface to a C function =
twice and import it in the same scope through two modules, because it =
will clash, even if private.

This makes the whole more complicated, but I think that a few recipes =
coding the exceptions like macro translations, macros/defs to suppress =
or rename it should work pretty well.

Once could analyze if different "views" of the same include file are =
compatible, and automatically check for double definitions.
It isn't an easy project, but it would be very useful if done correctly. =
I remember talking about it with Lindquist quite some time ago=85

Fawzi=
Oct 21 2011
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 10/22/2011 01:20 AM, Fawzi Mohamed wrote:

 The main problem with this approach is how to support different versions of a
library, or of OS. It quickly becomes difficult to support anything but the
latest, or a fixed version.
 It works beautifully for mature libs.

 I still cannot avoid thinking that a C frontend automatically generating D
modules with the help of recipes would be a better way.
 It will need some manual intervention for "difficult" cases, mainly giving
manual translation of some macros, but it should be small.

 One would set if all the files correspond to modules, or there are just some
"main" directories/files.

 Some things are easy:
 #define a
 enum { a=true }
 #define b "xyz"
 enum { b="xyz" }

 one could be tempted to replace
 	#ifdef x
 with
 	static if (is(typeof(x))&&  x)
 and treat other #if in a similar way, but in D a static if must contain a full
statement, as its content must be syntactically valid, whereas the C
preprocessor does not have this limitation.


D does not have this limitation either. Use string mixins. The only 
difference between C macros and D string mixins is that D is more 
explicit about that the feature is mere string manipulation. There is 
nothing you cannot do with string mixins that is possible with macros. 
(except hijacking existing code or making macro instantiations look like 
function calls for transparent interchangeability, of course).



 The way to work around this, if we create the headers on demand is simple: we
already evaluate all #if using the building definitions of the associated C
compiler (gcc -E -dD for example) and its default include paths (or directly


 real macros are more tricky, for example one could do

 #define isSmall(x) (x<2)
 isSmall(T)(T x){
 	return x<2;
 }



isSmall(x); // use macro in C code

string isSmall(string x) {
         return `{return ~x~`;}`;
}
mixin(isSmall(q{x}); // use macro in D.

or, with Kenji Hara's proposal:

mixin template isSmall(string x){
         enum isSmall = `{return ~x~`;}`;
}

isSmall!q{x} // use macro in D code



 #define c(x) { x , #x }


string c(string x){
         return `{ x , q{`~x~`} }`;
}

mixin(c(q{x})); // use macro in D code

or, again, with Kenji Hara's proposal:

mixin template c(string x){
         enum c = `{ x , q{`~x~`} }`;
}

c!q{x} // use macro in D code

multiple parameters would possibly be best handled like this:

mixin template ADD(string x){
     enum cc = {
         string p = x.split(",");
         assert(p.length == 2, "expected 2 parameters");
         return `( `~p[0] ~ '+' ~ p[0] ~ ` )`;
     }();
}

ADD!q{x,y} // use macro in D code
Oct 22 2011
prev sibling parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
On Oct 21, 2011, at 4:20 PM, Fawzi Mohamed wrote:


 The main problem with this approach is how to support different =


versions of a library, or of OS. It quickly becomes difficult to support =
anything but the latest, or a fixed version.

 It works beautifully for mature libs.
=20
 I still cannot avoid thinking that a C frontend automatically =


generating D modules with the help of recipes would be a better way.

 It will need some manual intervention for "difficult" cases, mainly =


giving manual translation of some macros, but it should be small.

=85 and it seems that in the time I was offline others came up with the =
same idea...
Oct 21 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2011 4:32 PM, Fawzi Mohamed wrote:

 On Oct 21, 2011, at 4:20 PM, Fawzi Mohamed wrote:


 The main problem with this approach is how to support different versions of
 a library, or of OS. It quickly becomes difficult to support anything but
 the latest, or a fixed version. It works beautifully for mature libs.




Since github has excellent support for branches, I don't see why this is a
major 
problem.



 I still cannot avoid thinking that a C frontend automatically generating D
 modules with the help of recipes would be a better way. It will need some
 manual intervention for "difficult" cases, mainly giving manual translation
 of some macros, but it should be small.


 … and it seems that in the time I was offline others came up with the same
 idea...


It's an old idea. The trouble is, as always, the C preprocessor. I'm currently 
converting the openssl .h files, and they are a zoo of metaprogramming using C 
preprocessor macros.

People are going to demand perfect translation if it is automatic.

The only way to do it is to work with the preprocessed output of the .h file, 
and just forget about the preprocessor.
Oct 21 2011
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/22/2011 04:33 AM, Walter Bright wrote:

 On 10/21/2011 4:32 PM, Fawzi Mohamed wrote:

 On Oct 21, 2011, at 4:20 PM, Fawzi Mohamed wrote:


 The main problem with this approach is how to support different
 versions of
 a library, or of OS. It quickly becomes difficult to support anything
 but
 the latest, or a fixed version. It works beautifully for mature libs.




 Since github has excellent support for branches, I don't see why this is
 a major problem.



 I still cannot avoid thinking that a C frontend automatically
 generating D
 modules with the help of recipes would be a better way. It will need
 some
 manual intervention for "difficult" cases, mainly giving manual
 translation
 of some macros, but it should be small.


 … and it seems that in the time I was offline others came up with the
 same
 idea...


 It's an old idea. The trouble is, as always, the C preprocessor. I'm
 currently converting the openssl .h files, and they are a zoo of
 metaprogramming using C preprocessor macros.

 People are going to demand perfect translation if it is automatic.

 The only way to do it is to work with the preprocessed output of the .h
 file, and just forget about the preprocessor.


Another way is to replace the preprocessor with CTFE and string mixins. 
I think that could be automated quite easily. (modulo the possibility of 
some extremely heavy abuse on the C side that could make the other parts 
of the translation a lot harder of course)
Oct 22 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I have a question about licensing. If you translate a C LGPL'ed header
file to D, and you keep the same license, are you still allowed to use
whichever license in your user code that uses the new D files? Because
I don't know whether using LGPL'ed .d files falls under "using the
library" or "extending the library".

For example, CairoD has translated Cairo LGPL/MPL'ed header files in
equivalent Boost-licensed .d files. I know I'm asking for free lawyer
advice here, but do you think this is going to be a problem?
Oct 22 2011
next sibling parent Johannes Pfau <spam example.com> writes:
Andrej Mitrovic wrote:

I have a question about licensing. If you translate a C LGPL'ed header
file to D, and you keep the same license, are you still allowed to use
whichever license in your user code that uses the new D files? Because
I don't know whether using LGPL'ed .d files falls under "using the
library" or "extending the library".

For example, CairoD has translated Cairo LGPL/MPL'ed header files in
equivalent Boost-licensed .d files. I know I'm asking for free lawyer
advice here, but do you think this is going to be a problem?


I'm interested in this as well. This is especially evil as we currently
have to link statically to D code. A D file consisting of only C
imports probably doesn't count as code (and there's actually no need
to compile it), but if you have macros which were translated into
functions or templates, wouldn't statically linking against this LGPL'd
'header' code require everything to be LGPL/GPL licensed? 

-- 
Johannes Pfau
Oct 22 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-22 15:41, Andrej Mitrovic wrote:

 I have a question about licensing. If you translate a C LGPL'ed header
 file to D, and you keep the same license, are you still allowed to use
 whichever license in your user code that uses the new D files? Because
 I don't know whether using LGPL'ed .d files falls under "using the
 library" or "extending the library".

 For example, CairoD has translated Cairo LGPL/MPL'ed header files in
 equivalent Boost-licensed .d files. I know I'm asking for free lawyer
 advice here, but do you think this is going to be a problem?


You still need to link to the C library.

-- 
/Jacob Carlborg
Oct 22 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/22/11, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-22 15:41, Andrej Mitrovic wrote:

 I have a question about licensing. If you translate a C LGPL'ed header
 file to D, and you keep the same license, are you still allowed to use
 whichever license in your user code that uses the new D files? Because
 I don't know whether using LGPL'ed .d files falls under "using the
 library" or "extending the library".

 For example, CairoD has translated Cairo LGPL/MPL'ed header files in
 equivalent Boost-licensed .d files. I know I'm asking for free lawyer
 advice here, but do you think this is going to be a problem?


 You still need to link to the C library.

 --
 /Jacob Carlborg


Yes, but *linking* isn't extending the library. Importing D files that
are LGPL licensed could be different.
Oct 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-10-22 21:12, Andrej Mitrovic wrote:

 On 10/22/11, Jacob Carlborg<doob me.com>  wrote:

 On 2011-10-22 15:41, Andrej Mitrovic wrote:

 I have a question about licensing. If you translate a C LGPL'ed header
 file to D, and you keep the same license, are you still allowed to use
 whichever license in your user code that uses the new D files? Because
 I don't know whether using LGPL'ed .d files falls under "using the
 library" or "extending the library".

 For example, CairoD has translated Cairo LGPL/MPL'ed header files in
 equivalent Boost-licensed .d files. I know I'm asking for free lawyer
 advice here, but do you think this is going to be a problem?


 You still need to link to the C library.

 --
 /Jacob Carlborg


 Yes, but *linking* isn't extending the library. Importing D files that
 are LGPL licensed could be different.


According to GPL it is and according to LGPL it is for static linking, IIRC.

-- 
/Jacob Carlborg
Oct 23 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, October 22, 2011 15:41:13 Andrej Mitrovic wrote:

 I have a question about licensing. If you translate a C LGPL'ed header
 file to D, and you keep the same license, are you still allowed to use
 whichever license in your user code that uses the new D files? Because
 I don't know whether using LGPL'ed .d files falls under "using the
 library" or "extending the library".
 
 For example, CairoD has translated Cairo LGPL/MPL'ed header files in
 equivalent Boost-licensed .d files. I know I'm asking for free lawyer
 advice here, but do you think this is going to be a problem?


I would _think_ that it would be okay as long as it's a direct translation, 
since all you're doing is making it so that you can link to the libraries as 
you would in C, not extending their functionality in any way. But I'm not a 
lawyer, and I'm far from an expert on software licensing.

- Jonathan M Davis
Oct 22 2011
prev sibling parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
On Oct 22, 2011, at 4:03 AM, Timon Gehr wrote:


 On 10/22/2011 04:33 AM, Walter Bright wrote:

 On 10/21/2011 4:32 PM, Fawzi Mohamed wrote:

=20
 On Oct 21, 2011, at 4:20 PM, Fawzi Mohamed wrote:
=20

 The main problem with this approach is how to support different
 versions of
 a library, or of OS. It quickly becomes difficult to support =








anything

 but
 the latest, or a fixed version. It works beautifully for mature =








libs.

=20
 Since github has excellent support for branches, I don't see why this =




is

 a major problem.




do you have a repo per library? if yes then indeed it is feasible. I =
didn't think about that.


 I still cannot avoid thinking that a C frontend automatically
 generating D
 modules with the help of recipes would be a better way. It will =








need

 some
 manual intervention for "difficult" cases, mainly giving manual
 translation
 of some macros, but it should be small.


=20
 =85 and it seems that in the time I was offline others came up with =






the

 same
 idea...


=20
 It's an old idea. The trouble is, as always, the C preprocessor. I'm
 currently converting the openssl .h files, and they are a zoo of
 metaprogramming using C preprocessor macros.
=20
 People are going to demand perfect translation if it is automatic.




that was the reason I talked about recipes that can add manual fixes =
where needed (for selected macros).


=20
 The only way to do it is to work with the preprocessed output of the =




.h

 file, and just forget about the preprocessor.


=20
 Another way is to replace the preprocessor with CTFE and string =


mixins. I think that could be automated quite easily. (modulo the =
possibility of some extremely heavy abuse on the C side that could make =
the other parts of the translation a lot harder of course)

I think string mixings are an extremely ugly solution to this problem, =
and I would try to avoid that, especially if they are used to represent =
a function that should be inlined, and might be replaced by a normal =
function in a later version.
Oct 22 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/22/2011 6:12 PM, Fawzi Mohamed wrote:

 Another way is to replace the preprocessor with CTFE and string mixins. I
 think that could be automated quite easily. (modulo the possibility of some
 extremely heavy abuse on the C side that could make the other parts of the
 translation a lot harder of course)


 I think string mixings are an extremely ugly solution to this problem, and I
 would try to avoid that, especially if they are used to represent a function
 that should be inlined, and might be replaced by a normal function in a later
 version.


Sure, but I've seen some pretty ugly use of preprocessor macros to paste 
together statements that need to be inserted. String mixins are the only hope.
Oct 22 2011
prev sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 22.10.2011, 04:33 Uhr, schrieb Walter Bright  =

<newshound2 digitalmars.com>:


 On 10/21/2011 4:32 PM, Fawzi Mohamed wrote:

 On Oct 21, 2011, at 4:20 PM, Fawzi Mohamed wrote:


 The main problem with this approach is how to support different  =







 versions of
 a library, or of OS. It quickly becomes difficult to support anythin=






g  =


 but
 the latest, or a fixed version. It works beautifully for mature libs=






.

 Since github has excellent support for branches, I don't see why this =


is  =


 a major problem.



 I still cannot avoid thinking that a C frontend automatically  =







 generating D
 modules with the help of recipes would be a better way. It will need=






  =


 some
 manual intervention for "difficult" cases, mainly giving manual  =







 translation
 of some macros, but it should be small.


 =E2=80=A6 and it seems that in the time I was offline others came up =




with the  =


 same
 idea...


 It's an old idea. The trouble is, as always, the C preprocessor. I'm  =



 currently converting the openssl .h files, and they are a zoo of  =



 metaprogramming using C preprocessor macros.

 People are going to demand perfect translation if it is automatic.

 The only way to do it is to work with the preprocessed output of the .=


h  =


 file, and just forget about the preprocessor.


Doesn't that mean no #ifdef WINDOWS ?
Oct 24 2011
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
Overall, I think that it sounds like a good idea, but I'd suggest that we 
namespace stuff. So, instead of

import foo;

you'd get something like

import capi.foo;

Otherwise, we unnecessarily increase the odds of module names conflicting with 
modules in other projects.

Also, if we're putting it up on github, we may want to come up with a cooler 
name than CAPI (though it _is_ right to the point). There's already a user 
with the name CAPI ( https://github.com/capi ), so aside from whether CAPI is 
a good name in its own right or not, that might cause problems. Maybe pick 
something suitably Greek or Roman, given that we already have Phobos? The name 
of one of the Titans would be fitting given that it relates to C (on the theory 
that the Greek gods are the successors of the Titans and D's standard library 
is named after a Greek god).

It also brings up the question of who is going to manage this project. Someone 
(or preferrably, a group of someones) is going to have to have commit access 
in order to merge in pull requests. Do want to just start off with that being 
the same group of folks with Phobos commit access and grow it from there as 
appropriate? We certainly wouldn't want to insist that the two groups be the 
same, but it does seem like a good place to start from.

- Jonathan M Davis
Oct 16 2011
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Mon, 17 Oct 2011 05:51:57 +0300, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:


 Also, if we're putting it up on github, we may want to come up with a  
 cooler
 name than CAPI (though it _is_ right to the point). There's already a  
 user
 with the name CAPI ( https://github.com/capi ), so aside from whether  
 CAPI is
 a good name in its own right or not, that might cause problems. Maybe  
 pick
 something suitably Greek or Roman, given that we already have Phobos?  
 The name
 of one of the Titans would be fitting given that it relates to C (on the  
 theory
 that the Greek gods are the successors of the Titans and D's standard  
 library
 is named after a Greek god).


I think the name's fine. I don't see "<4-letter word> is already taken" as  
a valid argument, considering the huge number of users. There's a GitHub  
user called "phobos" and one called "tools", etc.

Also, I'm pretty sure the D standard library is named after Mars's moon,  
considering that D was originally named "Mars Programming Language" (after  
the company name, "Digital Mars").

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 16 2011
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2011 8:17 PM, Vladimir Panteleev wrote:

 I think the name's fine. I don't see "<4-letter word> is already taken" as a
 valid argument, considering the huge number of users. There's a GitHub user
 called "phobos" and one called "tools", etc.

 Also, I'm pretty sure the D standard library is named after Mars's moon,
 considering that D was originally named "Mars Programming Language" (after the
 company name, "Digital Mars").


I thought maybe Diemos.
Oct 16 2011
parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
That sounds good.
Now D projects will have a mother and a father.
Phobos is the mother, which is always there for you and is the first
one you'll go to if you need help :-)
Diemos is the father, which is the backbone of your software family,
who you rely on to do all the hard work for you. :-)

It's generally a good idea to separate our own code from an external
one, which we provide support for.
And if the "remote modules" proposal gets implemented, there would be
no problems with using Diemos if it won't get included in DMD package.

On Mon, Oct 17, 2011 at 7:55 AM, Walter Bright
<newshound2 digitalmars.com> wrote:

 On 10/16/2011 8:17 PM, Vladimir Panteleev wrote:

 I think the name's fine. I don't see "<4-letter word> is already taken" as
 a
 valid argument, considering the huge number of users. There's a GitHub
 user
 called "phobos" and one called "tools", etc.

 Also, I'm pretty sure the D standard library is named after Mars's moon,
 considering that D was originally named "Mars Programming Language" (after
 the
 company name, "Digital Mars").


 I thought maybe Diemos.
Oct 17 2011
parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 17.10.2011, 10:20 Uhr, schrieb Gor Gyolchanyan  
<gor.f.gyolchanyan gmail.com>:


 That sounds good.
 Now D projects will have a mother and a father.
 Phobos is the mother, which is always there for you and is the first
 one you'll go to if you need help :-)
 Diemos is the father, which is the backbone of your software family,
 who you rely on to do all the hard work for you. :-)

 It's generally a good idea to separate our own code from an external
 one, which we provide support for.
 And if the "remote modules" proposal gets implemented, there would be
 no problems with using Diemos if it won't get included in DMD package.

 On Mon, Oct 17, 2011 at 7:55 AM, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 10/16/2011 8:17 PM, Vladimir Panteleev wrote:

 I think the name's fine. I don't see "<4-letter word> is already  
 taken" as
 a
 valid argument, considering the huge number of users. There's a GitHub
 user
 called "phobos" and one called "tools", etc.

 Also, I'm pretty sure the D standard library is named after Mars's  
 moon,
 considering that D was originally named "Mars Programming Language"  
 (after
 the
 company name, "Digital Mars").


 I thought maybe Diemos.




Deimos, people, its Deimos >.<
Oct 17 2011
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Is this repo intended for use by users or library writers? I don't
think encouraging writing C-style code in D is a good idea, that's why
I'm asking.
Oct 16 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2011 9:02 PM, Andrej Mitrovic wrote:

 Is this repo intended for use by users or library writers?


Both.


 I don't
 think encouraging writing C-style code in D is a good idea, that's why
 I'm asking.


D's ability to directly call any C code is a huge advantage, but one we've 
underutilized because of a lack of interface imports to the great mass of 
popular C libraries.

For example, are we going to reinvent openssl? libcurl? imagemajick? No way.
Oct 16 2011
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2011 7:51 PM, Jonathan M Davis wrote:

 Overall, I think that it sounds like a good idea, but I'd suggest that we
 namespace stuff. So, instead of

 import foo;

 you'd get something like

 import capi.foo;

 Otherwise, we unnecessarily increase the odds of module names conflicting with
 modules in other projects.


Perhaps you're right.



 Also, if we're putting it up on github, we may want to come up with a cooler
 name than CAPI (though it _is_ right to the point). There's already a user
 with the name CAPI ( https://github.com/capi ), so aside from whether CAPI is
 a good name in its own right or not, that might cause problems. Maybe pick
 something suitably Greek or Roman, given that we already have Phobos? The name
 of one of the Titans would be fitting given that it relates to C (on the theory
 that the Greek gods are the successors of the Titans and D's standard library
 is named after a Greek god).


Unless CAPI is trademarked, I think we're in good shape.



 It also brings up the question of who is going to manage this project. Someone
 (or preferrably, a group of someones) is going to have to have commit access
 in order to merge in pull requests. Do want to just start off with that being
 the same group of folks with Phobos commit access and grow it from there as
 appropriate? We certainly wouldn't want to insist that the two groups be the
 same, but it does seem like a good place to start from.


I figure initially the same team members as are on the phobos team, but it
would 
be a separate team with its own member list.
Oct 16 2011
parent reply Mirko Pilger <pilger cymotec.de> writes:
 Unless CAPI is trademarked, I think we're in good shape.


when i hear of CAPI i instantly think of this:

http://www.capi.org/pages/home.php
Oct 17 2011
parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
Yeah. Diemos is much better, IMO.

On Mon, Oct 17, 2011 at 12:30 PM, Mirko Pilger <pilger cymotec.de> wrote:

 Unless CAPI is trademarked, I think we're in good shape.


 when i hear of CAPI i instantly think of this:

 http://www.capi.org/pages/home.php
Oct 17 2011
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Mon, 17 Oct 2011 05:02:52 +0300, Walter Bright  
<newshound2 digitalmars.com> wrote:


 So I propose creating, on github.com/D-Programming-Language, a new  
 repository called CAPI.


Questions:

1) Will these be distributed with DMD? The bar to generate D import  
modules from C headers isn't much higher than having to find and download  
headers from the Internet.

2) This isn't too different to already existing projects. The bindings  
project on dsource ( http://dsource.org/projects/bindings ) already has  
bindings for various libraries. I can't say much about the project  
guidelines, though.

3) You suggest to place each library in its own directory, with C and D  
headers as subdirectories. This means that the user will still need to  
edit the import search path when using a new library. Is it realistic to  
put all D files in the same directory? (Perhaps do this only for libraries  
for which we don't expect name collisions?)

4) "Every effort will be made to avoid needing any D specific binary  
files." - What about import libraries?

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 16 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2011 8:10 PM, Vladimir Panteleev wrote:

 1) Will these be distributed with DMD? The bar to generate D import modules
from
 C headers isn't much higher than having to find and download headers from the
 Internet.


I was thinking, no at this time. I suspect it may grow to be quite large, and 
would become rather onerous. I also don't want to tie it to the DMD release
cycle.



 2) This isn't too different to already existing projects. The bindings project
 on dsource ( http://dsource.org/projects/bindings ) already has bindings for
 various libraries. I can't say much about the project guidelines, though.


The bindings project is a great resource, though it seems a little
disorganized. 
We've had great success with github, meaning it seems to be very good at 
encouraging community participation.



 3) You suggest to place each library in its own directory, with C and D headers
 as subdirectories. This means that the user will still need to edit the import
 search path when using a new library.


Yes.


 Is it realistic to put all D files in the
 same directory? (Perhaps do this only for libraries for which we don't expect
 name collisions?)


Hmm, you're right. Perhaps openssl.whatever.



 4) "Every effort will be made to avoid needing any D specific binary files." -
 What about import libraries?


What do you mean by import libraries? Do you mean Windows DLL import libraries? 
Those would be supplied by whoever supplied the C library. D can access those 
directly.

CAPI should be interface source code only library; the D equivalent of #include.
Oct 16 2011
prev sibling next sibling parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
I think there might be a few tricks to improve the C API without
adding any new code.
For example, replace by-pointer parameter declarations with _out_
parameters when applicable (the underlying function signature is the
same), replace const parameters with in parameters, etc.
This won't change the C API a single bit (won't even add new code),
but will vastly improve readability and sometimes safety of the API.
In other cases, some minor additions could be made, for example:
libjpeg provides API to register error handlers, instead of setting
errno and such. Those kind of situations could be used to throw
exceptions. It only takes a static this() and a few lines of code.
In case those kind of modifications/additions are made, there could be
a standard way to disable them and use the original version.

On Mon, Oct 17, 2011 at 6:02 AM, Walter Bright
<newshound2 digitalmars.com> wrote:

 Brad and I were talking about some D code that needed openssl support, wh=


en

 we ran into the same old problem:

 No D files corresponding to the openssl C .h files.

 It's not that these are a big problem to create, it's just that they are =


not

 done, and it tends to turn off people from using D. D is binary API
 compatible with C, but only with a corresponding D import file. This, out=


 of

 the box, makes D *harder* to use than C.

 Lots of people roll their own, but that work is hard to find and haphazar=


d.

 This problem keeps coming up again and again.

 So I propose creating, on github.com/D-Programming-Language, a new
 repository called CAPI.

 The CAPI Manifesto
 ------------------

 CAPI is a collection of C header files to publicly available C libraries
 and their translations to D. The idea is that if, in C, to interface to a
 library
 one would write:

 =A0 #include "foo.h"

 then the corresponding D code would look like:

 =A0 import foo;

 Each C .h file would have a corresponding .d file. Each C directory would
 have a corresponding D directory, for example:

 =A0 #include "bar/foo.h" =A0 // C

 =A0 import bar.foo; // D

 The top level directory of each library will have two subdirectories:

 =A0 C/
 =A0 D/

 and there will be a one-to-one correspondence of files and directory
 structure
 between them.

 The D import files will be a rote translation of the corresponding C .h
 file.
 No attempt will be made to fix, improve, or extend the C api. No attempt
 will
 be made to duplicate the C documentation, or replace it in any way. There
 will be no unittests. Every effort will be made to avoid needing any D
 specific
 binary files.

 When an updated version of the C header files becomes available, those wi=


ll

 get checked into the C subdirectory tree, and then the corresponding D fi=


les

 will get updated.

 Version tags used must match the version tags used by the C API files.

 The license used for the D versions should match the C ones, as they are =


a

 derived work.
Oct 16 2011
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2011 10:24 PM, Gor Gyolchanyan wrote:

 I think there might be a few tricks to improve the C API without
 adding any new code.
 For example, replace by-pointer parameter declarations with _out_
 parameters when applicable (the underlying function signature is the
 same), replace const parameters with in parameters, etc.
 This won't change the C API a single bit (won't even add new code),
 but will vastly improve readability and sometimes safety of the API.
 In other cases, some minor additions could be made, for example:
 libjpeg provides API to register error handlers, instead of setting
 errno and such. Those kind of situations could be used to throw
 exceptions. It only takes a static this() and a few lines of code.
 In case those kind of modifications/additions are made, there could be
 a standard way to disable them and use the original version.


If you want to add a layer on top of the C API, that would be fine. std.zlib is 
an example of that.

But the idea of CAPI is NOT to add a layer. Not fix, extend, refactor, improve,
etc.

Just the thinnest possible direct calls to the C API. Any improvements, fixes, 
whatever, should be a separate project.

I know the urge to do these fixes can be overpowering, but they end badly every 
time. It's like trying to mix the language's lexer up with the semantic
analysis :-)

They really are better off being separate and distinct.

For one thing, it makes the inevitable maintenance *FAR* easier, as those C
APIs 
will change. Updating the corresponding D module becomes simple then - just a 
line by line comparison and tweaking. There would be no tearing of hair and 
rending of garments.

For another it means you'll have to regenerate the C documentation, but with
all 
the changes you made. Then, as the C guys improve their documentation, your 
layer falls behind, gets neglected, and finally sucks in comparison.
Oct 16 2011
parent travert phare.normalesup.org (Christophe Travert) writes:
Walter Bright , dans le message (digitalmars.D:146786), a écrit :

 If you want to add a layer on top of the C API, that would be fine. std.zlib
is 
 an example of that.
 
 But the idea of CAPI is NOT to add a layer. Not fix, extend, refactor,
improve, etc.


I definitely agree with that, no attempt should be made to fix anything.

However, if you want only direct translation, the only way I see is to 
treat all defines as mixins. That mean all macro will become very 
tedious to use. Even there, there are choices to make (how do you 
translate multi-parameter macros ?). So I think a minimum of adaptation 
should be provided.

Here is an example of how this could be made:

#define square(x) ((x)*(x))

=>

// direct translation
mixin template!(string x)square
{
    enum square = '((' ~ x ~ ')*(' ~ x ~ '))';
}

// adapted version
T square(T)(T x) { return x*x; }


Which version should be added ?
Both do not do the same thing ! As you know, In the first one, if x is a 
function call, it is called twice, and it the other one, it is called 
only once.
So if I follow the rule: no fix, no extend, etc, I must include only the 
direct translation. But the CAPI library will not be as usable as the c 
one. Then I miss the purpose of CAPI: make d as usable as c when using 
c libraries.

So which version should be added: I think it is both:

// direct translation
mixin template!(string x)M_square
{
    enum M_square = '((' ~ x ~ ')*(' ~ x ~ '))';
}

// adapted version
T square(T)(T x) { return mixin M_square!q{x}; }


So this is what I propose:
Direct translation have to be provided. Adapted version can be provided 
alongside the direct translation of the d header. Adaptaded version must 
be a direct forward call to the direct version (note here that the 
adapted. Rules will have to be defined to avoid name clashes (for 
example, here, I used a the direct name for the usable version, and M_ 
prefix for the mixin version, but we could decide other rules).


Macros are a big issue.

And I think abvious translating, such as const char* to string, or 
pointer-length pairs to dynamic arrays is about the same problem.


double array_sum(double* a, size_t length);

=>
double array_sum(double* a, size_t length);
// AND
double array_sum(double[] a) { return array_sum(a.ptr, a.length); }


That is very little work. The direct translation is mandatory, and the 
adapted translation is not. But when the translation is obvious, there 
is no reason for everyone to make it on it's corner. Make it in the 
public header and share it!

I order to remain consistent, adaptation will have to obey to very 
precise rules, that have to be set. No fix, no extend, no refactor, no 
improve, etc. Just a forward call, to have, in addition to the C API, 
an API that use D's power: using enums, inline functions, for defines 
instead of the direct mixin translation. Using D's arrays instead of C 
arrays, etc. could be nice too.

What translation should be provided ? What rules to translate defines ? 
Is translation of pointer+length pair to array worth doing ? What about 
stringz and strings ? Where to draw the line ?


-- 
Christophe
Oct 28 2011
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Mon, 17 Oct 2011 08:24:26 +0300, Gor Gyolchanyan  
<gor.f.gyolchanyan gmail.com> wrote:


 For example, replace by-pointer parameter declarations with _out_
 parameters when applicable (the underlying function signature is the
 same)


Note that this would make it nearly impossible to pass null pointers. A  
human would need to decide if a null pointer can be specified, or the  
converter would need to be aware of proprietary extensions which specify  
such things (e.g. Microsoft's __in).

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 16 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2011 11:21 PM, Vladimir Panteleev wrote:

 On Mon, 17 Oct 2011 08:24:26 +0300, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:


 For example, replace by-pointer parameter declarations with _out_
 parameters when applicable (the underlying function signature is the
 same)


 Note that this would make it nearly impossible to pass null pointers. A human
 would need to decide if a null pointer can be specified, or the converter would
 need to be aware of proprietary extensions which specify such things (e.g.
 Microsoft's __in).



Right. You could also annotate C API functions with pure,  safe, etc., but
you'd 
have to be very careful that those functions actually were that way, and would 
not violate those attributes in the future.
Oct 16 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
OR

Maybe parallel to the CAPI, there could be the D-ified version of it,
that will be developed after the original CAPI.
This is, of course, not as urgent a the CAPI itself, but it would be
very useful to gradually help users get rid of unnecessarily dangerous
code.

On Mon, Oct 17, 2011 at 9:24 AM, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:

 I think there might be a few tricks to improve the C API without
 adding any new code.
 For example, replace by-pointer parameter declarations with _out_
 parameters when applicable (the underlying function signature is the
 same), replace const parameters with in parameters, etc.
 This won't change the C API a single bit (won't even add new code),
 but will vastly improve readability and sometimes safety of the API.
 In other cases, some minor additions could be made, for example:
 libjpeg provides API to register error handlers, instead of setting
 errno and such. Those kind of situations could be used to throw
 exceptions. It only takes a static this() and a few lines of code.
 In case those kind of modifications/additions are made, there could be
 a standard way to disable them and use the original version.

 On Mon, Oct 17, 2011 at 6:02 AM, Walter Bright
 <newshound2 digitalmars.com> wrote:

 Brad and I were talking about some D code that needed openssl support, w=




hen

 we ran into the same old problem:

 No D files corresponding to the openssl C .h files.

 It's not that these are a big problem to create, it's just that they are=




 not

 done, and it tends to turn off people from using D. D is binary API
 compatible with C, but only with a corresponding D import file. This, ou=




t of

 the box, makes D *harder* to use than C.

 Lots of people roll their own, but that work is hard to find and haphaza=




rd.

 This problem keeps coming up again and again.

 So I propose creating, on github.com/D-Programming-Language, a new
 repository called CAPI.

 The CAPI Manifesto
 ------------------

 CAPI is a collection of C header files to publicly available C libraries
 and their translations to D. The idea is that if, in C, to interface to =




a

 library
 one would write:

 =A0 #include "foo.h"

 then the corresponding D code would look like:

 =A0 import foo;

 Each C .h file would have a corresponding .d file. Each C directory woul=




d

 have a corresponding D directory, for example:

 =A0 #include "bar/foo.h" =A0 // C

 =A0 import bar.foo; // D

 The top level directory of each library will have two subdirectories:

 =A0 C/
 =A0 D/

 and there will be a one-to-one correspondence of files and directory
 structure
 between them.

 The D import files will be a rote translation of the corresponding C .h
 file.
 No attempt will be made to fix, improve, or extend the C api. No attempt
 will
 be made to duplicate the C documentation, or replace it in any way. Ther=




e

 will be no unittests. Every effort will be made to avoid needing any D
 specific
 binary files.

 When an updated version of the C header files becomes available, those w=




ill

 get checked into the C subdirectory tree, and then the corresponding D f=




iles

 will get updated.

 Version tags used must match the version tags used by the C API files.

 The license used for the D versions should match the C ones, as they are=




 a

 derived work.
Oct 16 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-17 04:02, Walter Bright wrote:

 Brad and I were talking about some D code that needed openssl support,
 when we ran into the same old problem:

 No D files corresponding to the openssl C .h files.

 It's not that these are a big problem to create, it's just that they are
 not done, and it tends to turn off people from using D. D is binary API
 compatible with C, but only with a corresponding D import file. This,
 out of the box, makes D *harder* to use than C.


I'm working on a Clang based tool for automatically converting C header 
files to D modules.

-- 
/Jacob Carlborg
Oct 17 2011
parent reply Steve Teale <steve.teale britseyeview.com> writes:
 I'm working on a Clang based tool for automatically converting C header
 files to D modules.


Great! Does it work yet?
Oct 17 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-10-17 13:27, Steve Teale wrote:

 I'm working on a Clang based tool for automatically converting C header
 files to D modules.


 Great! Does it work yet?


Well, yes. Some parts of it. I have mostly focused on converting 
Objective-C headers. I have also started to rewrite the tool to use 
libclang instead of embed it straight into clang.

https://github.com/jacob-carlborg/clang
https://github.com/jacob-carlborg/dstep
-- 
/Jacob Carlborg
Oct 17 2011
prev sibling next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On Oct 16, 2011, at 7:02 PM, Walter Bright wrote:

=20
 The CAPI Manifesto
 ------------------
=20
 CAPI is a collection of C header files to publicly available C =


libraries

 and their translations to D. The idea is that if, in C, to interface =


to a library

 one would write:
=20
   #include "foo.h"
=20
 then the corresponding D code would look like:
=20
   import foo;


If the C header file has a name that is a D keyword, an underscore will =
be appended to the D module name.  If a C type name matches a C function =
name (stat), the type name will have a "_t" appended.

There's also the occasional issue of something that doesn't translate =
into D.  As one slightly weird example, some of the the Posix routines =
in OSX have alternates with odd suffixes like "$2003" that are the =
versions which should be called on newer versions of the OS.  I'm still =
not sure of the best way to handle this, since D doesn't have macros.
Oct 17 2011
next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2011-10-17 10:21:45 +0000, Sean Kelly <sean invisibleduck.org> said:


 On Oct 16, 2011, at 7:02 PM, Walter Bright wrote:

 
 The CAPI Manifesto
 ------------------
 
 CAPI is a collection of C header files to publicly available C


 libraries

 and their translations to D. The idea is that if, in C, to interface


 to a library

 one would write:
 
 #include "foo.h"
 
 then the corresponding D code would look like:
 
 import foo;


 
 If the C header file has a name that is a D keyword, an underscore will
 be appended to the D module name.  If a C type name matches a C function
 name (stat), the type name will have a "_t" appended.


Hum, but _t in C stands for typedef. Wouldn't it be better to just 
append an underscore like for module names, that'd make only one rule 
to remember.



 There's also the occasional issue of something that doesn't translate
 into D.  As one slightly weird example, some of the the Posix routines
 in OSX have alternates with odd suffixes like "$2003" that are the
 versions which should be called on newer versions of the OS.  I'm still
 not sure of the best way to handle this, since D doesn't have macros.


I think what D needs to handle that is some pragma to manually specify 
the mangled name of a given function. Why would you need macros?


-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Oct 17 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-17 14:01, Michel Fortin wrote:

 On 2011-10-17 10:21:45 +0000, Sean Kelly <sean invisibleduck.org> said:


 On Oct 16, 2011, at 7:02 PM, Walter Bright wrote:

 The CAPI Manifesto
 ------------------

 CAPI is a collection of C header files to publicly available C


 libraries

 and their translations to D. The idea is that if, in C, to interface


 to a library

 one would write:

 #include "foo.h"

 then the corresponding D code would look like:

 import foo;


 If the C header file has a name that is a D keyword, an underscore will
 be appended to the D module name. If a C type name matches a C function
 name (stat), the type name will have a "_t" appended.


 Hum, but _t in C stands for typedef. Wouldn't it be better to just
 append an underscore like for module names, that'd make only one rule to
 remember.



 There's also the occasional issue of something that doesn't translate
 into D. As one slightly weird example, some of the the Posix routines
 in OSX have alternates with odd suffixes like "$2003" that are the
 versions which should be called on newer versions of the OS. I'm still
 not sure of the best way to handle this, since D doesn't have macros.


 I think what D needs to handle that is some pragma to manually specify
 the mangled name of a given function. Why would you need macros?


Perhaps the macro is used to determine if "foo" or "foo$2003" is 
supposed to be called, based on some condition.

-- 
/Jacob Carlborg
Oct 17 2011
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2011-10-17 13:41:14 +0000, Jacob Carlborg <doob me.com> said:


 On 2011-10-17 14:01, Michel Fortin wrote:

 On 2011-10-17 10:21:45 +0000, Sean Kelly <sean invisibleduck.org> said:
 

 There's also the occasional issue of something that doesn't translate
 into D. As one slightly weird example, some of the the Posix routines
 in OSX have alternates with odd suffixes like "$2003" that are the
 versions which should be called on newer versions of the OS. I'm still
 not sure of the best way to handle this, since D doesn't have macros.


 
 I think what D needs to handle that is some pragma to manually specify
 the mangled name of a given function. Why would you need macros?


 
 Perhaps the macro is used to determine if "foo" or "foo$2003" is 
 supposed to be called, based on some condition.


Indeed. The condition is which OS release you're targeting. That can be 
accomplished today through static ifs. Although it'd be a little more 
verbose since you'd have to repeat the function prototype.

If we had a way to do conditional attributes in D it'd be awesome for 
this use case. It could work this way for instance:

	static if (MAC_OS_X_VERSION_MIN_REQUIRED == 10.5)
		 deprecated_in_os_x_10_5 = deprecated;
	else
		 deprecated_in_os_x_10_5 = /* nothing */;

	 deprecated_in_os_x_10_5
	void some_function_deprecated_in_os_x_10_5();	
Or this way for the special mangled names:

	static if (MAC_OS_X_VERSION_MIN_REQUIRED == 10.5)		 darwin_alias(name) 
= pragma(symbol_name, name ~ "$UNIX2003");
	else
		 darwin_alias(name) = pragma(symbol_name, name);

	 darwin_alias("fwrite")
	size_t fwrite(const void * /*__restrict*/, size_t, size_t, FILE * 
/*__restrict*/);

Internally, when the compiler sees  darwin_alias("fwrite") it just 
replaces it with the attributes  darwin_alias was supposed to be. Note 
that I'm *not* proposing a macro system: this would work at the 
semantic level as a special kind of attribute.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Oct 17 2011
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Well then my vote goes for "let's do it". Simple bindings can be
started right away, probably by copying from dsource bindings and
doing any modifications necessary.

For non-trivial C headers we can discuss them here methinks.
Oct 17 2011
prev sibling parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Michel Fortin" <michel.fortin michelf.com> wrote in message 
news:j7h5gp$2d7n$1 digitalmars.com...

 I think what D needs to handle that is some pragma to manually specify the 
 mangled name of a given function. Why would you need macros?


I've got a patch to do this, in the pragma_mangle branch of my fork.  One 
day I'll get around to fixing it up and making a pull request.
Oct 17 2011
prev sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 17.10.2011 12:21, schrieb Sean Kelly:

 On Oct 16, 2011, at 7:02 PM, Walter Bright wrote:

 The CAPI Manifesto
 ------------------

 CAPI is a collection of C header files to publicly available C libraries
 and their translations to D. The idea is that if, in C, to interface to a
library
 one would write:

    #include "foo.h"

 then the corresponding D code would look like:

    import foo;


 If the C header file has a name that is a D keyword, an underscore will be
appended to the D module name.  If a C type name matches a C function name
(stat), the type name will have a "_t" appended.

 There's also the occasional issue of something that doesn't translate into D. 
As one slightly weird example, some of the the Posix routines in OSX have
alternates with odd suffixes like "$2003" that are the versions which should be
called on newer versions of the OS.  I'm still not sure of the best way to
handle this, since D doesn't have macros.


What about function-like macros, e.g. the Linux/POSIX cmsg stuff 
(CMSG_FIRSTHDR(), CMSG_NXTHDR(), CMSG_LEN() etc) needed to use functions 
like recvmsg() and sendmsg()?

Will there be a direct D translation of the functionality or will they 
be omitted completely?

Cheers,
- Daniel
Oct 17 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/17/2011 1:24 PM, Daniel Gibson wrote:

 What about function-like macros, e.g. the Linux/POSIX cmsg stuff
 (CMSG_FIRSTHDR(), CMSG_NXTHDR(), CMSG_LEN() etc) needed to use functions like
 recvmsg() and sendmsg()?

 Will there be a direct D translation of the functionality or will they be
 omitted completely?


Consider:

    #define FOO(x)  bar((x) + 1)

Do this:

    int FOO()(int x) { return bar(x) + 1; }

Note that it's a function template with no template parameters. This will
enable 
it to be "header only" and not require linking to some library to resolve FOO().
Oct 17 2011
parent reply Peter Alexander <peter.alexander.au gmail.com> writes:
On 17/10/11 10:33 PM, Walter Bright wrote:

 On 10/17/2011 1:24 PM, Daniel Gibson wrote:

 What about function-like macros, e.g. the Linux/POSIX cmsg stuff
 (CMSG_FIRSTHDR(), CMSG_NXTHDR(), CMSG_LEN() etc) needed to use
 functions like
 recvmsg() and sendmsg()?

 Will there be a direct D translation of the functionality or will they be
 omitted completely?


 Consider:

 #define FOO(x) bar((x) + 1)

 Do this:

 int FOO()(int x) { return bar(x) + 1; }

 Note that it's a function template with no template parameters. This
 will enable it to be "header only" and not require linking to some
 library to resolve FOO().


int FOO()(int x) { return bar(x + 1); }

would probably work better :-)

+1 for CAPI btw.
Oct 17 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/17/2011 4:04 PM, Peter Alexander wrote:

 int FOO()(int x) { return bar(x + 1); }

 would probably work better :-)


:-)
Oct 17 2011
prev sibling next sibling parent reply "Regan Heath" <regan netmail.co.nz> writes:
I like it!  :)
Oct 17 2011
next sibling parent mta`chrono <chrono mta-international.net> writes:
+1 Let's do it!
Oct 17 2011
prev sibling parent "Nick Sabalausky" <a a.a> writes:
"Regan Heath" <regan netmail.co.nz> wrote in message 
news:op.v3h9w20554xghj puck.auriga.bhead.co.uk...

I like it!  :)


vote++
Oct 17 2011
prev sibling parent reply so <so so.so> writes:
With D being binary compatible with C, i don't know why we worry on such  
things.
Wasn't being able to access C libraries the point? If it wasn't, what is  
the worthwhile point for this constraint?
Wouldn't (sorry for the poor horse) separate compilers solve the most  
problems (if not all) we face on these issues?
C never changes and every compiler vendor have an implementation.

--
import anyapi; // anyapi would be a D module or a C header (anyapi.h...)  
in directory paths.
--

Structs are pod in both languages.
Matching of the standard types is something we can take care of with  
documentation (RTFM)
and with compiler errors generated (when we call functions from the other  
language).

Sorry once again if this should sound stupid or impossible to implement  
(if so, someone please enlighten me), it probably is because everytime we  
open this discussion i feel i am the only one seeing the big picture, the  
potential of D.

On Mon, 17 Oct 2011 05:02:52 +0300, Walter Bright  
<newshound2 digitalmars.com> wrote:


 Brad and I were talking about some D code that needed openssl support,  
 when we ran into the same old problem:

 No D files corresponding to the openssl C .h files.

 It's not that these are a big problem to create, it's just that they are  
 not done, and it tends to turn off people from using D. D is binary API  
 compatible with C, but only with a corresponding D import file. This,  
 out of the box, makes D *harder* to use than C.

 Lots of people roll their own, but that work is hard to find and  
 haphazard.

 This problem keeps coming up again and again.

 So I propose creating, on github.com/D-Programming-Language, a new  
 repository called CAPI.

 The CAPI Manifesto
 ------------------

 CAPI is a collection of C header files to publicly available C libraries
 and their translations to D. The idea is that if, in C, to interface to  
 a library
 one would write:

     #include "foo.h"

 then the corresponding D code would look like:

     import foo;

 Each C .h file would have a corresponding .d file. Each C directory would
 have a corresponding D directory, for example:

     #include "bar/foo.h"   // C

     import bar.foo; // D

 The top level directory of each library will have two subdirectories:

     C/
     D/

 and there will be a one-to-one correspondence of files and directory  
 structure
 between them.

 The D import files will be a rote translation of the corresponding C .h  
 file.
 No attempt will be made to fix, improve, or extend the C api. No attempt  
 will
 be made to duplicate the C documentation, or replace it in any way. There
 will be no unittests. Every effort will be made to avoid needing any D  
 specific
 binary files.

 When an updated version of the C header files becomes available, those  
 will
 get checked into the C subdirectory tree, and then the corresponding D  
 files
 will get updated.

 Version tags used must match the version tags used by the C API files.

 The license used for the D versions should match the C ones, as they are  
 a
 derived work.
Oct 17 2011
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, October 17, 2011 17:09 so wrote:

 With D being binary compatible with C, i don't know why we worry on such
 things.
 Wasn't being able to access C libraries the point? If it wasn't, what is
 the worthwhile point for this constraint?
 Wouldn't (sorry for the poor horse) separate compilers solve the most
 problems (if not all) we face on these issues?
 C never changes and every compiler vendor have an implementation.
 
 --
 import anyapi; // anyapi would be a D module or a C header (anyapi.h...)
 in directory paths.
 --
 
 Structs are pod in both languages.
 Matching of the standard types is something we can take care of with
 documentation (RTFM)
 and with compiler errors generated (when we call functions from the other
 language).
 
 Sorry once again if this should sound stupid or impossible to implement
 (if so, someone please enlighten me), it probably is because everytime we
 open this discussion i feel i am the only one seeing the big picture, the
 potential of D.


The problem is that for C code to be usable in D code, the C declarations must 
be redone in D, since D can't just include header files. Translating C header 
files to D is a pain and time consuming, and it would benefit us all to have a 
place to go to find common C headers translated to D so that such work doesn't 
have to be duplicated. Hence CAPI has been proposed. And if we're going to 
have it, it also benefits us to be organized about how we lay it out. 
Programmers can name modules in their code whatever they want, but being 
organized about how modules are named and laid out in a large project such as 
this makes it much easier to maintain and find what you want in it.

- Jonathan M Davis
Oct 17 2011
parent reply so <so so.so> writes:
On Tue, 18 Oct 2011 03:32:17 +0300, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:



 The problem is that for C code to be usable in D code, the C  
 declarations must
 be redone in D, since D can't just include header files.


I don't understand why.


 Translating C header
 files to D is a pain and time consuming, and it would benefit us all to  
 have a
 place to go to find common C headers translated to D so that such work  
 doesn't
 have to be duplicated.


With the second compiler, there won't be any need for us to do anything,  
like C++ we could use C libraries seamlessly.
Oct 17 2011
parent reply "Nick Sabalausky" <a a.a> writes:
"so" <so so.so> wrote in message 
news:op.v3ivsvb8mpw3zg localhost.localdomain...

 On Tue, 18 Oct 2011 03:32:17 +0300, Jonathan M Davis <jmdavisProg gmx.com> 
 wrote:


 Translating C header
 files to D is a pain and time consuming, and it would benefit us all to 
 have a
 place to go to find common C headers translated to D so that such work 
 doesn't
 have to be duplicated.


 With the second compiler, there won't be any need for us to do anything, 
 like C++ we could use C libraries seamlessly.


The only reason C++ is able to do that is because C++ (realistically, even 
if not *technically*) is a proper superset of C. D isn't.
Oct 19 2011
parent reply so <so so.so> writes:
On Thu, 20 Oct 2011 00:26:58 +0300, Nick Sabalausky <a a.a> wrote:


 "so" <so so.so> wrote in message
 news:op.v3ivsvb8mpw3zg localhost.localdomain...

 On Tue, 18 Oct 2011 03:32:17 +0300, Jonathan M Davis  
 <jmdavisProg gmx.com>
 wrote:


 Translating C header
 files to D is a pain and time consuming, and it would benefit us all to
 have a
 place to go to find common C headers translated to D so that such work
 doesn't
 have to be duplicated.


 With the second compiler, there won't be any need for us to do anything,
 like C++ we could use C libraries seamlessly.


 The only reason C++ is able to do that is because C++ (realistically,  
 even
 if not *technically*) is a proper superset of C. D isn't.


Right, but more importantly it is ABI compatible which is what D also has,  
this opens some doors.
For most C libraries if you exclude macros, you just fill structs, and  
call functions.
Oct 21 2011
parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
That's ALL you can do in C. fill structs and call functions
(fundamental type manipulation doesn't count).
My personal research shows the following use cases of C macros (sorted
by popularity in descending order):
1. enum
2. alias (most notably, conditionally compiled ones)
3. CTFE function
4. mixin template
5. syntactic alias
6. syntactic mixin template

only the last 2 out of 6 cannot be translated to D.
An example of a syntactic alias is this very common piece of C code:
#ifdef __VERY_VERY_OLD_C_COMPILER__
    #define CONST
#else
    #define CONST const
#endif

An example of a syntactic mixin template is this piece of code, which
i never actually saw anywhere (possible only in C99 and C++):
#define N_TIMES(n) for(int i = 0; i != n; ++i)

The last use case is very rare. The only legitimate example i ever saw
is in libjpeg, where a macro is used to define function pointers of
API functions.
The use case before that is mostly used for portability reasons, which
is not necessary in D.
Some non-standard extension encapsulating macros are almost always
used in C libraries, which can be removed altogether.

The translation can go on regarding the above use cases and the last
two cases can be evaluated in-line, commented out and warned about for
manual translation.

On Fri, Oct 21, 2011 at 11:48 AM, so <so so.so> wrote:

 On Thu, 20 Oct 2011 00:26:58 +0300, Nick Sabalausky <a a.a> wrote:


 "so" <so so.so> wrote in message
 news:op.v3ivsvb8mpw3zg localhost.localdomain...

 On Tue, 18 Oct 2011 03:32:17 +0300, Jonathan M Davis
 <jmdavisProg gmx.com>
 wrote:


 Translating C header
 files to D is a pain and time consuming, and it would benefit us all to
 have a
 place to go to find common C headers translated to D so that such work
 doesn't
 have to be duplicated.


 With the second compiler, there won't be any need for us to do anything,
 like C++ we could use C libraries seamlessly.


 The only reason C++ is able to do that is because C++ (realistically, even
 if not *technically*) is a proper superset of C. D isn't.


 Right, but more importantly it is ABI compatible which is what D also has,
 this opens some doors.
 For most C libraries if you exclude macros, you just fill structs, and call
 functions.
Oct 21 2011
next sibling parent reply so <so so.so> writes:
Indeed, macros is a language in itself.
Then again it all boils down to type manipulation and function calls.
Not sure if it is doable but a special operator like "__cmacro" might be  
an answer.

#define FUN(a, b) ....
#define DATA ....

could be accessed like:

__cmacro(FUN, a, b);
__cmacro(DATA);

I am pushing this because the outcome well worths all these ugly hacks.

On Fri, 21 Oct 2011 11:32:32 +0300, Gor Gyolchanyan  
<gor.f.gyolchanyan gmail.com> wrote:


 That's ALL you can do in C. fill structs and call functions
 (fundamental type manipulation doesn't count).
 My personal research shows the following use cases of C macros (sorted
 by popularity in descending order):
 1. enum
 2. alias (most notably, conditionally compiled ones)
 3. CTFE function
 4. mixin template
 5. syntactic alias
 6. syntactic mixin template

 only the last 2 out of 6 cannot be translated to D.
 An example of a syntactic alias is this very common piece of C code:
 #ifdef __VERY_VERY_OLD_C_COMPILER__
     #define CONST
 #else
     #define CONST const
 #endif

 An example of a syntactic mixin template is this piece of code, which
 i never actually saw anywhere (possible only in C99 and C++):
 #define N_TIMES(n) for(int i = 0; i != n; ++i)

 The last use case is very rare. The only legitimate example i ever saw
 is in libjpeg, where a macro is used to define function pointers of
 API functions.
 The use case before that is mostly used for portability reasons, which
 is not necessary in D.
 Some non-standard extension encapsulating macros are almost always
 used in C libraries, which can be removed altogether.

 The translation can go on regarding the above use cases and the last
 two cases can be evaluated in-line, commented out and warned about for
 manual translation.
Oct 21 2011
parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
This will defeat the philosophy of D, which stands for core
correctness, simplicity, maintainability and flexibility.
A much better solution would be to implement the AST macros, which
were discussed in a video-talk a long time ago by Walter and Andrei,
for which the macro keyword was reserved.
After that, all C macros will be translatable.

On Fri, Oct 21, 2011 at 2:17 PM, so <so so.so> wrote:

 Indeed, macros is a language in itself.
 Then again it all boils down to type manipulation and function calls.
 Not sure if it is doable but a special operator like "__cmacro" might be =


an

 answer.

 #define FUN(a, b) ....
 #define DATA ....

 could be accessed like:

 __cmacro(FUN, a, b);
 __cmacro(DATA);

 I am pushing this because the outcome well worths all these ugly hacks.

 On Fri, 21 Oct 2011 11:32:32 +0300, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:


 That's ALL you can do in C. fill structs and call functions
 (fundamental type manipulation doesn't count).
 My personal research shows the following use cases of C macros (sorted
 by popularity in descending order):
 1. enum
 2. alias (most notably, conditionally compiled ones)
 3. CTFE function
 4. mixin template
 5. syntactic alias
 6. syntactic mixin template

 only the last 2 out of 6 cannot be translated to D.
 An example of a syntactic alias is this very common piece of C code:
 #ifdef __VERY_VERY_OLD_C_COMPILER__
 =A0 =A0#define CONST
 #else
 =A0 =A0#define CONST const
 #endif

 An example of a syntactic mixin template is this piece of code, which
 i never actually saw anywhere (possible only in C99 and C++):
 #define N_TIMES(n) for(int i =3D 0; i !=3D n; ++i)

 The last use case is very rare. The only legitimate example i ever saw
 is in libjpeg, where a macro is used to define function pointers of
 API functions.
 The use case before that is mostly used for portability reasons, which
 is not necessary in D.
 Some non-standard extension encapsulating macros are almost always
 used in C libraries, which can be removed altogether.

 The translation can go on regarding the above use cases and the last
 two cases can be evaluated in-line, commented out and warned about for
 manual translation.
Oct 21 2011
parent Don <nospam nospam.com> writes:
On 21.10.2011 13:42, Gor Gyolchanyan wrote:

 This will defeat the philosophy of D, which stands for core
 correctness, simplicity, maintainability and flexibility.
 A much better solution would be to implement the AST macros, which
 were discussed in a video-talk a long time ago by Walter and Andrei,
 for which the macro keyword was reserved.
 After that, all C macros will be translatable.


Unfortunately, the AST macros described in the conference video don't 
work (they are *far* too weak). Nobody has ever come up with a 
replacement proposal.
Oct 23 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-21 10:32, Gor Gyolchanyan wrote:

 That's ALL you can do in C. fill structs and call functions
 (fundamental type manipulation doesn't count).
 My personal research shows the following use cases of C macros (sorted
 by popularity in descending order):
 1. enum
 2. alias (most notably, conditionally compiled ones)
 3. CTFE function
 4. mixin template
 5. syntactic alias
 6. syntactic mixin template


I guess it's quite difficult for a compiler to recognize the differences 
between these use cases.


 only the last 2 out of 6 cannot be translated to D.
 An example of a syntactic alias is this very common piece of C code:
 #ifdef __VERY_VERY_OLD_C_COMPILER__
      #define CONST
 #else
      #define CONST const
 #endif

 An example of a syntactic mixin template is this piece of code, which
 i never actually saw anywhere (possible only in C99 and C++):
 #define N_TIMES(n) for(int i = 0; i != n; ++i)

 The last use case is very rare. The only legitimate example i ever saw
 is in libjpeg, where a macro is used to define function pointers of
 API functions.
 The use case before that is mostly used for portability reasons, which
 is not necessary in D.
 Some non-standard extension encapsulating macros are almost always
 used in C libraries, which can be removed altogether.

 The translation can go on regarding the above use cases and the last
 two cases can be evaluated in-line, commented out and warned about for
 manual translation.


Something similar is used in the Boost library for its "foreach" macro.

-- 
/Jacob Carlborg
Oct 21 2011
parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
 I guess it's quite difficult for a compiler to recognize the differences
between these use cases.


Well, that's because C macros suck big-time.
I don't see any other solution to the C-to-D translation problem.
Oct 21 2011
prev sibling parent reply "Marco Leise" <Marco.Leise gmx.de> writes:
Am 21.10.2011, 10:32 Uhr, schrieb Gor Gyolchanyan  
<gor.f.gyolchanyan gmail.com>:


 That's ALL you can do in C. fill structs and call functions
 (fundamental type manipulation doesn't count).
 My personal research shows the following use cases of C macros (sorted
 by popularity in descending order):
 1. enum
 2. alias (most notably, conditionally compiled ones)
 3. CTFE function
 4. mixin template
 5. syntactic alias
 6. syntactic mixin template

 only the last 2 out of 6 cannot be translated to D.
 An example of a syntactic alias is this very common piece of C code:
 #ifdef __VERY_VERY_OLD_C_COMPILER__
     #define CONST
 #else
     #define CONST const
 #endif


Maybe these cases can often be solved when, as you say, they are for  
compatibility with other compilers. The header converter would simply  
assume to be the latest and greatest of the known compilers and evaluate  
the code like this:
- there is an #ifdef
- if it a define from the list of "known C compilers with quirks"
   - jump right to the else block
The #define obviously can become more than one thing in the .d file. But  
that is determined by the instantiation site. The above CONST would  
probably be ignored when it is used in a const parameter declaration,  
because of D's transitivity and become 'immutable __gshared' when used on  
a global variable. Do you get the idea?
Oct 24 2011
next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
Sure. You're right. Weird macros are often used for handy shortcuts
(like the aforementioned for loop header), which are not handy enough
to invent a just-as-weird way of translating them. Everything else is
either for compiler extensions (which either have built-in support in
D or are simply nit translatable) or hacks for compile-time activity
(can be replaced by more intuitive counterparts in D).
Translating _any_ C code is impossible. The C preprocessor cannot be
fully simulated in D. But translating _this_ C code is not as hard,
because it's a bad practice to abuse macros in C and no good C library
would do that.
The translator should not be a "_any_ C code translator", it should be
an "arbitrary _this_ C code translator", which makes a few assumptions
and does not guarantee 100% convertibility of the code.
The most important part of the translator should be the problem
reporter, which should point you to all the suspicious and
untranslated parts of the C code, so you can deal with them manually.

On Tue, Oct 25, 2011 at 6:36 AM, Marco Leise <Marco.Leise gmx.de> wrote:

 Am 21.10.2011, 10:32 Uhr, schrieb Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com>:


 That's ALL you can do in C. fill structs and call functions
 (fundamental type manipulation doesn't count).
 My personal research shows the following use cases of C macros (sorted
 by popularity in descending order):
 1. enum
 2. alias (most notably, conditionally compiled ones)
 3. CTFE function
 4. mixin template
 5. syntactic alias
 6. syntactic mixin template

 only the last 2 out of 6 cannot be translated to D.
 An example of a syntactic alias is this very common piece of C code:
 #ifdef __VERY_VERY_OLD_C_COMPILER__
 =A0 =A0#define CONST
 #else
 =A0 =A0#define CONST const
 #endif


 Maybe these cases can often be solved when, as you say, they are for
 compatibility with other compilers. The header converter would simply ass=


ume

 to be the latest and greatest of the known compilers and evaluate the cod=


e

 like this:
 - there is an #ifdef
 - if it a define from the list of "known C compilers with quirks"
 =A0- jump right to the else block
 The #define obviously can become more than one thing in the .d file. But
 that is determined by the instantiation site. The above CONST would proba=


bly

 be ignored when it is used in a const parameter declaration, because of D=


's

 transitivity and become 'immutable __gshared' when used on a global
 variable. Do you get the idea?
Oct 25 2011
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
In the end, I'm sure, that manual intervention won't be necessary and
the CAPI repository will be replaced by std.c.translate or something
like that.
You'd just do this:

mixin cInclude("sqlite3.h");

and voila.

The mixin will print out all macros, which i couldn't translate and
had to evaluate in-line. If some of those macros are important to the
user, then translating a C header would consist of including the
header and adding the missing macros.

I think this is a much better idea, then manually translating
everything, because this is cheaper, covers more libraries.

On Tue, Oct 25, 2011 at 12:13 PM, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:

 Sure. You're right. Weird macros are often used for handy shortcuts
 (like the aforementioned for loop header), which are not handy enough
 to invent a just-as-weird way of translating them. Everything else is
 either for compiler extensions (which either have built-in support in
 D or are simply nit translatable) or hacks for compile-time activity
 (can be replaced by more intuitive counterparts in D).
 Translating _any_ C code is impossible. The C preprocessor cannot be
 fully simulated in D. But translating _this_ C code is not as hard,
 because it's a bad practice to abuse macros in C and no good C library
 would do that.
 The translator should not be a "_any_ C code translator", it should be
 an "arbitrary _this_ C code translator", which makes a few assumptions
 and does not guarantee 100% convertibility of the code.
 The most important part of the translator should be the problem
 reporter, which should point you to all the suspicious and
 untranslated parts of the C code, so you can deal with them manually.

 On Tue, Oct 25, 2011 at 6:36 AM, Marco Leise <Marco.Leise gmx.de> wrote:

 Am 21.10.2011, 10:32 Uhr, schrieb Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com>:


 That's ALL you can do in C. fill structs and call functions
 (fundamental type manipulation doesn't count).
 My personal research shows the following use cases of C macros (sorted
 by popularity in descending order):
 1. enum
 2. alias (most notably, conditionally compiled ones)
 3. CTFE function
 4. mixin template
 5. syntactic alias
 6. syntactic mixin template

 only the last 2 out of 6 cannot be translated to D.
 An example of a syntactic alias is this very common piece of C code:
 #ifdef __VERY_VERY_OLD_C_COMPILER__
 =A0 =A0#define CONST
 #else
 =A0 =A0#define CONST const
 #endif


 Maybe these cases can often be solved when, as you say, they are for
 compatibility with other compilers. The header converter would simply as=




sume

 to be the latest and greatest of the known compilers and evaluate the co=




de

 like this:
 - there is an #ifdef
 - if it a define from the list of "known C compilers with quirks"
 =A0- jump right to the else block
 The #define obviously can become more than one thing in the .d file. But
 that is determined by the instantiation site. The above CONST would prob=




ably

 be ignored when it is used in a const parameter declaration, because of =




D's

 transitivity and become 'immutable __gshared' when used on a global
 variable. Do you get the idea?
Oct 25 2011
prev sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
The CAPI repository could be replaced by a package in Phobos with
aforementioned tiny modules with a single cImport and some additions.
They would be very small and conveniently available from Phobos.

On Tue, Oct 25, 2011 at 12:26 PM, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:

 In the end, I'm sure, that manual intervention won't be necessary and
 the CAPI repository will be replaced by std.c.translate or something
 like that.
 You'd just do this:

 mixin cInclude("sqlite3.h");

 and voila.

 The mixin will print out all macros, which i couldn't translate and
 had to evaluate in-line. If some of those macros are important to the
 user, then translating a C header would consist of including the
 header and adding the missing macros.

 I think this is a much better idea, then manually translating
 everything, because this is cheaper, covers more libraries.

 On Tue, Oct 25, 2011 at 12:13 PM, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:

 Sure. You're right. Weird macros are often used for handy shortcuts
 (like the aforementioned for loop header), which are not handy enough
 to invent a just-as-weird way of translating them. Everything else is
 either for compiler extensions (which either have built-in support in
 D or are simply nit translatable) or hacks for compile-time activity
 (can be replaced by more intuitive counterparts in D).
 Translating _any_ C code is impossible. The C preprocessor cannot be
 fully simulated in D. But translating _this_ C code is not as hard,
 because it's a bad practice to abuse macros in C and no good C library
 would do that.
 The translator should not be a "_any_ C code translator", it should be
 an "arbitrary _this_ C code translator", which makes a few assumptions
 and does not guarantee 100% convertibility of the code.
 The most important part of the translator should be the problem
 reporter, which should point you to all the suspicious and
 untranslated parts of the C code, so you can deal with them manually.

 On Tue, Oct 25, 2011 at 6:36 AM, Marco Leise <Marco.Leise gmx.de> wrote:

 Am 21.10.2011, 10:32 Uhr, schrieb Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com>:


 That's ALL you can do in C. fill structs and call functions
 (fundamental type manipulation doesn't count).
 My personal research shows the following use cases of C macros (sorted
 by popularity in descending order):
 1. enum
 2. alias (most notably, conditionally compiled ones)
 3. CTFE function
 4. mixin template
 5. syntactic alias
 6. syntactic mixin template

 only the last 2 out of 6 cannot be translated to D.
 An example of a syntactic alias is this very common piece of C code:
 #ifdef __VERY_VERY_OLD_C_COMPILER__
 =A0 =A0#define CONST
 #else
 =A0 =A0#define CONST const
 #endif


 Maybe these cases can often be solved when, as you say, they are for
 compatibility with other compilers. The header converter would simply a=






ssume

 to be the latest and greatest of the known compilers and evaluate the c=






ode

 like this:
 - there is an #ifdef
 - if it a define from the list of "known C compilers with quirks"
 =A0- jump right to the else block
 The #define obviously can become more than one thing in the .d file. Bu=






t

 that is determined by the instantiation site. The above CONST would pro=






bably

 be ignored when it is used in a const parameter declaration, because of=






 D's

 transitivity and become 'immutable __gshared' when used on a global
 variable. Do you get the idea?
Oct 25 2011
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"so" <so so.so> wrote in message 
news:op.v3itelnbmpw3zg localhost.localdomain...

 With D being binary compatible with C, i don't know why we worry on such 
 things.
 Wasn't being able to access C libraries the point? If it wasn't, what is 
 the worthwhile point for this constraint?
 Wouldn't (sorry for the poor horse) separate compilers solve the most 
 problems (if not all) we face on these issues?
 C never changes and every compiler vendor have an implementation.

 --
 import anyapi; // anyapi would be a D module or a C header (anyapi.h...) 
 in directory paths.
 --


Thats would mean that every D compiler would have to *also* be a C compiler.
Oct 17 2011
parent reply so <so so.so> writes:
On Tue, 18 Oct 2011 03:52:13 +0300, Nick Sabalausky <a a.a> wrote:


 Thats would mean that every D compiler would have to *also* be a C  
 compiler.


Indeed, but i see nothing wrong with it, like i see nothing wrong with  
inline asm, C never changes,
Oct 17 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/17/2011 5:56 PM, so wrote:

 On Tue, 18 Oct 2011 03:52:13 +0300, Nick Sabalausky <a a.a> wrote:


 Thats would mean that every D compiler would have to *also* be a C compiler.


 Indeed, but i see nothing wrong with it, like i see nothing wrong with inline
 asm, C never changes,


While C code can be directly translated to D, the C macros are another matter.
Oct 17 2011
next sibling parent reply so <so so.so> writes:
On Tue, 18 Oct 2011 06:01:37 +0300, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 10/17/2011 5:56 PM, so wrote:

 On Tue, 18 Oct 2011 03:52:13 +0300, Nick Sabalausky <a a.a> wrote:


 Thats would mean that every D compiler would have to *also* be a C  
 compiler.


 Indeed, but i see nothing wrong with it, like i see nothing wrong with  
 inline
 asm, C never changes,


 While C code can be directly translated to D, the C macros are another  
 matter.


You are right, i forgot about macros, Is it only this or is there anything  
else?
Oct 21 2011
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2011 12:41 AM, so wrote:

 You are right, i forgot about macros, Is it only this or is there anything
else?


The only other thing is what does one do about 'char' - make it a byte, ubyte, 
or char D type?
Oct 21 2011
parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Sat, 22 Oct 2011 04:34:59 +0200, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 10/21/2011 12:41 AM, so wrote:

 You are right, i forgot about macros, Is it only this or is there  
 anything else?


 The only other thing is what does one do about 'char' - make it a byte,  
 ubyte, or char D type?


One should not loose the implicit conversion of string literals to
const char pointers. So the answer might be to use char where something
is used in a string context and (u)byte where it's not. In most cases this  
is
a no-brainer.

martin
Oct 24 2011
prev sibling parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Walter Bright wrote:

 On 10/17/2011 5:56 PM, so wrote:

 On Tue, 18 Oct 2011 03:52:13 +0300, Nick Sabalausky <a a.a> wrote:


 Thats would mean that every D compiler would have to *also* be a C
 compiler.


 Indeed, but i see nothing wrong with it, like i see nothing wrong with
 inline
 asm, C never changes,


 While C code can be directly translated to D, the C macros are another
 matter.


One way is to use "probably number one C/C++ preprocessor now available 
in the world.": http://mcpp.sourceforge.net/

It's portable, BSD licensed and implements all of C90, C99 and C++98 
specifications.
Oct 21 2011
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2011 5:29 AM, Piotr Szturmaj wrote:

 It's portable, BSD licensed and implements all of C90, C99 and C++98
 specifications.


Preprocessing the text is not the problem. The problem is determining a D 
translation of the macros.
Oct 21 2011