www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Distributing libraries made with D

reply Niko Korhonen <niktheblak hotmail.com> writes:
Greetings, I'm a newcomer in D and I've been very happy with the 
language so far. There's one topic that I can't figure out though; how 
am I supposed to distribute libraries for D.

Let's suppose I want to program a /closed-source/ library for D. People 
should be able to use that library but without access to the source code.

When using Java I can provide a .jar package with the necessary .class 
files. Users can can import this package and specify the .jar file as a 
compiler argument.

In .NET I provide the generated .dll file and the .xml documentation 
file. These can be plugged in and used in Visual Studio very easily.

In C/C++ I provide the binaries (.obj, .a, .lib etc) and the header 
files. No source code required so far (although Bytecode and MSIL are 
easy to decompile).

It seems that the D compiler requires the full source code for D 
libraries. Whenever I import a package in D source code as such:



The compiler wants mylib.d. Passing mylib.obj or mylib.lib as argument 
doesn't do the trick. Am I missing something here? Since I want to 
provide directly usable libraries for D I really wouldn't like to limit 
myself to that export(C) assembler stuff.

So can I build full-featured (not exported) libraries with D, for D, 
without opening the source?
Jul 28 2004
next sibling parent reply Derek Parnell <derek psych.ward> writes:
On Thu, 29 Jul 2004 02:41:30 +0300, Niko Korhonen wrote:


 Greetings, I'm a newcomer in D and I've been very happy with the 
 language so far. There's one topic that I can't figure out though; how 
 am I supposed to distribute libraries for D.
 
 Let's suppose I want to program a /closed-source/ library for D. People 
 should be able to use that library but without access to the source code.
 
 When using Java I can provide a .jar package with the necessary .class 
 files. Users can can import this package and specify the .jar file as a 
 compiler argument.
 
 In .NET I provide the generated .dll file and the .xml documentation 
 file. These can be plugged in and used in Visual Studio very easily.
 
 In C/C++ I provide the binaries (.obj, .a, .lib etc) and the header 
 files. No source code required so far (although Bytecode and MSIL are 
 easy to decompile).
 
 It seems that the D compiler requires the full source code for D 
 libraries. Whenever I import a package in D source code as such:
 

 
 The compiler wants mylib.d. Passing mylib.obj or mylib.lib as argument 
 doesn't do the trick. Am I missing something here? Since I want to 
 provide directly usable libraries for D I really wouldn't like to limit 
 myself to that export(C) assembler stuff.
 
 So can I build full-featured (not exported) libraries with D, for D, 
 without opening the source?


It can be done! 

I worked it out with the help of a few people here. I was *not* intuitive
and involved a bit of /smoke & mirrors/. Unfortunately, I can't remember
all the steps involved. I'll try to nut it out again and this time write
down the process. ;-)

-- 
Derek
Melbourne, Australia
29/Jul/04 10:18:26 AM
Jul 28 2004
parent Derek Parnell <derek psych.ward> writes:
On Thu, 29 Jul 2004 10:21:32 +1000, Derek Parnell wrote:


 On Thu, 29 Jul 2004 02:41:30 +0300, Niko Korhonen wrote:
 

 Greetings, I'm a newcomer in D and I've been very happy with the 
 language so far. There's one topic that I can't figure out though; how 
 am I supposed to distribute libraries for D.
 
 Let's suppose I want to program a /closed-source/ library for D. People 
 should be able to use that library but without access to the source code.
 
 When using Java I can provide a .jar package with the necessary .class 
 files. Users can can import this package and specify the .jar file as a 
 compiler argument.
 
 In .NET I provide the generated .dll file and the .xml documentation 
 file. These can be plugged in and used in Visual Studio very easily.
 
 In C/C++ I provide the binaries (.obj, .a, .lib etc) and the header 
 files. No source code required so far (although Bytecode and MSIL are 
 easy to decompile).
 
 It seems that the D compiler requires the full source code for D 
 libraries. Whenever I import a package in D source code as such:
 

 
 The compiler wants mylib.d. Passing mylib.obj or mylib.lib as argument 
 doesn't do the trick. Am I missing something here? Since I want to 
 provide directly usable libraries for D I really wouldn't like to limit 
 myself to that export(C) assembler stuff.
 
 So can I build full-featured (not exported) libraries with D, for D, 
 without opening the source?


 
 It can be done! 
 
 I worked it out with the help of a few people here. I was *not* intuitive
 and involved a bit of /smoke & mirrors/. Unfortunately, I can't remember
 all the steps involved. I'll try to nut it out again and this time write
 down the process. ;-)


Okay. This is what I did...

** Created a folder called 'IMP'
** Stored my source code in here. These were implementation code, and not
'headers'.
   example:
   --- file: mfoo.d ---
   module mfoo;
   import std.stdio;

   class Foo
   {
    void foo(int x){ writef("FOO\n");}
   }

   --- file: mbar.d ---
   module mbar;
   import std.stdio;
   import mfoo;
   class Bar: Foo
   {
    alias Foo.foo foo;
    void foo(char[] x){writef("BAR\n");};
   }

** Compile these with the -c switch.
** Move the resulting *.obj files into a library file (eg. via the
DigitalMars lib program) 
** Copy the source files. eg. mfoo.d --> mfoo_exp.d mbar.d --> mbar_exp.d
** Edit the _exp.d files and remove all the implementation stuff.
   example:
   --- file: mfoo_exp.d ---
   module mfoo;
   import std.stdio;

   class Foo
   {
    void foo(int x);
   }

   --- file: mbar.d ---
   module mbar;
   import std.stdio;
   import mfoo;
   class Bar: Foo
   {
    alias Foo.foo foo;
    void foo(char[] x);
   }
** Create another folder. eg. EXPORT
** Copy the library from IMP to EXPORT
** Copy the _exp.d files from IMP to EXPORT
** Rename the copied _exp.d files in EXPORT to their .d names (remove the
_exp) part.
** The EXPORT folder is now your distribution package.

To use the distribution package. Create your application code as normal and
compile it using the .d files and .lib file from the distribution. 
   example:
   --- file: test.d ---
   import mfoo;
   import mbar;
   void main()
   {
    Foo f = new Foo;
    Bar b = new Bar;
    int i;
    char[] s = "test";

    b.foo(i);
    b.foo(s);
   }
   --------------------------

and compiled with ...
 
   dmd test mylib.lib


So the net result is that the user of your library only gets the .LIB file
and the stripped down source code files, containing definitions but no
implementation details.

-- 
Derek
Melbourne, Australia
29/Jul/04 11:13:49 AM
Jul 28 2004
prev sibling next sibling parent "Vathix" <vathixSpamFix dprogramming.com> writes:
"Niko Korhonen" <niktheblak hotmail.com> wrote in message
news:ce9dgi$phm$1 digitaldaemon.com...

 Greetings, I'm a newcomer in D and I've been very happy with the
 language so far. There's one topic that I can't figure out though; how
 am I supposed to distribute libraries for D.

 Let's suppose I want to program a /closed-source/ library for D. People
 should be able to use that library but without access to the source code.

 When using Java I can provide a .jar package with the necessary .class
 files. Users can can import this package and specify the .jar file as a
 compiler argument.

 In .NET I provide the generated .dll file and the .xml documentation
 file. These can be plugged in and used in Visual Studio very easily.

 In C/C++ I provide the binaries (.obj, .a, .lib etc) and the header
 files. No source code required so far (although Bytecode and MSIL are
 easy to decompile).

 It seems that the D compiler requires the full source code for D
 libraries. Whenever I import a package in D source code as such:



 The compiler wants mylib.d. Passing mylib.obj or mylib.lib as argument
 doesn't do the trick. Am I missing something here? Since I want to
 provide directly usable libraries for D I really wouldn't like to limit
 myself to that export(C) assembler stuff.

 So can I build full-featured (not exported) libraries with D, for D,
 without opening the source?


Yes, make a stripped D file just for importing. Given the actual source file
is foo.d, make foo_imp.d:

module foo; // Not foo_imp.
class Foo
{
   int publicfield;
   void stuff();
}
int bar();
// etc ,,..,
Jul 28 2004
prev sibling next sibling parent Andy Friesen <andy ikagames.com> writes:
Niko Korhonen wrote:

 Greetings, I'm a newcomer in D and I've been very happy with the 
 language so far. There's one topic that I can't figure out though; how 
 am I supposed to distribute libraries for D.
 
 Let's suppose I want to program a /closed-source/ library for D. People 
 should be able to use that library but without access to the source code.
 
 So can I build full-featured (not exported) libraries with D, for D, 
 without opening the source?


The D compiler needs declarations, but doesn't care about definitions.

Except for templates, you can just leave out the definition:

     class Foo : Bar {
         this();
         void thing();
         int x;
         etc();
     }

     int nonmember_function();

The linker will sort it all out.

  -- andy
Jul 28 2004
prev sibling next sibling parent reply "Walter" <newshound digitalmars.com> writes:
"Niko Korhonen" <niktheblak hotmail.com> wrote in message
news:ce9dgi$phm$1 digitaldaemon.com...

 So can I build full-featured (not exported) libraries with D, for D,
 without opening the source?


Sure! Look at phobos/object.d and phobos/internal/object.d for an example.
The trick is to remove the function bodies from the shipped version of the
source, and leave the bodies in for the compiled library that you ship.
Jul 28 2004
parent reply David Tiktin <dtiktin nospam.totally-bogus.com> writes:
On 28 Jul 2004, "Walter" <newshound digitalmars.com> wrote:


 "Niko Korhonen" <niktheblak hotmail.com> wrote in message
 news:ce9dgi$phm$1 digitaldaemon.com...

 So can I build full-featured (not exported) libraries with D, for
 D, without opening the source?


 
 Sure! Look at phobos/object.d and phobos/internal/object.d for an
 example. The trick is to remove the function bodies from the
 shipped version of the source, and leave the bodies in for the
 compiled library that you ship. 


I think this is an important issue and not just for keeping source 
code closed.  One feature I have not yet found in D is a clean 
solution for separating interface from implementation in the way C 
prototypes and forward declarations do.  So far, this method of 
stripping out the implementation seems to be the *only* offered 
solution, and it seems kludgy.

Would it be possible to do something like this?  A module could 
(optionally) use 2 files, say x.d and x.i.  x.d would be the 
"stripped" interface file.  x.i would be the "full" implementation 
file.  x.d would have a statement like: 

module x;
implementation x;

When building the object x.obj, the compiler would "bring in" the 
implementation and compile the "combined" file.  When compiling any 
other object using "import x" it would not.  It would just see the .d 
interface file.  You'd just ship the .lib containing the .obj and .d 
files.  Just a thought.

BTW, do you have a tool or script of some kind that strips the 
function bodies?  It looks to me like that might be hard to do 
reliably without actually parsing the code to distinguish function 
bodies from other types of blocks.

Dave

-- 
D.a.v.i.d  T.i.k.t.i.n
t.i.k.t.i.n [at] a.d.v.a.n.c.e.d.r.e.l.a.y [dot] c.o.m
Jul 29 2004
next sibling parent h3r3tic <h3r3tic_member pathlink.com> writes:
I think this is an important issue and not just for keeping source 
code closed.  One feature I have not yet found in D is a clean 
solution for separating interface from implementation in the way C 
prototypes and forward declarations do.  So far, this method of 
stripping out the implementation seems to be the *only* offered 
solution, and it seems kludgy.


Exactly the thing I've been thinking about recently.


  x.i would be the "full" implementation 
file.  x.d would have a statement like: 

module x;
implementation x;


That I haven't thought about ;)
It might get better:

module implementation x;

and I think it wouldn't be too painful to parse if that's concerned
Jul 29 2004
prev sibling next sibling parent reply Ilya Minkov <minkov cs.tum.edu> writes:
David Tiktin schrieb:


 I think this is an important issue and not just for keeping source 
 code closed.  One feature I have not yet found in D is a clean 
 solution for separating interface from implementation in the way C 
 prototypes and forward declarations do.  So far, this method of 
 stripping out the implementation seems to be the *only* offered 
 solution, and it seems kludgy.


It's really nice for development to have it together. Like, there are 
different opinions, and i do program in C++ primarily, but i still feel 
much more comfortable with both implementation and declarations in one 
file. Nice for people coming from Delphi (like myself) or Java originally.

What else can probably be done, make 2 sections within one file, the 
upper for (public) declarations, the lower for implementation. I don't 
know whether it always works, but i think it should, so it's worth a 
try. And where it doesn't, report a bug.


 Would it be possible to do something like this?  A module could 
 (optionally) use 2 files, say x.d and x.i.  x.d would be the 
 "stripped" interface file.  x.i would be the "full" implementation 
 file.  x.d would have a statement like: 


Yikes! It was such a relief that D doesn't force one to do that. but... 
you really want it back?

Walters opinion to this is, is that interface mixed with implementation 
is OK, but one needs some documentation tool. Nice HTMLs are far easier 
to read than stripped source or headers.


 module x;
 implementation x;
 
 When building the object x.obj, the compiler would "bring in" the 
 implementation and compile the "combined" file.  When compiling any 
 other object using "import x" it would not.  It would just see the .d 
 interface file.  You'd just ship the .lib containing the .obj and .d 
 files.  Just a thought.


If you can use separate interface from implementation as above, one 
could expand on it. For example, by using a version statement and a 
simple tool to strip out the contents of it, or by chaining the 
preprocessor into the build process.


 BTW, do you have a tool or script of some kind that strips the 
 function bodies?  It looks to me like that might be hard to do 
 reliably without actually parsing the code to distinguish function 
 bodies from other types of blocks.


I recall dig compiler driver (look for undig at dsource) would usually 
be able to do the trick, but someone was working on a better utility.

-eye
Jul 29 2004
next sibling parent reply David Tiktin <dtiktin nospam.totally-bogus.com> writes:
On 29 Jul 2004, Ilya Minkov <minkov cs.tum.edu> wrote:


 David Tiktin schrieb:
 

 I think this is an important issue and not just for keeping
 source code closed.  One feature I have not yet found in D is a
 clean solution for separating interface from implementation in
 the way C prototypes and forward declarations do.  So far, this
 method of stripping out the implementation seems to be the *only*
 offered solution, and it seems kludgy.


 
 It's really nice for development to have it together. Like, there
 are different opinions, and i do program in C++ primarily, but i
 still feel much more comfortable with both implementation and
 declarations in one file. Nice for people coming from Delphi (like
 myself) or Java originally. 
 
 What else can probably be done, make 2 sections within one file,
 the upper for (public) declarations, the lower for implementation.
 I don't know whether it always works, but i think it should, so
 it's worth a try. And where it doesn't, report a bug.


You'd still have to physically remove the lower part when shipping a 
library if you wanted to keep the source code closed.


 Would it be possible to do something like this?  A module could 
 (optionally) use 2 files, say x.d and x.i.  x.d would be the 
 "stripped" interface file.  x.i would be the "full"
 implementation file.  x.d would have a statement like: 





 Yikes! It was such a relief that D doesn't force one to do that.
 but... you really want it back?


I did say "optionally" :-).  But yes, I guess I do want something 

same thing, and have the same problem.  They have spawned a code 
obfuscation industry so people can ship classes that can't be 
decompiled.  Yuck!)  I'm not suggesting that D require this, just 
that it support it.  I think conceptual separation of interface and 
implmenatation is the first principle of good software engineering.  
So I think there should be language constructs which support good 
abstraction barriers.  But I agree that forcing people to be good 
isn't nice :-).


 Walters opinion to this is, is that interface mixed with
 implementation is OK, but one needs some documentation tool. Nice
 HTMLs are far easier to read than stripped source or headers.


I agree with the last part, but that also introduces lots of 
redundancy in the code.  You have the code *and* HTML comments that 
explain it.  And we know how easy it is for the 2 to get out of 
synch.

I think this is a really *hard* problem.  No one wants the same thing 
in 2 places.  Having the same thing in 2 places is the first 
principle of bad software engineering!


 module x;
 implementation x;
 
 When building the object x.obj, the compiler would "bring in" the
 implementation and compile the "combined" file.  When compiling
 any other object using "import x" it would not.  It would just
 see the .d interface file.  You'd just ship the .lib containing
 the .obj and .d files.  Just a thought.


 
 If you can use separate interface from implementation as above,
 one could expand on it. For example, by using a version statement
 and a simple tool to strip out the contents of it, or by chaining
 the preprocessor into the build process.


Yes, there could be a solution along these lines.  But does D allow 
you to "reopen" a class definition?

[pseudo code]

class X
{
  this(int);
}

version (implementation)
{
  class X
  {
     this(int x)
     {
        x_ = x;
     }
     private int x_;
  }
}

[/pseudo code]
 
If it did, I suppose you could just do:

[pseudo code]

version (implementation)
{
  import x.impl;
}

[/pseudo code]

Or something like that.


 BTW, do you have a tool or script of some kind that strips the 
 function bodies?  It looks to me like that might be hard to do 
 reliably without actually parsing the code to distinguish
 function bodies from other types of blocks.


 
 I recall dig compiler driver (look for undig at dsource) would
 usually be able to do the trick, but someone was working on a
 better utility. 


Thanks, I'll look there.

Dave

-- 
D.a.v.i.d  T.i.k.t.i.n
t.i.k.t.i.n [at] a.d.v.a.n.c.e.d.r.e.l.a.y [dot] c.o.m
Jul 29 2004
next sibling parent Niko Korhonen <niktheblak hotmail.com> writes:
David Tiktin wrote:



 They have spawned a code obfuscation industry so people can ship
 classes that can't be decompiled.  Yuck!)


OT:

The obfuscation industry is just exploiting people's paranoia. 
Decompiling Bytecode/MSIL IMO isn't such a big threat. A company can 
just put a "reverse engineering prohibited" discalimer in the EULA and 
sue the hell out of anyone who might be doing so. Individuals who can't 
afford lawyers develop open source anyway :)

Remember that you can disassemble any executable. But figuring out what 
the code does is harder. *Much* harder. But it's still /possible/ :)
Jul 29 2004
prev sibling parent Ilya Minkov <minkov cs.tum.edu> writes:
David Tiktin schrieb:

 Yes, there could be a solution along these lines.  But does D allow 
 you to "reopen" a class definition?


No. I think it should rather done like in C++: you make a complete class 
declaration in the "header" part, excluding code, and then fill in the 
methods below in the version(implementation) part, e.g.

X.this(int x) {
    x_ = x;
}

x_ has to be defined in the public part anyway, because else the class 
definition would be incomplete, size unknown, and thus it would all 
break when you create a class derived from X.


 If it did, I suppose you could just do:


[...]

   import x.impl;


[...]

This line cannot work properly, because qualified names for classes 
would be different. Perhaps one could make alias fix that, but i think 
then it would be too seducing to place data members here, and thus break 
the definition.

You should also be aware, that D automatically evaluates functions for 
inlining, if their source is available during compilation. If not, 
then... bad luck. D is geared towards whole-program optimizations, which 
might be the next big thing which might cause Java and CLI/.NET to grow 
stronger, and C++ to fall back a bit.

This question of source protection has actually popped up a couple of 
times in the long history of D newsgroups, back when it wasn't that 
crowded and i read all the couple of thousands messages per year. It was 
agreed upon that when commercial compilers appear, they could foresee a 
format to replace lib, containing part compiled code, part declarations, 
part parsed representation of inlinable functions. For current 
development, it would be putting a priority on a wrong spot - there is 
way too much primary things to iron out. But an external tool would be 
thinkable, which would be based on a stripper anyway, plus an archive 
and compilation manager. Have a mercy with Walter: he cannot clone 
himself, people hardly help him, and he doesn't have a time machine.

-eye
Jul 30 2004
prev sibling parent "C. Sauls" <ibisbasenji yahoo.com> writes:
Ilya Minkov wrote:

 I recall dig compiler driver (look for undig at dsource) would usually 
 be able to do the trick, but someone was working on a better utility.


I was at one time, is that the one you mean?  I basically scrapped most 
of it pending some changes in both D and Mango... I guess I ought to get 
that thing written now.  Just hadn't needed it myself, so there was no 
hurry.

-Chris S.
-Invironz
Jul 29 2004
prev sibling parent reply Andy Friesen <andy ikagames.com> writes:
David Tiktin wrote:


 Would it be possible to do something like this?  A module could 
 (optionally) use 2 files, say x.d and x.i.  x.d would be the 
 "stripped" interface file.  x.i would be the "full" implementation 
 file.  x.d would have a statement like: 
 
 module x;
 implementation x;
 
 When building the object x.obj, the compiler would "bring in" the 
 implementation and compile the "combined" file.  When compiling any 
 other object using "import x" it would not.  It would just see the .d 
 interface file.  You'd just ship the .lib containing the .obj and .d 
 files.  Just a thought.


A simpler way would be to allow a class to be redeclared in a different 
source file within the same compilation phase iff there are no 
conflicting definitions.

Then we could do

foo.d

     module foo;

     class Foo : FooBase {
         this();
         void do_things();
     }

foo.impl.d (or whatever you want to name it)

     module foo; // foo, not foo_impl!  Currently not legal.

     class Foo : FooBase {
         this() { ... } // implemented this time
         void do_things() { ... }
     }

All the compiler really needs to do is make sure that any conflicting 
definitions are identical, save for omitted function bodies.  The linker 
can already catch any duplicated or missing definitions.

With respect to the filename not matching the module name, there are two 
solutions that come to mind: silently ignore it, or make the ".impl.d" 
extension (or something like it) explicitly part of the language.

Silently ignoring it is probably much easier: code declared in 
foo.impl.d is compiled as though it was part of module foo. (name 
mangling and so forth) Modules importing foo will get the stub instead, 
as per existing rules, which should be okay in most situations.

Or we could just write a good stripping tool and stay free from manual 
duplication that way. :)

  -- andy
Jul 29 2004
parent David Tiktin <dtiktin nospam.totally-bogus.com> writes:
On 29 Jul 2004, Andy Friesen <andy ikagames.com> wrote:


 Or we could just write a good stripping tool and stay free from
 manual duplication that way. :)


That could work as long as the stripping tool is a standard part of the 
D toolchain.  After all, you really want to use it only when the code 
is done (the interface is stable) and you're building the library for 
distribution.  It could be part of the release build process to 
*optionally* generate the "stripped" source files.

Dave

-- 
D.a.v.i.d  T.i.k.t.i.n
t.i.k.t.i.n [at] a.d.v.a.n.c.e.d.r.e.l.a.y [dot] c.o.m
Jul 29 2004
prev sibling parent reply Niko Korhonen <niktheblak hotmail.com> writes:
Thanks for the responses everyone! I already suspected that I had to 
provide some sort of stripped-down (let's not call them headers) 
versions of the source code.

Ok, this works well enough. Technically I'm against having /two/ sets of 
source files since they are bound to drift apart as time goes by.

How nightmarish C/C++ headers may be, they still work well enough since 
the compiler processes them too, hopefully emitting a warning if the 
interface and implementation differ too much. But if I'm keeping two 
sets of D source, I develop actively the other set and pretty much 
ignore the "header" (Ok, now I called it a header) set. Since the 
compiler doesn't check these "headers" while compiling the library, 
there's probably going to be some nasty surprises sooner or later.

The Java/.NET way of using only one set of source files is IMO 
definitely the way to go. But I guess what is done with Java/.NET 
(distribute only .jar or .dll files, no headers or source code) is 
impossible in D, since Bytecode/MSIL contains all the information the 
source code does. The binary code that C/C++/D compiler emits doesn't.
Jul 29 2004
next sibling parent reply h3r3tic <h3r3tic dev.null> writes:
Though about it for a longer moment and the best option now really seems 
to have a good tool that would generate the headers. That tool should, 
being open-source and crossplatform would be used in every IDE so that 
one would only click on an icon 'generate headers' or sth like that and 
immediately have them with no pain at all. I think this wouldnt hurt anybody
Jul 29 2004
parent reply parabolis <parabolis softhome.net> writes:
h3r3tic wrote:


 Though about it for a longer moment and the best option now really seems 
 to have a good tool that would generate the headers. That tool should, 
 being open-source and crossplatform would be used in every IDE so that 
 one would only click on an icon 'generate headers' or sth like that and 
 immediately have them with no pain at all. I think this wouldnt hurt 
 anybody


It would be more universal (and ergo more helpful) if it were 
actually a compiler option...
Jul 29 2004
next sibling parent reply stonecobra <scott stonecobra.com> writes:
parabolis wrote:


 h3r3tic wrote:
 

 Though about it for a longer moment and the best option now really 
 seems to have a good tool that would generate the headers. That tool 
 should, being open-source and crossplatform would be used in every IDE 
 so that one would only click on an icon 'generate headers' or sth like 
 that and immediately have them with no pain at all. I think this 
 wouldnt hurt anybody


 
 
 It would be more universal (and ergo more helpful) if it were actually a 
 compiler option...


the front-end source is available, no?

Perhaps you could code it up and submit a patch to Walter.  Then it 
would be a compiler option

Scott
Jul 29 2004
parent parabolis <parabolis softhome.net> writes:
stonecobra wrote:

 parabolis wrote:
 

 h3r3tic wrote:


 Though about it for a longer moment and the best option now really 
 seems to have a good tool that would generate the headers. That tool 
 should, being open-source and crossplatform would be used in every 
 IDE so that one would only click on an icon 'generate headers' or sth 
 like that and immediately have them with no pain at all. I think this 
 wouldnt hurt anybody




 It would be more universal (and ergo more helpful) if it were actually 
 a compiler option...


 
 
 the front-end source is available, no?
 
 Perhaps you could code it up and submit a patch to Walter.  Then it 
 would be a compiler option


That is a good point... Although I was just thinking over Martin 
M. Pedersen's post in which he suggested shoving it all into an 
.obj file which would happen in the optlink...
Jul 29 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <ceccnk$21uc$1 digitaldaemon.com>, parabolis says...


It would be more universal (and ergo more helpful) if it were 
actually a compiler option...


<rant>
It would be "helpful" only for those people wishing to hide their source code
from the rest of us.

Make the closed source brigade do their own dirty work, I say. If they're going
to be charging us money for their closed source software they can damn well pay
for their own stripping tool.
</rant>

Jill
Jul 30 2004
next sibling parent "Matthew" <admin.hat stlsoft.dot.org> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cectuh$2aa6$1 digitaldaemon.com...

 In article <ceccnk$21uc$1 digitaldaemon.com>, parabolis says...


It would be more universal (and ergo more helpful) if it were
actually a compiler option...


 <rant>
 It would be "helpful" only for those people wishing to hide their source code
 from the rest of us.

 Make the closed source brigade do their own dirty work, I say. If they're going
 to be charging us money for their closed source software they can damn well pay
 for their own stripping tool.
 </rant>


I trust that was wind-up?

It most certainly should be an option in the compiler. Absence of such is not
going to persuade people to release their
IP, it will merely dissuade them from using D.
Jul 30 2004
prev sibling parent reply Niko Korhonen <niktheblak hotmail.com> writes:
Arcane Jill wrote:

 It would be "helpful" only for those people wishing to hide their source code
 from the rest of us.


I'm not asking "how can I close my sources" just out of spite. In fact, 
I /am/ a fan of open source, and all of my personal projects are 
BSD-licensed anyway.

But the fact is that not all software development is done by open source 
-oriented individuals. Only a small minority of it is. It's a common 
business practice not to divulge your intellectual property. I asked the 
question strictly with business considerations of the D language in mind.
Jul 30 2004
next sibling parent reply "Matthew" <admin.hat stlsoft.dot.org> writes:
"Niko Korhonen" <niktheblak hotmail.com> wrote in message
news:cedfu1$2ifg$1 digitaldaemon.com...

 Arcane Jill wrote:

 It would be "helpful" only for those people wishing to hide their source code
 from the rest of us.


 I'm not asking "how can I close my sources" just out of spite. In fact,
 I /am/ a fan of open source, and all of my personal projects are
 BSD-licensed anyway.

 But the fact is that not all software development is done by open source
 -oriented individuals. Only a small minority of it is. It's a common
 business practice not to divulge your intellectual property. I asked the
 question strictly with business considerations of the D language in mind.


Quite right. The idea that all software must be open-source is indicative of a
peurile notion amongst many NG-hangers,
i.e. that everything in the IT world should be free. I've never had a paying
client for whom an open-source model could
ever conceivably be appropriate. To not recognise this, or to not care, is
simply stupid, and demonstrates an attitude
that is not commensurate with the long term aims of D (and of Walter).

I completely agree that the DMD distribution should include tools to assist
with closed-source development, and urge
Walter (or anyone with a bit of spare time) to consider how we can get this
with 1.0.
Jul 30 2004
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <cedhr2$2jjk$2 digitaldaemon.com>, Matthew says...

Quite right. The idea that all software must be open-source is indicative of a
peurile notion amongst many NG-hangers,
i.e. that everything in the IT world should be free. I've never had a paying
client for whom an open-source model could
ever conceivably be appropriate. To not recognise this, or to not care, is
simply stupid, and demonstrates an attitude
that is not commensurate with the long term aims of D (and of Walter).

I completely agree that the DMD distribution should include tools to assist
with closed-source development, and urge
Walter (or anyone with a bit of spare time) to consider how we can get this
with 1.0.


Agreed.  But to the open source folks I will say that I'm quite willing to pay
extra for libraries that ship with full source code, so long as the license
doesn't require me to give credit in my docs or any other such nonsense.


Sean
Jul 30 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cedhr2$2jjk$2 digitaldaemon.com>, Matthew says...

The idea that all software must be open-source is indicative of a peurile
notion amongst many NG-hangers,
i.e. that everything in the IT world should be free. 


Quite right. Why? Did someone on this newsgroup claim that all software must be
open source? Or that everything in the IT world should be free? I must have
missed that post.

I don't think it unreasonable though to suggest that if someone is likely to
gain commercially from the use of a tool, that they shouldn't necessarily expect
that tool to be handed to them on a plate free of charge. But that's just an
opinion, and it does seems a shame that one can't express such things on this
forum without incurring overreaction.

Jill

PS. Come on, lighten up, Matthew. Nothing enclosed within <rant> and </rant>
should ever be taken too seriously. Just imagine like I'm complaining about the
price of fish or the state of the bus service or something, and take it on that
level. I thought Niko's response to my post was perfectly reasonable. Why spoil
it? Let's keep this place fun.
Jul 30 2004
parent "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cedvfk$2rfq$1 digitaldaemon.com...

 In article <cedhr2$2jjk$2 digitaldaemon.com>, Matthew says...

The idea that all software must be open-source is indicative of a peurile
notion amongst many NG-hangers,
i.e. that everything in the IT world should be free.


 Quite right. Why? Did someone on this newsgroup claim that all software must be
 open source? Or that everything in the IT world should be free? I must have
 missed that post.

 I don't think it unreasonable though to suggest that if someone is likely to
 gain commercially from the use of a tool, that they shouldn't necessarily
expect
 that tool to be handed to them on a plate free of charge. But that's just an
 opinion, and it does seems a shame that one can't express such things on this
 forum without incurring overreaction.

 Jill

 PS. Come on, lighten up, Matthew. Nothing enclosed within <rant> and </rant>
 should ever be taken too seriously. Just imagine like I'm complaining about the
 price of fish or the state of the bus service or something, and take it on that
 level. I thought Niko's response to my post was perfectly reasonable. Why spoil
 it? Let's keep this place fun.


I am light. I just like carping on from time to time. (Which I know you know by
now.)
Jul 30 2004
prev sibling parent Helmut Leitner <helmut.leitner wikiservice.at> writes:
Niko Korhonen wrote:

 
 Arcane Jill wrote:

 It would be "helpful" only for those people wishing to hide their source code
 from the rest of us.


 
 I'm not asking "how can I close my sources" just out of spite. In fact,
 I /am/ a fan of open source, and all of my personal projects are
 BSD-licensed anyway.
 
 But the fact is that not all software development is done by open source
 -oriented individuals. Only a small minority of it is. It's a common
 business practice not to divulge your intellectual property. I asked the
 question strictly with business considerations of the D language in mind.


I remember that Burton Radons built a solution for his libraries
and distributed them this way.

-- 
Helmut Leitner    leitner hls.via.at
Graz, Austria   www.hls-software.com
Jul 31 2004
prev sibling next sibling parent reply Charlie <Charlie_member pathlink.com> writes:
Ive gotten the DMD front-end to compile ( super cool design i might add!) , and
am working out the details of the symbol table ( i dont have any experience with
this sort of thing ) , but it actually shouldnt be too hard to write a stripper
tool given the front end.  In fact its doable right now, I plan to work on this
when I get back from vacation ( pop quiz, what state is the self proclaimed
Sportsmans Paradise  :) ).

Charlie

In article <cebo1r$1p37$1 digitaldaemon.com>, Niko Korhonen says...

Thanks for the responses everyone! I already suspected that I had to 
provide some sort of stripped-down (let's not call them headers) 
versions of the source code.

Ok, this works well enough. Technically I'm against having /two/ sets of 
source files since they are bound to drift apart as time goes by.

How nightmarish C/C++ headers may be, they still work well enough since 
the compiler processes them too, hopefully emitting a warning if the 
interface and implementation differ too much. But if I'm keeping two 
sets of D source, I develop actively the other set and pretty much 
ignore the "header" (Ok, now I called it a header) set. Since the 
compiler doesn't check these "headers" while compiling the library, 
there's probably going to be some nasty surprises sooner or later.

The Java/.NET way of using only one set of source files is IMO 
definitely the way to go. But I guess what is done with Java/.NET 
(distribute only .jar or .dll files, no headers or source code) is 
impossible in D, since Bytecode/MSIL contains all the information the 
source code does. The binary code that C/C++/D compiler emits doesn't.
Jul 29 2004
parent J C Calvarese <jcc7 cox.net> writes:
Charlie wrote:

 Ive gotten the DMD front-end to compile ( super cool design i might add!) , and


Great!


 am working out the details of the symbol table ( i dont have any experience
with
 this sort of thing ) , but it actually shouldnt be too hard to write a stripper
 tool given the front end.  In fact its doable right now, I plan to work on this


For what it's worth, dig's strip.d isn't particularly long, but it's 
filled with "pointer magic" that hurts my brain to look at it. It's 
probably easy to understand for someone experienced in C++, but it 
doesn't seem like a very D way of doing it.


 when I get back from vacation ( pop quiz, what state is the self proclaimed
 Sportsmans Paradise  :) ).


Have fun!


 
 Charlie
 
 In article <cebo1r$1p37$1 digitaldaemon.com>, Niko Korhonen says...
 

Thanks for the responses everyone! I already suspected that I had to 
provide some sort of stripped-down (let's not call them headers) 
versions of the source code.

Ok, this works well enough. Technically I'm against having /two/ sets of 
source files since they are bound to drift apart as time goes by.

How nightmarish C/C++ headers may be, they still work well enough since 
the compiler processes them too, hopefully emitting a warning if the 
interface and implementation differ too much. But if I'm keeping two 
sets of D source, I develop actively the other set and pretty much 
ignore the "header" (Ok, now I called it a header) set. Since the 
compiler doesn't check these "headers" while compiling the library, 
there's probably going to be some nasty surprises sooner or later.

The Java/.NET way of using only one set of source files is IMO 
definitely the way to go. But I guess what is done with Java/.NET 
(distribute only .jar or .dll files, no headers or source code) is 
impossible in D, since Bytecode/MSIL contains all the information the 
source code does. The binary code that C/C++/D compiler emits doesn't.




-- 
Justin (a/k/a jcc7)
http://jcc_7.tripod.com/d/
Jul 29 2004
prev sibling next sibling parent David L. Davis <SpottedTiger yahoo.com> writes:
In article <cebo1r$1p37$1 digitaldaemon.com>, Niko Korhonen says...

...



The Java/.NET way of using only one set of source files is IMO 
definitely the way to go. But I guess what is done with Java/.NET 
(distribute only .jar or .dll files, no headers or source code) is 
impossible in D, since Bytecode/MSIL contains all the information the 
source code does. The binary code that C/C++/D compiler emits doesn't.


Niko Korhonen : If you're wanting to place Free Functions with default
parameters (I haven't figured out the exporting Classes to a DLL and then using
them with a D app yet) into a DLL without the need of exposing your D code. I
have a project called the "Financial Functions Package" on my site, that creates
a Windows' DLL in D that's then explicitly loaded into a D Application. Please
download and check out the source code in financial_dll.d and loadfindll.d (the
loader / D App) from the project at the link below. It's always good to have
more options. :))

http://spottedtiger.tripod.com/D_Language/D_Financial_Package_XP.html

Also I've written an article (1st part - the 2nd part will be about the loader)
of how I created the D DLL and some of the pitfalls here:
http://spottedtiger.tripod.com/D_Language/D_Demystifying_D_DLLs1_XP.html

Hope this helps a little.

David L.

-------------------------------------------------------------------
"Dare to reach for the Stars...Dare to Dream, Build, and Achieve!"
Jul 29 2004
prev sibling next sibling parent reply "Martin M. Pedersen" <martin moeller-pedersen.dk> writes:
"Niko Korhonen" <niktheblak hotmail.com> skrev i en meddelelse
news:cebo1r$1p37$1 digitaldaemon.com...

 The Java/.NET way of using only one set of source files is IMO
 definitely the way to go.


I agree, wholeheartedly.


 But I guess what is done with Java/.NET
 (distribute only .jar or .dll files, no headers or source code) is
 impossible in D, since Bytecode/MSIL contains all the information the
 source code does. The binary code that C/C++/D compiler emits doesn't.


Because the binary code that the D compiler emits now, does not include all
that information, it does not mean that it could not be made to do it. I
cannot imagine that there is not a way to put that information into object
files and/or libraries. It could be as simple a producing the stripped down
version of the input source file and put it into the object files as some
predefined symbol or stuff it into a comment field. No new file formats
needs to be invented - only a built-in "stripper" in the compiler and the
ability to parse a source embedded in an object file.

Regards,
Martin M. Pedersen
Jul 29 2004
parent reply parabolis <parabolis softhome.net> writes:
Martin M. Pedersen wrote:


 Because the binary code that the D compiler emits now, does not include all
 that information, it does not mean that it could not be made to do it. I
 cannot imagine that there is not a way to put that information into object
 files and/or libraries. It could be as simple a producing the stripped down
 version of the input source file and put it into the object files as some
 predefined symbol or stuff it into a comment field. No new file formats
 needs to be invented - only a built-in "stripper" in the compiler and the
 ability to parse a source embedded in an object file.


This is actually an intriguing idea. (Besides the fact that 
humans would not be able to read the stripped down files.) The 
OMF format (which is what dm uses) does have a comment field 
that might be repurposed to do just that. (COMMENT type 0xDA 
should work)
Jul 29 2004
parent reply "Martin M. Pedersen" <martin moeller-pedersen.dk> writes:
"parabolis" <parabolis softhome.net> skrev i en meddelelse
news:cecegg$22k5$1 digitaldaemon.com...

 Martin M. Pedersen wrote:
 This is actually an intriguing idea. (Besides the fact that
 humans would not be able to read the stripped down files.)


It would require only a very simple tool to extract the interfaces.

I think it is desirable to keep the interface with compiled code. If they
are separate as in C, it opens up for inconsistencies - something that I'm
sure every C/C++ programmer here has learned to hate. Many years ago I was
working with Turbo Pascal, and it's TPU-files was the combination of the
compiled code and the interface. It worked very well, and also allowed the
compiler to do lazy recompilation minimizing the need for tools such as
"make". I don't know how Delphi works today. I think this approach would be
very similar to Turbo Pascal's TPU-files, except that they are compatible
with a C development environment, and we have the complexities of libraries
to consider too.


 OMF format (which is what dm uses) does have a comment field
 that might be repurposed to do just that. (COMMENT type 0xDA
 should work)


I thought so :-)

Regards,
Martin M. Pedersen
Jul 30 2004
parent parabolis <parabolis softhome.net> writes:
Martin M. Pedersen wrote:


 "parabolis" <parabolis softhome.net> skrev i en meddelelse
 news:cecegg$22k5$1 digitaldaemon.com...
 

Martin M. Pedersen wrote:
This is actually an intriguing idea. (Besides the fact that
humans would not be able to read the stripped down files.)


 
 
 It would require only a very simple tool to extract the interfaces.
 
 I think it is desirable to keep the interface with compiled code. If they
 are separate as in C, it opens up for inconsistencies - something that I'm
 sure every C/C++ programmer here has learned to hate. Many years ago I was
 working with Turbo Pascal, and it's TPU-files was the combination of the
 compiled code and the interface. It worked very well, and also allowed the
 compiler to do lazy recompilation minimizing the need for tools such as
 "make". I don't know how Delphi works today. I think this approach would be
 very similar to Turbo Pascal's TPU-files, except that they are compatible
 with a C development environment, and we have the complexities of libraries
 to consider too.
 


Actually I saw someone suggest .dar files which sound like they 
could do what you want them to. In Java .jar files are .zip 
files which (usually compression level of 0). The internals are 
treated in a manner similar to /lib/... So the suggestion allows 
for:
     /lib/... obj files here
     /inc/... auto created stripped down/include files here
     /src/... full source file here if you want
     /kitchen-sink/... self explanatory

Provided the compiler can read/write directly to .dar files then 
it can also automatically save the stripped down .d file in the 
appropriate /inc/ directory.

Then D would also need a seperate .dar management tool but given 
zlib capabilities already exist the whole idea is really 
probably quite simply realizable.

You will thus be able to distribute your whole package with full 
or stripped down source along with any necc. .obj files and with 
decent compression.
Jul 30 2004
prev sibling parent reply J C Calvarese <jcc7 cox.net> writes:
Niko Korhonen wrote:

 Thanks for the responses everyone! I already suspected that I had to 
 provide some sort of stripped-down (let's not call them headers) 
 versions of the source code.
 
 Ok, this works well enough. Technically I'm against having /two/ sets of 
 source files since they are bound to drift apart as time goes by.
 
 How nightmarish C/C++ headers may be, they still work well enough since 
 the compiler processes them too, hopefully emitting a warning if the 
 interface and implementation differ too much. But if I'm keeping two 
 sets of D source, I develop actively the other set and pretty much 
 ignore the "header" (Ok, now I called it a header) set. Since the 
 compiler doesn't check these "headers" while compiling the library, 
 there's probably going to be some nasty surprises sooner or later.
 
 The Java/.NET way of using only one set of source files is IMO 
 definitely the way to go. But I guess what is done with Java/.NET 
 (distribute only .jar or .dll files, no headers or source code) is 
 impossible in D, since Bytecode/MSIL contains all the information the 
 source code does. The binary code that C/C++/D compiler emits doesn't.


If this is really important to you, I don't know why we (and by we and 
mean Walter) couldn't implement a .dar file format. It could be 
.zip-format file containing stripped headers and a corresponding .lib. 
This is definitely post-1.0 talk, but I don't think it'd have to be too 
hard. We'd want to get the compiler to write .dar files (as well as read 
them).

I suppose if someone was really industrious that could create a tool to 
do all of this now as a build system that calls dmd.exe and lib.exe. 
(And, no, I'm not that person. I'm busy doing other stuff...)

-- 
Justin (a/k/a jcc7)
http://jcc_7.tripod.com/d/
Jul 29 2004
parent reply Niko Korhonen <niktheblak hotmail.com> writes:
J C Calvarese wrote:

 If this is really important to you, I don't know why we (and by we and 
 mean Walter) couldn't implement a .dar file format. It could be 
 .zip-format file containing stripped headers and a corresponding .lib. 


While I can't say that doesn't sound tempting, I'll say no. It's not 
really /that/ important. It doesn't apply to me or my personal 
developing methodologies very much, what I asked was more of a 
theoretical question for (future) business aspects of D.

I asked whether distributing closed-source libraries for D is possible, 
and I got my answer. I'm satisfied. It's currently about as easy as in 
C, I just have to write additional "header" files. If I'm targeting a 
widely-usable C library, I going to have to use extern(C) and write the 
C headers anyway.

What you suggested or what Martin M. Pedersen said about putting the 
necessary information into .obj/.lib files /would/ be a relief though, 
and it would push D a notch into the Java/.NET realm. Whether this is 
desirable or not is probably a subject of debate.

But no, as of yet this isn't necessary. If a required number of people 
start asking about this, *then* you could possibly take it into 
consideration :)
Jul 30 2004
parent J C Calvarese <jcc7 cox.net> writes:
In article <cedhpe$2jja$1 digitaldaemon.com>, Niko Korhonen says...

J C Calvarese wrote:

 If this is really important to you, I don't know why we (and by we and 
 mean Walter) couldn't implement a .dar file format. It could be 
 .zip-format file containing stripped headers and a corresponding .lib. 


While I can't say that doesn't sound tempting, I'll say no. It's not 
really /that/ important. It doesn't apply to me or my personal 
developing methodologies very much, what I asked was more of a 
theoretical question for (future) business aspects of D.


Fair enough.

I wasn't trying to guilt anyone into doing anything. 

I was just hinting that since D is pretty easy to parse, the DMD front end
source is public, and open source archiving code is out there (such as std.zip),
this is probably something that someone could develop -- even without Walter's
help. (And Walter has shown that he'll accept donated code.)

This particular topic has come up before (not recently, but several times). I
guess I've just read too many "Oh, D really needs this" messages in the past.

D is designed to allow both open and closed source projects, so that everyone
can be happy. And allowing stripped headers is a part of that idea.

jcc7
Jul 30 2004