digitalmars.D - A question about modules
- Myron Alexander (34/34) Jun 12 2007 Hello.
- orgoton (4/4) Jun 12 2007 I suppose what you mean is to hide the source of you library, and/or not...
- Myron Alexander (22/28) Jun 12 2007 Orgoton,
- Lutger (4/9) Jun 12 2007 Perhaps this is a stupid question, but what is the benefit of this?
- Myron Alexander (31/41) Jun 12 2007 The problem is having to manage separate header and obj files. This was
- BCS (3/5) Jun 12 2007 IIRC Ada does somthing along this line, but I have never used Ada so I d...
- Myron Alexander (6/15) Jun 12 2007 I think Delphi units work this way. I am not a Delphi programmer but I
- Robert Fraser (3/54) Jun 12 2007 That's a great idea!
- Ary Manzana (6/57) Jun 12 2007 Nice idea!
- BCS (8/21) Jun 12 2007 I think it might be possible to write a program that would extract the s...
- Myron Alexander (5/13) Jun 12 2007 This will not work with compile time features such as templates.
- =?ISO-8859-1?Q?Julio_C=E9sar_Carrascal_Urquijo?= (5/6) Jun 12 2007 If you are using Subversion for both projects you can setup one folder
- Myron Alexander (24/31) Jun 13 2007 I didn't know you could do this.
- Lutger (15/41) Jun 13 2007 dsss is meant for this and I still think it fulfills it's purpose
- Myron Alexander (25/41) Jun 13 2007 Lutger,
- Robert Fraser (4/53) Jun 13 2007 Hi Myron,
- Frits van Bommel (15/16) Jun 13 2007 Actually, I believe the way DDL works is that you compile as if you're
- Myron Alexander (33/54) Jun 13 2007 I have split my library into two libraries, dbapi and sqlite. The sqlite...
Hello. My understanding of D is still maturing so I may make some glaringly incorrect statements. Please correct me if I am wrong. I may also ramble a bit but then I have not slept since midday yesterday. (BTW I did search on the newsgroups, found nothing on this topic so I hope I am not opening any old wounds). I am trying to package my library for use in another project of mine. My problem is that I make extensive use of templates in the library so I cannot just create a static library with some prototype .di files. I am used to the Java jar files and would like to know if it is possible to do something like that for D (even if it is only a zipped up source library). Right now I am copying my library source tree into the project, not a great solution. I was also looking at how to hide the template implementation which got me thinking. I would also like to know if there are any technical reasons that D could not adopt a binary format module mechanism. In Java, you import the compiled class file, not the source. I cannot think of a reason why D should not be able to do the same. The library would not be a C compatible library but then this is the language D. As it stands, if you want C compatibility, you have to code C specific interfaces so nothing would change on that front, except that you would have to tell the compiler that you are generating a C compatible library. Thus we have a compiler that spits out D modules by default and can produce C libraries on demand (not advocating automatic conversion, the developer would have to do the interface dirty work as it currently stands). I hope I made some sense. What are you thoughts on this matter? Regards, Myron Alexander dprogramming...myron...alexander...com replace the first ... with , remove the second, and replace the third with ".".
Jun 12 2007
I suppose what you mean is to hide the source of you library, and/or not to have the source available at compile time. The solution is simple, pre-compile it. In short, compile your library into a .lib file and ask DMD to generate a interface file (.di). Then, on your modules import the .di file and add: pragma(lib, "libfile.lib"); //libfile.lib must be inside the dirs searched by DMD or add the -Ilibfile.lib to your DMD command. The downside is that you either compile the lib in release/debug mode and that's it. If you want to change the build mode, you'll have to recompile the library. But that's not a real problem.
Jun 12 2007
orgoton wrote:I suppose what you mean is to hide the source of you library, and/or not to have the source available at compile time. The solution is simple, pre-compile it. In short, compile your library into a .lib file and ask DMD to generate a interface file (.di). Then, on your modules import the .di file and add: pragma(lib, "libfile.lib"); //libfile.lib must be inside the dirs searched by DMD or add the -Ilibfile.lib to your DMD command. The downside is that you either compile the lib in release/debug mode and that's it. If you want to change the build mode, you'll have to recompile the library. But that's not a real problem.Orgoton, I am aware of the .di files (although I did not see that dmd can generate them for you so thanks for the info) but what I want to do is eliminate the need for a separate .lib and .di files. The way Java works is that you do not have to include a textual header with the .class files, the .class files act as the header and obj in one file. I believe that it is possible to do it for D, but it means some work on the front-end and back-end so the linker will have to have an additional output mode for D specific modules. You would not be able link a D module with a C library directly, the linker would have to extract the executable object from the binary module and then link it all together into an exe. Since I have not written a compiler in nearly 2 decades, I will not say I know it is possible; I hope someone in the know could inform me on the viability of this suggestion. If there are no technical reasons preventing the implementation of such a mechanism, and it is just a matter of Walter not having the time, then so be it; but I would like to know, just in case something could be worked out. Best Regards, Myron.
Jun 12 2007
Myron Alexander wrote:I am aware of the .di files (although I did not see that dmd can generate them for you so thanks for the info) but what I want to do is eliminate the need for a separate .lib and .di files. The way Java works is that you do not have to include a textual header with the .class files, the .class files act as the header and obj in one file.Perhaps this is a stupid question, but what is the benefit of this? Templates are handled ok with .lib and .di files. If the problem is ease of use of packaging, then this is taken care of by tools like dsss.
Jun 12 2007
Lutger wrote:Myron Alexander wrote:The problem is having to manage separate header and obj files. This was always a pain in C and it is a pain in D, less of a pain but one none the less (especially with bug 1133). The other problem with .di files is that the template code is in plain sight - not so good for closed source / commercial libraries. Imagine this scenario with the binary modules: Starting with sources, you compile the library and generate a .dar file (D archive). The .dar file is very much like an uber jar file (with all the meta-data issues sorted out). The .dar file is an archive that contains .dm files or D module files. The D module file acts like the .di file when sources are compiled against it, and it acts like the .obj file in the link phase. Although it acts like the .di, it is not source, it is some syntax tree or optimized representation so it actually speeds up compilation time. The .dm file contains templates and other compile time goodies but without showing them in plain text. Because of how it works, with the right tools, you could convert the .di portion back into readable code, ala java jad, but it would not be as readable as the .di, just like jad. This is not an attempt to lock away the interface code, just not make it so obvious. The main benefit to .dar files is that you have a one file library that you drop into a directory and compile/link against simplifying distribution. Unlike jar files, these are not class loaded modules, they are linked into the binary just like .libs. I hope I made sense, my brain is shutting down and I am having some difficulty concentrating. If you have worked with jar files you will understand the convenience of not having to maintain separate include directories and a lib dir. Best Regards, Myron.I am aware of the .di files (although I did not see that dmd can generate them for you so thanks for the info) but what I want to do is eliminate the need for a separate .lib and .di files. The way Java works is that you do not have to include a textual header with the .class files, the .class files act as the header and obj in one file.Perhaps this is a stupid question, but what is the benefit of this? Templates are handled ok with .lib and .di files. If the problem is ease of use of packaging, then this is taken care of by tools like dsss.
Jun 12 2007
Reply to Myron,The .dm file contains templates and other compile time goodies but without showing them in plain text.IIRC Ada does somthing along this line, but I have never used Ada so I don't know how well this applys.
Jun 12 2007
BCS wrote:Reply to Myron,I think Delphi units work this way. I am not a Delphi programmer but I heard that .dcu compiled units are binary modules. Delphi also has something called a package which is a collection of units. Regards, Myron.The .dm file contains templates and other compile time goodies but without showing them in plain text.IIRC Ada does somthing along this line, but I have never used Ada so I don't know how well this applys.
Jun 12 2007
That's a great idea! But as far as java decompilation goes... never used jad, but cavaj generates a picture-perfect representation of the code except for local variable/parameter names. If it had those in the bytecode, I'd use it as a code formatter/beautifier. Myron Alexander Wrote:Lutger wrote:Myron Alexander wrote:The problem is having to manage separate header and obj files. This was always a pain in C and it is a pain in D, less of a pain but one none the less (especially with bug 1133). The other problem with .di files is that the template code is in plain sight - not so good for closed source / commercial libraries. Imagine this scenario with the binary modules: Starting with sources, you compile the library and generate a .dar file (D archive). The .dar file is very much like an uber jar file (with all the meta-data issues sorted out). The .dar file is an archive that contains .dm files or D module files. The D module file acts like the .di file when sources are compiled against it, and it acts like the .obj file in the link phase. Although it acts like the .di, it is not source, it is some syntax tree or optimized representation so it actually speeds up compilation time. The .dm file contains templates and other compile time goodies but without showing them in plain text. Because of how it works, with the right tools, you could convert the .di portion back into readable code, ala java jad, but it would not be as readable as the .di, just like jad. This is not an attempt to lock away the interface code, just not make it so obvious. The main benefit to .dar files is that you have a one file library that you drop into a directory and compile/link against simplifying distribution. Unlike jar files, these are not class loaded modules, they are linked into the binary just like .libs. I hope I made sense, my brain is shutting down and I am having some difficulty concentrating. If you have worked with jar files you will understand the convenience of not having to maintain separate include directories and a lib dir. Best Regards, Myron.I am aware of the .di files (although I did not see that dmd can generate them for you so thanks for the info) but what I want to do is eliminate the need for a separate .lib and .di files. The way Java works is that you do not have to include a textual header with the .class files, the .class files act as the header and obj in one file.Perhaps this is a stupid question, but what is the benefit of this? Templates are handled ok with .lib and .di files. If the problem is ease of use of packaging, then this is taken care of by tools like dsss.
Jun 12 2007
Nice idea! I also had this idea some weeks ago (but, I never told it)... And, of course, the name was also "dar". It would realy be cool. Especially since in Descent I could replace all ocurrences of the word "jar" with "dar" and voilà! :-P Myron Alexander escribió:Lutger wrote:Myron Alexander wrote:The problem is having to manage separate header and obj files. This was always a pain in C and it is a pain in D, less of a pain but one none the less (especially with bug 1133). The other problem with .di files is that the template code is in plain sight - not so good for closed source / commercial libraries. Imagine this scenario with the binary modules: Starting with sources, you compile the library and generate a .dar file (D archive). The .dar file is very much like an uber jar file (with all the meta-data issues sorted out). The .dar file is an archive that contains .dm files or D module files. The D module file acts like the .di file when sources are compiled against it, and it acts like the .obj file in the link phase. Although it acts like the .di, it is not source, it is some syntax tree or optimized representation so it actually speeds up compilation time. The .dm file contains templates and other compile time goodies but without showing them in plain text. Because of how it works, with the right tools, you could convert the .di portion back into readable code, ala java jad, but it would not be as readable as the .di, just like jad. This is not an attempt to lock away the interface code, just not make it so obvious. The main benefit to .dar files is that you have a one file library that you drop into a directory and compile/link against simplifying distribution. Unlike jar files, these are not class loaded modules, they are linked into the binary just like .libs. I hope I made sense, my brain is shutting down and I am having some difficulty concentrating. If you have worked with jar files you will understand the convenience of not having to maintain separate include directories and a lib dir. Best Regards, Myron.I am aware of the .di files (although I did not see that dmd can generate them for you so thanks for the info) but what I want to do is eliminate the need for a separate .lib and .di files. The way Java works is that you do not have to include a textual header with the .class files, the .class files act as the header and obj in one file.Perhaps this is a stupid question, but what is the benefit of this? Templates are handled ok with .lib and .di files. If the problem is ease of use of packaging, then this is taken care of by tools like dsss.
Jun 12 2007
Reply to Myron,what I want to do is eliminate the need for a separate .lib and .di files. The way Java works is that you do not have to include a textual header with the .class files, the .class files act as the header and obj in one file.[...]If there are no technical reasons preventing the implementation of such a mechanism, and it is just a matter of Walter not having the time, then so be it; but I would like to know, just in case something could be worked out. Best Regards, Myron.I think it might be possible to write a program that would extract the symbol names from an object file, un-mangle them and build a .di file from this information. If this could be done, then this could be linked into the compiler and then pass this .di file off to the code that parses it. (or the .di generator could be re worked so that it inserts the symbol directly into the symbol table)
Jun 12 2007
BCS wrote:I think it might be possible to write a program that would extract the symbol names from an object file, un-mangle them and build a .di file from this information. If this could be done, then this could be linked into the compiler and then pass this .di file off to the code that parses it. (or the .di generator could be re worked so that it inserts the symbol directly into the symbol table)This will not work with compile time features such as templates. See my answer to Lutger. Best Regards, Myron.
Jun 12 2007
Myron Alexander wrote: I am copying my library source tree into the project, not agreat solution.If you are using Subversion for both projects you can setup one folder to automatically import the other project (Even on different repositories) using an external property. That is what I usually do.
Jun 12 2007
Julio César Carrascal Urquijo wrote:Myron Alexander wrote: I am copying my library source tree into the project, not aI didn't know you could do this. Although this will work for my project, it does not solve the library distribution problem. We end up having the same problem that the C repositories have. I'm hoping to solve that problem once and for all. Perhaps for now, we could come up with a convention such as this: All external libraries exist under an ext directory. The ext directory has the following structure: ext/<libname>/di/<toplevel>/... ext/<libname>/<libname>.lib Then with the rebuild/bud/dmd tool or whatever, you could just have an option such as -import=<libname> and it would add the .di to the include path, and the .lib to the link path. This is a workable solution until we get binary modules. If that works, we could go to the next level where we zip up the library. Same directory structure, just zipped up and call it .dlz for D library zipped. Then the rebuild/bud or library tool could unzip the contents into a temp/cache directory and add those directories/.libs to the paths. It does not have to be built into bud/rebuild, it could be a separate library tool that creates a few environment variables to add to the bud/rebuild/dmd command line. What do you think? Regards, Myron.great solution.If you are using Subversion for both projects you can setup one folder to automatically import the other project (Even on different repositories) using an external property. That is what I usually do.
Jun 13 2007
Myron Alexander wrote:Perhaps for now, we could come up with a convention such as this: All external libraries exist under an ext directory. The ext directory has the following structure: ext/<libname>/di/<toplevel>/... ext/<libname>/<libname>.lib Then with the rebuild/bud/dmd tool or whatever, you could just have an option such as -import=<libname> and it would add the .di to the include path, and the .lib to the link path. This is a workable solution until we get binary modules. If that works, we could go to the next level where we zip up the library. Same directory structure, just zipped up and call it .dlz for D library zipped. Then the rebuild/bud or library tool could unzip the contents into a temp/cache directory and add those directories/.libs to the paths. It does not have to be built into bud/rebuild, it could be a separate library tool that creates a few environment variables to add to the bud/rebuild/dmd command line. What do you think? Regards, Myron.dsss is meant for this and I still think it fulfills it's purpose nicely. Rather than creating another convention, it is perhaps more effective to discuss what you find lacking in dsss and how to improve on that. Keep in mind that dsss already does what you mentioned and a lot more too: install over the net (with automatic dependency installs), documentation generation with candydoc, pre- and post build steps... Only thing it doesn't do is hiding .di files for closed source distribution, but that will require another solution altogether. I'm not used to Java style distribution, so I may miss something worthwhile here, but I find the D way not so problematic. It is certainly a lot better than C, and C++ too. This is largely because D has a true module system instead of text replacement, and tools like dsss, rebuild and bud are very good.
Jun 13 2007
Lutger wrote:dsss is meant for this and I still think it fulfills it's purpose nicely. Rather than creating another convention, it is perhaps more effective to discuss what you find lacking in dsss and how to improve on that. Keep in mind that dsss already does what you mentioned and a lot more too: install over the net (with automatic dependency installs), documentation generation with candydoc, pre- and post build steps... Only thing it doesn't do is hiding .di files for closed source distribution, but that will require another solution altogether. I'm not used to Java style distribution, so I may miss something worthwhile here, but I find the D way not so problematic. It is certainly a lot better than C, and C++ too. This is largely because D has a true module system instead of text replacement, and tools like dsss, rebuild and bud are very good.Lutger, I have taken another look at DSSS. When I first looked at it, I was sleep deprived so I did miss some things. Now I see that it does something like I suggested and is better in the usual use-case but it does not cover all the use-cases. The problem with DSSS is that it ties you to the DSSS repository. Apart from the fact that few libraries use it, it is intended, and designed, around release libraries. What about debug libraries? Also, how does it handle versioning, specifically unreleased experimental versions that are distributed to a few alpha/beta testers? What about libraries that have special build conditions? Some example build conditions (outside of portability issues): trial versions, lite versions (only core features), modular vs monolithic versions, and so on. I'm not saying that DSSS cannot handle those cases, I have yet to use it properly so I reserve my judgement, but, from the documentation those are not the typical use-case scenarios. My suggestion complements, rather than replaces, DSSS and could provide that middle ground; might even make Gregor's life easier. If I misunderstand something about DSSS, please correct me. Best Regards, Myron. dprogramming...myron...alexander...com replace the first ... with , remove the second, and replace the third with ".".
Jun 13 2007
Hi Myron, You can use DSSS without putting the library in the net repository. DSSS also has support for version statements, etc., and you can always have multiple copies of a library in DSSS, for different purposes. As far as your other suggestion goes, have you checked out DDL (http://www.dsource.org/projects/ddl )? It's designed primarily around dynamic libraries, but the format certainly contains enough information for static linking. It would be great if the same format could be used for both static and dynamic linking, though exactly how this would work I can't imagine. The project looks dead, but I think pragma said he plans to get back to it once tango gets to 1.0. Myron Alexander Wrote:Lutger wrote:dsss is meant for this and I still think it fulfills it's purpose nicely. Rather than creating another convention, it is perhaps more effective to discuss what you find lacking in dsss and how to improve on that. Keep in mind that dsss already does what you mentioned and a lot more too: install over the net (with automatic dependency installs), documentation generation with candydoc, pre- and post build steps... Only thing it doesn't do is hiding .di files for closed source distribution, but that will require another solution altogether. I'm not used to Java style distribution, so I may miss something worthwhile here, but I find the D way not so problematic. It is certainly a lot better than C, and C++ too. This is largely because D has a true module system instead of text replacement, and tools like dsss, rebuild and bud are very good.Lutger, I have taken another look at DSSS. When I first looked at it, I was sleep deprived so I did miss some things. Now I see that it does something like I suggested and is better in the usual use-case but it does not cover all the use-cases. The problem with DSSS is that it ties you to the DSSS repository. Apart from the fact that few libraries use it, it is intended, and designed, around release libraries. What about debug libraries? Also, how does it handle versioning, specifically unreleased experimental versions that are distributed to a few alpha/beta testers? What about libraries that have special build conditions? Some example build conditions (outside of portability issues): trial versions, lite versions (only core features), modular vs monolithic versions, and so on. I'm not saying that DSSS cannot handle those cases, I have yet to use it properly so I reserve my judgement, but, from the documentation those are not the typical use-case scenarios. My suggestion complements, rather than replaces, DSSS and could provide that middle ground; might even make Gregor's life easier. If I misunderstand something about DSSS, please correct me. Best Regards, Myron. dprogramming...myron...alexander...com replace the first ... with , remove the second, and replace the third with ".".
Jun 13 2007
Robert Fraser wrote:As far as your other suggestion goes, have you checked out DDL (http://www.dsource.org/projects/ddl )? It's designed primarily around dynamic libraries, but the format certainly contains enough information for static linking. It would be great if the same format could be used for both static and dynamic linking, though exactly how this would work I can't imagine. The project looks dead, but I think pragma said he plans to get back to it once tango gets to 1.0.Actually, I believe the way DDL works is that you compile as if you're going to link statically to it, and then it doesn't actually link until run-time. So linking to files extracted from it should work just fine. Put the contents of a .di in a metadata field and you have a file containing both a binary module and the header to link to it. The build utility in use would just have to extract them at the appropriate time and use them. (It doesn't look like DDL provides a stand-alone extraction utility but I think one could easily be created, either by linking to DDL or by writing it from scratch[1]) That would only work for contained binaries the current toolchain supports though, not arbitrary ones (as IIRC DDL itself supports). [1] The file format for .ddl files is at http://dsource.org/projects/ddl/wiki/Reference/ABI/DDL and looks easy enough to parse.
Jun 13 2007
Robert Fraser wrote:You can use DSSS without putting the library in the net repository. DSSS also has support for version statements, etc., and you can always have multiple copies of a library in DSSS, for different purposes.I have split my library into two libraries, dbapi and sqlite. The sqlite library is the glue code that used to be in the "dbapi/c" directory but I decided that it should be split from the dbapi code to make it easier to support different sqlite library versions. The dbapi depends on the sqlite library. The sqlite library is made up of two files: sqlite3.lib and sqlite3.di. By hand, I create the library as follows: implib /s out\sqlite3.lib sqlitedll\sqlite3.dll dmd -c -o- -H -Hdout\di\sqlite src\sqlite\sqlite3.d Now I want to translate it to DSSS and allow DSSS to store the result in the repository. It took a while, but I finally figured out how to do it. Here is my config file:name = sqlite3-glue-library-3.3.17 version = 3.3.17 [+sqlite] prebuild = implib /s out\SDD-sqlite-3.3.17.lib sqlitedll\sqlite3.dll postbuild = dmd -c -o- -H -Hdout\di\sqlite src\sqlite\sqlite3.d postinstall = install out/SDD-sqlite-3.3.17.lib $LIB_PREFIX; \ install out/di/sqlite/sqlite3.di $INCLUDE_PREFIX/sqliteNow that I have used DSSS, I can safely say that my original comments about DSSS versioning weakness stands. It just copies all the includes into a global include directory, and all the libs into a global lib directory. The name is just for a package manifest. Here is what I said:What about debug libraries? Also, how does it handle versioning, specifically unreleased experimental versions that are distributed to a few alpha/beta testers? What about libraries that have special build conditions? Some example build conditions (outside of portability issues): trial versions, lite versions (only core features), modular vs monolithic versions, and so on.Now let say that, for some reason, I have to make a tested version of dbapi linked to sqlite3.dll version 3.3.10. This could be due to some corporate that is using that version, happy with it, and won't change it. I create a copy of my 3.3.17 library and hand edit the .d file based on diffs between the .h versions (as I have been doing when upping the library version). The module name and relative location does not change, how do I install the .di file for 3.3.10 without overwriting 3.3.17? The library is no problem as I will change the name to be SDD-sqlite-3.3.10.lib which will sit happily with SDD-sqlite-3.3.17.lib. I hope I have shown that DSSS does not solve all problems, yet. What Gregor is trying to achieve is not trivial, but not impossible.As far as your other suggestion goes, have you checked out DDL (http://www.dsource.org/projects/ddl )? It's designed primarily around dynamic libraries, but the format certainly contains enough information for static linking. It would be great if the same format could be used for both static and dynamic linking, though exactly how this would work I can't imagine. The project looks dead, but I think pragma said he plans to get back to it once tango gets to 1.0.Something to check out once the project gets going again. Myron. dprogramming...myron...alexander...com replace the first ... with , remove the second, and replace the third with ".".
Jun 13 2007