www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Suggestion: Object filenames should be fully-qualified module names

reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
I originally heard this idea proposed by Gregor Richards in #d, and I
think it should become DMD's default behavior. If it is not the default 
behavior, then it should at least be available as an option.

Perhaps my biggest grievance with both the DMD and GDC compilers is
their handling of object files. DMD's default behavior is to dump all
object files into the current directory. If the -od option is specified,
the object files will be placed into the specified directory instead. If
-op is specified, the object files are placed alongside the original 
source files.

The default behavior and using -od on its own both fail if any two 
source files in the project have the same name, even if they are in 
different packages. Using -op by itself is unappealing for two reasons:

1) It is not unreasonable to expect a system to place libraries in 
directories to which the user does not have write access. Placing object 
files alongside the source files would therefore fail.

2) It pollutes the source directories with object files. I much prefer 
keeping my object files somewhere to the side, in a designated "build" 
directory. This makes keeping projects in version control much easier, 
as I can simply exclude the one directory to keep object files out of 
version control.

Specifying both -op and -od causes things to get a little more 
interesting. Take the following:

test.d
testpkg
   test.d

// test.d
module test;
import testpkg.test : foo;
void main() {
     foo();
}

// testpkg/test.d
module testpkg.test;
import std.stdio : writefln;
void foo() { writefln("foo"); }

If we compile with this:

$ dmd test.d testpkg/test.d -op -odbuild

The "build" directory has the following structure:

build
   test.obj
   testpkg
     test.obj

This is all and well. However, if we compile like this:

$ dmd test.d /path/to/testpkg/test.d -op -odbuild

Then DMD doesn't know what to do, and it places testpkg/test.obj 
alongside the source file. (More specifically, the full path is "joined" 
to the path specified by -od, which works out to just being the full path.)

This ambiguity can be disposed of if an object file's name is its 
fully-qualified module name. If this were true, then we could just say

$ dmd test.d testpkg/test.d -odbuild

and the result would be the build directory looking like this:

build
   test.obj
   testpkg.test.obj

I find this very clean and simple. Since the compiler fails anyway if 
two modules have the same name, there should not ever be overlaps in 
object file names with this scheme. The -op option could probably be 
safely deprecated.

As someone pointed out in #d, this would fail on NTFS if the module's 
fully-qualified name exceeds 255 characters. Though I cannot recall ever 
using a module whose name even approached that limit, this should be 
solved in most cases by truncating the filename at the start. 
(Hopefully, the last 255 characters are unique.) If the object file 
would fail to be unique even then, it can probably be safely declared 
the coder's fault for using a stupid naming scheme.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://pyd.dsource.org
Jan 20 2007
next sibling parent Johan Granberg <lijat.meREM OVE.gmail.com> writes:
Kirk McDonald wrote:


 I originally heard this idea proposed by Gregor Richards in #d, and I
 think it should become DMD's default behavior. If it is not the default
 behavior, then it should at least be available as an option.
 
 Perhaps my biggest grievance with both the DMD and GDC compilers is
 their handling of object files. DMD's default behavior is to dump all
 object files into the current directory. If the -od option is specified,
 the object files will be placed into the specified directory instead. If
 -op is specified, the object files are placed alongside the original
 source files.
 
 The default behavior and using -od on its own both fail if any two
 source files in the project have the same name, even if they are in
 different packages. Using -op by itself is unappealing for two reasons:
 
 1) It is not unreasonable to expect a system to place libraries in
 directories to which the user does not have write access. Placing object
 files alongside the source files would therefore fail.
 
 2) It pollutes the source directories with object files. I much prefer
 keeping my object files somewhere to the side, in a designated "build"
 directory. This makes keeping projects in version control much easier,
 as I can simply exclude the one directory to keep object files out of
 version control.
 
 Specifying both -op and -od causes things to get a little more
 interesting. Take the following:
 
 test.d
 testpkg
    test.d
 
 // test.d
 module test;
 import testpkg.test : foo;
 void main() {
      foo();
 }
 
 // testpkg/test.d
 module testpkg.test;
 import std.stdio : writefln;
 void foo() { writefln("foo"); }
 
 If we compile with this:
 
 $ dmd test.d testpkg/test.d -op -odbuild
 
 The "build" directory has the following structure:
 
 build
    test.obj
    testpkg
      test.obj
 
 This is all and well. However, if we compile like this:
 
 $ dmd test.d /path/to/testpkg/test.d -op -odbuild
 
 Then DMD doesn't know what to do, and it places testpkg/test.obj
 alongside the source file. (More specifically, the full path is "joined"
 to the path specified by -od, which works out to just being the full
 path.)
 
 This ambiguity can be disposed of if an object file's name is its
 fully-qualified module name. If this were true, then we could just say
 
 $ dmd test.d testpkg/test.d -odbuild
 
 and the result would be the build directory looking like this:
 
 build
    test.obj
    testpkg.test.obj
 
 I find this very clean and simple. Since the compiler fails anyway if
 two modules have the same name, there should not ever be overlaps in
 object file names with this scheme. The -op option could probably be
 safely deprecated.
 
 As someone pointed out in #d, this would fail on NTFS if the module's
 fully-qualified name exceeds 255 characters. Though I cannot recall ever
 using a module whose name even approached that limit, this should be
 solved in most cases by truncating the filename at the start.
 (Hopefully, the last 255 characters are unique.) If the object file
 would fail to be unique even then, it can probably be safely declared
 the coder's fault for using a stupid naming scheme.
 


I very much agree that something must be done and this solution is as good
as any.
Jan 20 2007
prev sibling next sibling parent "Rioshin an'Harthen" <rharth75 hotmail.com> writes:
"Kirk McDonald" <kirklin.mcdonald gmail.com> wrote:

 This ambiguity can be disposed of if an object file's name is its 
 fully-qualified module name. If this were true, then we could just say

 $ dmd test.d testpkg/test.d -odbuild

 and the result would be the build directory looking like this:

 build
   test.obj
   testpkg.test.obj

 I find this very clean and simple. Since the compiler fails anyway if two 
 modules have the same name, there should not ever be overlaps in object 
 file names with this scheme. The -op option could probably be safely 
 deprecated.

 As someone pointed out in #d, this would fail on NTFS if the module's 
 fully-qualified name exceeds 255 characters. Though I cannot recall ever 
 using a module whose name even approached that limit, this should be 
 solved in most cases by truncating the filename at the start. (Hopefully, 
 the last 255 characters are unique.) If the object file would fail to be 
 unique even then, it can probably be safely declared the coder's fault for 
 using a stupid naming scheme.


Votes++
Jan 20 2007
prev sibling next sibling parent "Chris Miller" <chris dprogramming.com> writes:
On Sat, 20 Jan 2007 07:17:45 -0500, Kirk McDonald  
<kirklin.mcdonald gmail.com> wrote:
[snip]

 This ambiguity can be disposed of if an object file's name is its  
 fully-qualified module name. If this were true, then we could just say

 $ dmd test.d testpkg/test.d -odbuild

 and the result would be the build directory looking like this:

 build
    test.obj
    testpkg.test.obj

 I find this very clean and simple. Since the compiler fails anyway if  
 two modules have the same name, there should not ever be overlaps in  
 object file names with this scheme. The -op option could probably be  
 safely deprecated.


Agreed, this is a much better way of handling it.


 As someone pointed out in #d, this would fail on NTFS if the module's  
 fully-qualified name exceeds 255 characters. Though I cannot recall ever  
 using a module whose name even approached that limit, this should be  
 solved in most cases by truncating the filename at the start.  
 (Hopefully, the last 255 characters are unique.) If the object file  
 would fail to be unique even then, it can probably be safely declared  
 the coder's fault for using a stupid naming scheme.


I don't think it's an actual limitation of NTFS, but rather most programs  
use buffers for file paths of MAX_PATH characters (260), and to use more  
requires special handling (using some path prefix). I believe the linker  
would be one of those programs that can't handle larger paths.

As a compromise, perhaps N characters from both the head and tail of the  
fully-qualified module name could be used, such as 75 chars from the  
beginning and 75 from the end, and then it still leaves almost half the  
range (MAX_PATH chars) available for storing the path that leads up to the  
module.

For example:
    C:\dmd\src\phobos\std\D\is\a\systems\programming\language\Its\focus\is\on\combining\the\power\and\high\performance\of\C\and\Cpp\with\the\programmer\productivity\of\modern\languages\like\Ruby\and\Python\Special\attention\is\given\to\the\needs\of\quality\assurance\documentation\management\portability\and\reliability.d
Would shrink down to:
    std.D.is.a.systems.programming.language.Its.focus.is.on.combining.the.powers.of.quality.assurance.documentation.management.portability.and.reliability.obj
and still have room to store the directory it's saved in (e.g.  
C:\dmd\src\phobos on the front).
;)
Jan 20 2007
prev sibling next sibling parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Kirk McDonald wrote:

 I originally heard this idea proposed by Gregor Richards in #d, and I
 think it should become DMD's default behavior. If it is not the default 
 behavior, then it should at least be available as an option.
 
 Perhaps my biggest grievance with both the DMD and GDC compilers is
 their handling of object files. DMD's default behavior is to dump all
 object files into the current directory. If the -od option is specified,
 the object files will be placed into the specified directory instead. If
 -op is specified, the object files are placed alongside the original 
 source files.
 
 The default behavior and using -od on its own both fail if any two 
 source files in the project have the same name, even if they are in 
 different packages. Using -op by itself is unappealing for two reasons:
 
 1) It is not unreasonable to expect a system to place libraries in 
 directories to which the user does not have write access. Placing object 
 files alongside the source files would therefore fail.
 
 2) It pollutes the source directories with object files. I much prefer 
 keeping my object files somewhere to the side, in a designated "build" 
 directory. This makes keeping projects in version control much easier, 
 as I can simply exclude the one directory to keep object files out of 
 version control.
 
 Specifying both -op and -od causes things to get a little more 
 interesting. Take the following:
 
 test.d
 testpkg
   test.d
 
 // test.d
 module test;
 import testpkg.test : foo;
 void main() {
     foo();
 }
 
 // testpkg/test.d
 module testpkg.test;
 import std.stdio : writefln;
 void foo() { writefln("foo"); }
 
 If we compile with this:
 
 $ dmd test.d testpkg/test.d -op -odbuild
 
 The "build" directory has the following structure:
 
 build
   test.obj
   testpkg
     test.obj
 
 This is all and well. However, if we compile like this:
 
 $ dmd test.d /path/to/testpkg/test.d -op -odbuild
 
 Then DMD doesn't know what to do, and it places testpkg/test.obj 
 alongside the source file. (More specifically, the full path is "joined" 
 to the path specified by -od, which works out to just being the full path.)
 
 This ambiguity can be disposed of if an object file's name is its 
 fully-qualified module name. If this were true, then we could just say
 
 $ dmd test.d testpkg/test.d -odbuild
 
 and the result would be the build directory looking like this:
 
 build
   test.obj
   testpkg.test.obj
 
 I find this very clean and simple. Since the compiler fails anyway if 
 two modules have the same name, there should not ever be overlaps in 
 object file names with this scheme. The -op option could probably be 
 safely deprecated.
 
 As someone pointed out in #d, this would fail on NTFS if the module's 
 fully-qualified name exceeds 255 characters. Though I cannot recall ever 
 using a module whose name even approached that limit, this should be 
 solved in most cases by truncating the filename at the start. 
 (Hopefully, the last 255 characters are unique.) If the object file 
 would fail to be unique even then, it can probably be safely declared 
 the coder's fault for using a stupid naming scheme.
 


Personally, I would love it.  (I already use a little utility to do this to
DDoc output, 
changing pkg/Module.html to pkg.Module.html.)

-- Chris Nicholson-Sauls
Jan 20 2007
prev sibling next sibling parent =?ISO-8859-1?Q?Lu=EDs_Marques?= <luismarques+spam gmail.com> writes:
Kirk McDonald wrote:

 Perhaps my biggest grievance with both the DMD and GDC compilers is
 their handling of object files. DMD's default behavior is to dump all
 object files into the current directory. If the -od option is specified,
 the object files will be placed into the specified directory instead. If
 -op is specified, the object files are placed alongside the original 
 source files.


I have found the default behavior quite senseless too.
A big approval to changing this from me! :)

--
Luís Marques
Jan 20 2007
prev sibling next sibling parent Gregor Richards <Richards codu.org> writes:
Kirk McDonald wrote:

 I originally heard this idea proposed by Gregor Richards in #d, and I
 think it should become DMD's default behavior. If it is not the default 
 behavior, then it should at least be available as an option.
 


Bizarrely, I find myself agreeing with this ;)

  - Gregor Richards
Jan 20 2007
prev sibling next sibling parent reply Bradley Smith <digitalmars-com baysmith.com> writes:
Have you seen issue 492 (http://d.puremagic.com/issues/show_bug.cgi?id=492)?


Kirk McDonald wrote:

 I originally heard this idea proposed by Gregor Richards in #d, and I
 think it should become DMD's default behavior. If it is not the default 
 behavior, then it should at least be available as an option.
 
 Perhaps my biggest grievance with both the DMD and GDC compilers is
 their handling of object files. DMD's default behavior is to dump all
 object files into the current directory. If the -od option is specified,
 the object files will be placed into the specified directory instead. If
 -op is specified, the object files are placed alongside the original 
 source files.
 
 The default behavior and using -od on its own both fail if any two 
 source files in the project have the same name, even if they are in 
 different packages. Using -op by itself is unappealing for two reasons:
 
 1) It is not unreasonable to expect a system to place libraries in 
 directories to which the user does not have write access. Placing object 
 files alongside the source files would therefore fail.
 
 2) It pollutes the source directories with object files. I much prefer 
 keeping my object files somewhere to the side, in a designated "build" 
 directory. This makes keeping projects in version control much easier, 
 as I can simply exclude the one directory to keep object files out of 
 version control.
 
 Specifying both -op and -od causes things to get a little more 
 interesting. Take the following:
 
 test.d
 testpkg
   test.d
 
 // test.d
 module test;
 import testpkg.test : foo;
 void main() {
     foo();
 }
 
 // testpkg/test.d
 module testpkg.test;
 import std.stdio : writefln;
 void foo() { writefln("foo"); }
 
 If we compile with this:
 
 $ dmd test.d testpkg/test.d -op -odbuild
 
 The "build" directory has the following structure:
 
 build
   test.obj
   testpkg
     test.obj
 
 This is all and well. However, if we compile like this:
 
 $ dmd test.d /path/to/testpkg/test.d -op -odbuild
 
 Then DMD doesn't know what to do, and it places testpkg/test.obj 
 alongside the source file. (More specifically, the full path is "joined" 
 to the path specified by -od, which works out to just being the full path.)
 
 This ambiguity can be disposed of if an object file's name is its 
 fully-qualified module name. If this were true, then we could just say
 
 $ dmd test.d testpkg/test.d -odbuild
 
 and the result would be the build directory looking like this:
 
 build
   test.obj
   testpkg.test.obj
 
 I find this very clean and simple. Since the compiler fails anyway if 
 two modules have the same name, there should not ever be overlaps in 
 object file names with this scheme. The -op option could probably be 
 safely deprecated.
 
 As someone pointed out in #d, this would fail on NTFS if the module's 
 fully-qualified name exceeds 255 characters. Though I cannot recall ever 
 using a module whose name even approached that limit, this should be 
 solved in most cases by truncating the filename at the start. 
 (Hopefully, the last 255 characters are unique.) If the object file 
 would fail to be unique even then, it can probably be safely declared 
 the coder's fault for using a stupid naming scheme.
Jan 20 2007
parent reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Bradley Smith wrote:

 Have you seen issue 492 
 (http://d.puremagic.com/issues/show_bug.cgi?id=492)?
 


No, I hadn't. It's basically the same suggestion, only with using path 
seperators instead of dots. This might have the advantage of not running 
into any theoretical length limits on filenames. Still, being able to 
(sanely) dump object files into a flat directory is also nice.

Either one of these would be good.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://pyd.dsource.org
Jan 20 2007
parent reply Gregor Richards <Richards codu.org> writes:
Kirk McDonald wrote:

 Bradley Smith wrote:

 Have you seen issue 492 
 (http://d.puremagic.com/issues/show_bug.cgi?id=492)?


 
 No, I hadn't. It's basically the same suggestion, only with using path 
 seperators instead of dots. This might have the advantage of not running 
 into any theoretical length limits on filenames. Still, being able to 
 (sanely) dump object files into a flat directory is also nice.
 
 Either one of these would be good.
 


Part of why I like my suggestion of using fully-qualified dot-names is 
how .a files work on UNIX. It's a directory-free archive. So you'll have 
foo.o and foo.o and foo.o :-P

Yes, it can work like this, but it makes extracting .a files a huge PITA.

  - Gregor Richards
Jan 20 2007
parent Gregor Richards <Richards codu.org> writes:
Gregor Richards wrote:

 Kirk McDonald wrote:

 Bradley Smith wrote:

 Have you seen issue 492 
 (http://d.puremagic.com/issues/show_bug.cgi?id=492)?


 No, I hadn't. It's basically the same suggestion, only with using path 
 seperators instead of dots. This might have the advantage of not 
 running into any theoretical length limits on filenames. Still, being 
 able to (sanely) dump object files into a flat directory is also nice.

 Either one of these would be good.


 
 Part of why I like my suggestion of using fully-qualified dot-names is 
 how .a files work on UNIX. It's a directory-free archive. So you'll have 
 foo.o and foo.o and foo.o :-P
 
 Yes, it can work like this, but it makes extracting .a files a huge PITA.
 
  - Gregor Richards


(AFAIK, .lib files can't be extracted at all, so this doesn't translate 
to DMD/Windows at all)

  - Gregor Richards
Jan 20 2007
prev sibling next sibling parent reply just jeff <jeffrparsons optusnet.com.au> writes:
Votes++ for either this or support for building object files to a mirror 
of the source tree. Both sound like good solutions to me. :)
Jan 20 2007
parent Kirk McDonald <kirklin.mcdonald gmail.com> writes:
just jeff wrote:

 Votes++ for either this or support for building object files to a mirror 
 of the source tree. Both sound like good solutions to me. :)


There's a problem with mirroring the source tree. In effect, this is 
what combining -op and -od does now. The problem is: What if you have 
source files from outside your own project's directory? What if you have 
to go all the way to the root before finding a directory that they have 
in common? There's no particularly sensible way to solve this. Basing 
object file names on the module names, rather than however the source 
files are organized, is the only solution I can think of.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://pyd.dsource.org
Jan 20 2007
prev sibling parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Yes please!
I've spent the last week sending mail back and forth with some poor soul 
who was trying to use my Luigi GUI lib, but couldn't seem to link it for 
some reason.

Finally today I tracked it down to the only difference between my 
working build file and his not working one being his use of the -od 
flag.  I didn't know what -od was for, but after figuring it out, I 
remembered this thread.  -od flag and .obj files were silently being 
overwritten with other obj files of the same name.

EVIL!

Isn't Kirk's proposal also the way Java compilers do it?  I seem to 
recall a bunch of foo.bar.baz.class files appearing last time I played 
with Java.

--bb

Kirk McDonald wrote:

 I originally heard this idea proposed by Gregor Richards in #d, and I
 think it should become DMD's default behavior. If it is not the default 
 behavior, then it should at least be available as an option.
 
 Perhaps my biggest grievance with both the DMD and GDC compilers is
 their handling of object files. DMD's default behavior is to dump all
 object files into the current directory. If the -od option is specified,
 the object files will be placed into the specified directory instead. If
 -op is specified, the object files are placed alongside the original 
 source files.
 
 The default behavior and using -od on its own both fail if any two 
 source files in the project have the same name, even if they are in 
 different packages. Using -op by itself is unappealing for two reasons:
 
 1) It is not unreasonable to expect a system to place libraries in 
 directories to which the user does not have write access. Placing object 
 files alongside the source files would therefore fail.
 
 2) It pollutes the source directories with object files. I much prefer 
 keeping my object files somewhere to the side, in a designated "build" 
 directory. This makes keeping projects in version control much easier, 
 as I can simply exclude the one directory to keep object files out of 
 version control.
 
 Specifying both -op and -od causes things to get a little more 
 interesting. Take the following:
 
 test.d
 testpkg
   test.d
 
 // test.d
 module test;
 import testpkg.test : foo;
 void main() {
     foo();
 }
 
 // testpkg/test.d
 module testpkg.test;
 import std.stdio : writefln;
 void foo() { writefln("foo"); }
 
 If we compile with this:
 
 $ dmd test.d testpkg/test.d -op -odbuild
 
 The "build" directory has the following structure:
 
 build
   test.obj
   testpkg
     test.obj
 
 This is all and well. However, if we compile like this:
 
 $ dmd test.d /path/to/testpkg/test.d -op -odbuild
 
 Then DMD doesn't know what to do, and it places testpkg/test.obj 
 alongside the source file. (More specifically, the full path is "joined" 
 to the path specified by -od, which works out to just being the full path.)
 
 This ambiguity can be disposed of if an object file's name is its 
 fully-qualified module name. If this were true, then we could just say
 
 $ dmd test.d testpkg/test.d -odbuild
 
 and the result would be the build directory looking like this:
 
 build
   test.obj
   testpkg.test.obj
 
 I find this very clean and simple. Since the compiler fails anyway if 
 two modules have the same name, there should not ever be overlaps in 
 object file names with this scheme. The -op option could probably be 
 safely deprecated.
 
 As someone pointed out in #d, this would fail on NTFS if the module's 
 fully-qualified name exceeds 255 characters. Though I cannot recall ever 
 using a module whose name even approached that limit, this should be 
 solved in most cases by truncating the filename at the start. 
 (Hopefully, the last 255 characters are unique.) If the object file 
 would fail to be unique even then, it can probably be safely declared 
 the coder's fault for using a stupid naming scheme.
Jan 25 2007
parent reply Bradley Smith <digitalmars-com baysmith.com> writes:
Bill Baxter wrote:

 Isn't Kirk's proposal also the way Java compilers do it?  I seem to 
 recall a bunch of foo.bar.baz.class files appearing last time I played 
 with Java.


Java uses directories. It would be foo/bar/baz.class.
Jan 26 2007
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Bradley Smith wrote:

 Bill Baxter wrote:
  >

 Isn't Kirk's proposal also the way Java compilers do it?  I seem to 
 recall a bunch of foo.bar.baz.class files appearing last time I played 
 with Java.


 
 Java uses directories. It would be foo/bar/baz.class.


What does it do with inner classes then?  Maybe that's what I was 
thinking of.

--bb
Jan 26 2007
parent Bradley Smith <digitalmars-com baysmith.com> writes:
Bill Baxter wrote:

 Bradley Smith wrote:

 Bill Baxter wrote:
  >

 Isn't Kirk's proposal also the way Java compilers do it?  I seem to 
 recall a bunch of foo.bar.baz.class files appearing last time I 
 played with Java.


 Java uses directories. It would be foo/bar/baz.class.


 
 What does it do with inner classes then?  Maybe that's what I was 
 thinking of.


Something like foo/bar/baz$inner.class.
Jan 28 2007