www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A suggestion for modules names / sharing code between projects

reply Sebastien Alaiwan <ace17 free.fr> writes:
Hi all,

I've came across the following problem number of times.
Let's say I have project A and B, sharing code but having a 
different tree structure.

$ find projectA
./projectA/internal/math/algo.d
./projectA/internal/math/lcp.d
./projectA/internal/math/optimize.d
./projectA/gui/main.d

$ find projectB
./projectB/app/render/gfx.d
./projectB/app/render/algo.d
./projectB/app/physics/math/algo.d
./projectB/app/physics/math/lcp.d
./projectB/app/physics/math/optimize.d
./projectB/main.d

The directory "math" is shared between projects. It actually is 
an external/submodule. So it has a standalone existence as a 
library, and might one day be used by projectC.
(In the context of this issue, I'm using separate compilation).

I'd like to be able to write, in projectA's main:

import internal.math.optimize;

This requires me to add, at the beginning of "optimize.d" file, 
this module definition:
module internal.math.optimize;

However, this "optimize.d" file is shared between projects, now 
it's becoming specific to projectA.

How am I supposed to share code between projects then?

Clearly, putting everyone back into the same "file namespace" by 
adding every subdirectory of my project as import paths through 
the command line is not going to scale.

After many projects spent thinking of this issue, I came to the 
conclusion that being able to specify, on the command line, the 
module name of the module currently being compiled, thus allowing 
to override it to be inferred to the basename of the file, would 
solve this problem.

As a side-problem related to this one, having to repeat the 
"absolute" module path to all module declarations/importations is 
tedious. What if I change the name of the package? Now I might 
have to change hundreds of module declarations from "module 
oldname.algo;" to "module newname.algo;".

Clearly, something must be wrong here (and I hope it's me!)
I'd be very happy if someone showed me how solve this problem, 
this has bothered me for ages.
Feb 29 2016
next sibling parent ag0aep6g <anonymous example.com> writes:
On 29.02.2016 20:03, Sebastien Alaiwan wrote:

 $ find projectA
 ./projectA/internal/math/algo.d
 ./projectA/internal/math/lcp.d
 ./projectA/internal/math/optimize.d
 ./projectA/gui/main.d

 $ find projectB
 ./projectB/app/render/gfx.d
 ./projectB/app/render/algo.d
 ./projectB/app/physics/math/algo.d
 ./projectB/app/physics/math/lcp.d
 ./projectB/app/physics/math/optimize.d
 ./projectB/main.d

 The directory "math" is shared between projects. It actually is an
 external/submodule. So it has a standalone existence as a library, and
 might one day be used by projectC.
 (In the context of this issue, I'm using separate compilation).


[...]

 Clearly, putting everyone back into the same "file namespace" by adding
 every subdirectory of my project as import paths through the command
 line is not going to scale.


How is not going to scale? You have to add an import path for every 
location where packages are to be found. How many such locations are you 
going to have that this is unreasonable?

With your example, you'd have -Iinternal in project A, and -Iapp/physics 
in project B. The module declaration would of course be `module 
math.optimize;`.
Feb 29 2016
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, 29 February 2016 at 19:03:53 UTC, Sebastien Alaiwan 
wrote:

 Clearly, something must be wrong here (and I hope it's me!)
 I'd be very happy if someone showed me how solve this problem, 
 this has bothered me for ages.


My solution would be to never have the same file be part of 
multiple projects. Share code using libraries rather than just 
sharing the files. Then modules won't be changing where they are 
in a project or anything like that. The shared code goes into a 
library, and whichever projects need it link against that 
library. You get better modularization and encapsulation that way 
and completely avoid the problem that you're having.

- Jonathan M Davis
Feb 29 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
On Monday, 29 February 2016 at 19:23:08 UTC, Jonathan M Davis 
wrote:

 My solution would be to never have the same file be part of 
 multiple projects. Share code using libraries rather than just 
 sharing the files. Then modules won't be changing where they 
 are in a project or anything like that. The shared code goes 
 into a library, and whichever projects need it link against 
 that library. You get better modularization and encapsulation 
 that way and completely avoid the problem that you're having.


Thanks for your answer ; from what I understand, you're basically 
telling me to only import binary/precompiled libraries, right?

Although I had to admit I had never considered this solution, 
never have the same file be part of multiple projects seems 
extremely restrictive to me.
How is it going to work if I want to share heavily templated 
code, like a container library?
Feb 29 2016
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, 29 February 2016 at 20:00:48 UTC, Sebastien Alaiwan 
wrote:

 On Monday, 29 February 2016 at 19:23:08 UTC, Jonathan M Davis 
 wrote:

 My solution would be to never have the same file be part of 
 multiple projects. Share code using libraries rather than just 
 sharing the files. Then modules won't be changing where they 
 are in a project or anything like that. The shared code goes 
 into a library, and whichever projects need it link against 
 that library. You get better modularization and encapsulation 
 that way and completely avoid the problem that you're having.


 Thanks for your answer ; from what I understand, you're 
 basically telling me to only import binary/precompiled 
 libraries, right?

 Although I had to admit I had never considered this solution, 
 never have the same file be part of multiple projects seems 
 extremely restrictive to me.
 How is it going to work if I want to share heavily templated 
 code, like a container library?


Phobos is a library, and templates work with it just fine. You're 
still importing the .d files. They're just compiled in as part of 
the library that you link against rather than compiled as part of 
your project - e.g. pretty much every D program is linked against 
libphobos, and none of them compile in modules like std.stdio. 
They just import them. They were compiled as part of libphobos 
are aside from templates are not compiled as part of your 
program, just linked.

As for templated code, the templates are generated when your code 
is built, so they're not really linked in in the normal sense, 
but those modules are still part of the library and not part of 
your code, so you'd never do something like add 
std/algorithm/searching.d to the list of files that you're 
compiling. You'd just import it.

If someone wants to hide code, then they'd use .di files and give 
you those to link against, in which case non-templated code could 
be hidden, but templated code would still be in those files, 
because the compiler has to see their source when they're used. 
And most folks will likely just use .d files, since it's simpler 
and allows stuff like CTFE and inlining to work.

Most of the projects on code.dlang.org are libraries, and you're 
using dub, they're easy to pull in and link against. You can do 
the same with your code.

- Jonathan M Davis
Feb 29 2016
prev sibling next sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 29 February 2016 at 19:03:53 UTC, Sebastien Alaiwan 
wrote:

 Hi all,

 I've came across the following problem number of times.
 Let's say I have project A and B, sharing code but having a 
 different tree structure.

 $ find projectA
 ./projectA/internal/math/algo.d
 ./projectA/internal/math/lcp.d
 ./projectA/internal/math/optimize.d
 ./projectA/gui/main.d

 $ find projectB
 ./projectB/app/render/gfx.d
 ./projectB/app/render/algo.d
 ./projectB/app/physics/math/algo.d
 ./projectB/app/physics/math/lcp.d
 ./projectB/app/physics/math/optimize.d
 ./projectB/main.d


I've used this pattern.

  ./projectA/lib/math/algo.d
  ./projectA/lib/math/lcp.d
  ./projectA/lib/math/optimize.d
  ./projectA/gui/main.d

  ./projectB/app/render/gfx.d
  ./projectB/app/render/algo.d
  ./projectB/lib/math/algo.d
  ./projectB/lib/math/lcp.d
  ./projectB/lib/math/optimize.d
  ./projectB/main.d


Dub doesn't like me too much, but in general it works.

Note that it doesn't matter where your modules live, if they have 
the declared module name in the file:

module math.optimize;

If DMD reads this file all your imports should look like:

import math.optimize;

Even if the file is in app/physics/math.
Feb 29 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
On Monday, 29 February 2016 at 19:56:20 UTC, Jesse Phillips wrote:

 I've used this pattern.

  ./projectA/lib/math/algo.d
  ./projectA/lib/math/lcp.d
  ./projectA/lib/math/optimize.d
  ./projectA/gui/main.d

  ./projectB/app/render/gfx.d
  ./projectB/app/render/algo.d
  ./projectB/lib/math/algo.d
  ./projectB/lib/math/lcp.d
  ./projectB/lib/math/optimize.d
  ./projectB/main.d

 Dub doesn't like me too much, but in general it works.

 Note that it doesn't matter where your modules live, if they 
 have the declared module name in the file:

 module math.optimize;

 If DMD reads this file all your imports should look like:

 import math.optimize;

 Even if the file is in app/physics/math.


Yeah, I'm using this pattern too at the moment ;
Although, I'm trying to avoid having these redundant module 
declaration directives at the beginning of each of my library 
files.
Feb 29 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 29 February 2016 at 20:05:11 UTC, Sebastien Alaiwan 
wrote:

 Although, I'm trying to avoid having these redundant module 
 declaration directives at the beginning of each of my library 
 files.


Those module declarations aren't redundant - they are virtually 
required (I think it is a mistake that they aren't explicitly 
required in all cases, actually)

The file layout does not matter to the language itself. Only that 
module declaration does - it is NOT optional if you want a 
package name.
Feb 29 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
On Monday, 29 February 2016 at 20:59:45 UTC, Adam D. Ruppe wrote:

 On Monday, 29 February 2016 at 20:05:11 UTC, Sebastien Alaiwan 
 wrote:

 Although, I'm trying to avoid having these redundant module 
 declaration directives at the beginning of each of my library 
 files.


 Those module declarations aren't redundant - they are virtually 
 required (I think it is a mistake that they aren't explicitly 
 required in all cases, actually)

 The file layout does not matter to the language itself. Only 
 that module declaration does - it is NOT optional if you want a 
 package name.


$ find
main.d
foo/hello.d

$ cat main.d
import foo.hello;

$ cat foo/hello.d
module foo.hello;

Ok so now let's say I rename the directory "lib" to "foo". If I 
don't change the "import lib.hello" to "import foo.hello", how is 
the compiler going to find "hello.d"?
(As a reminder, as I said, I'm using separate compilation)
Feb 29 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 29 February 2016 at 21:04:52 UTC, Sebastien Alaiwan 
wrote:

 Ok so now let's say I rename the directory "lib" to "foo". If I 
 don't change the "import lib.hello" to "import foo.hello", how 
 is the compiler going to find "hello.d"?


You have to tell the compiler where it is. Either way, the module 
name is *not* optional.


 (As a reminder, as I said, I'm using separate compilation)


meh that's part of your problem, why are you doing it that way?
Feb 29 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
On Monday, 29 February 2016 at 21:33:37 UTC, Adam D. Ruppe wrote:

 On Monday, 29 February 2016 at 21:04:52 UTC, Sebastien Alaiwan 
 wrote:

 Ok so now let's say I rename the directory "lib" to "foo". If 
 I don't change the "import lib.hello" to "import foo.hello", 
 how is the compiler going to find "hello.d"?


 You have to tell the compiler where it is.


Is it only possible with separate compilation?


 Either way, the module name is *not* optional.


 (As a reminder, as I said, I'm using separate compilation)


 meh that's part of your problem, why are you doing it that way?


Because it reduces turnaround time (this is something I actually 
measured on the project I'm currently working on). Having to 
recompile everything each time I make a modification takes up to 
25s on this project. By the way, I'm using gdc. Is it also part 
of my problem? :-)
Feb 29 2016
parent reply Mike Parker <aldacron gmail.com> writes:
On Monday, 29 February 2016 at 21:43:31 UTC, Sebastien Alaiwan 
wrote:


 Ok so now let's say I rename the directory "lib" to "foo". If 
 I don't change the "import lib.hello" to "import foo.hello", 
 how is the compiler going to find "hello.d"?


 You have to tell the compiler where it is.


 Is it only possible with separate compilation?



Given two modules, src/lib/foo.d and and src/main.d, you can pass 
them both to the command line and not worry about passing -I to 
the compiler.

dmd src/main.d src/lib/foo.d

In this case, foo.d can have a module statement that gives it any 
name at all. It does not need to match its directory structure:

module wicked.cool.mod;

This works because you've given the module directly to the 
compiler and it doesn't need to search for it. If you compile 
them separately:

dmd -c src/lib/foo.d
dmd src/main.d foo.obj

Now you will see a compiler error. You've given the compiler 
foo's object file, but it doesn't know where the source file is. 
When main.d imported wicked.foo.mod, the compiler tried to find 
wicked/foo/mod.d from its current working directory. No joy. You 
can test this changing the module declaration in foo.d to 
lib.foo, then importing lib.foo in main.d. The you can do this:

cd src
dmd -c lib/foo.d
dmd main.d foo.obj

Now when it encounters import lib.foo, it will look for lib/foo.d 
and find it. Now do this:

cd ..
dmd -c src/lib/foo.d
dmd src/main.d foo.obj

Again, failure, because it's looking for lib/foo.d. So now you 
have to tell it where to find lib/foo.d:

dmd -Isrc src/main.d foo.obj

Now it will look for src/lib/foo.d and find it.

This is why you shouldn't muck about with package names. If you 
want to change the name of the src directory, that's fine. You 
can simply pass -I to the compiler with the correct value. But if 
you want to change the name of lib, you are now affecting the 
compiler's ability to find imports. This becomes a problem with 
separate compilation (as above), which also means it's a problem 
when using libraries. If you change a package name for something 
you're distributing as a library, then all the import statements 
in client code will need to be changed as well.
Feb 29 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
Hi Mike, thanks for taking the time to answer.
If I understand you correctly, you're advising me to keep my file 
hierarchy mostly in sync with my module hierarchy (except for one 
potential leading "src/" or "libs/" that I would add to the 
import search directories). That's fine with me.

However, what's the point of having module declarations 
directives in the language then?

If I understand correctly, the only way to make an interesting 
use of these directives makes separate compilation impossible.

I know dmd is lightning-fast and everything, and not having to 
manage build dependencies anymore is very attractive.

However, we still need separate compilation. Otherwise your 
turnaround time is going to resemble a tractor pulling 
competition as your project grows. Recompiling everything 
everytime you change one module is not an option ; Some of the 
source files I work on in my company require 30s to compile 
*alone* ; it's easy to reach, especially when the language has 
features like templates and CTFE (Pegged anyone?).
Feb 29 2016
parent Guillaume Piolat <contact gam3sfrommars.fr> writes:
On Tuesday, 1 March 2016 at 06:33:06 UTC, Sebastien Alaiwan wrote:

 However, we still need separate compilation. Otherwise your 
 turnaround time is going to resemble a tractor pulling 
 competition as your project grows. Recompiling everything 
 everytime you change one module is not an option ; Some of the 
 source files I work on in my company require 30s to compile 
 *alone* ; it's easy to reach, especially when the language has 
 features like templates and CTFE (Pegged anyone?).


Let me hype DUB a bit. :)

DUB provides completely composable dependencies, in a chain (A => 
B => C) with A not knowing about C.

Compile-times are excellent since DUB can do package-wise, 
file-wise or all-at-once builds from the same description and use 
caching.

Having a declarative build description allows for some goodies 
like IDE project generation.

The package namespace is global, but hierarchical with names like 
"myorg:mypackage".

However, you can't have versionned dependencies for private 
packages [painlessly] which is quite a big limitation.
Mar 02 2016
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 29 February 2016 at 19:03:53 UTC, Sebastien Alaiwan 
wrote:

 What if I change the name of the package? Now I might have to 
 change hundreds of module declarations from "module 
 oldname.algo;" to "module newname.algo;".


That's easy to automate, just run a find/replace across all the 
files, or if you want to go really fancy, modify hackerpilot's 
dfix to do it for you.
Feb 29 2016
parent Sebastien Alaiwan <ace17 free.fr> writes:
On Monday, 29 February 2016 at 21:35:48 UTC, Adam D. Ruppe wrote:

 On Monday, 29 February 2016 at 19:03:53 UTC, Sebastien Alaiwan 
 wrote:

 What if I change the name of the package? Now I might have to 
 change hundreds of module declarations from "module 
 oldname.algo;" to "module newname.algo;".


 That's easy to automate, just run a find/replace across all the 
 files, or if you want to go really fancy, modify hackerpilot's 
 dfix to do it for you.


Yes, I know ; but how can you say there's no redundancy when I 
have to rely on find/replace to keep all these declarations in 
sync?
Feb 29 2016
prev sibling parent reply Guillaume Piolat <name.lastname gmail.com> writes:
On Monday, 29 February 2016 at 19:03:53 UTC, Sebastien Alaiwan 
wrote:

 Hi all,

 I've came across the following problem number of times.
 Let's say I have project A and B, sharing code but having a 
 different tree structure.

 $ find projectA
 ./projectA/internal/math/algo.d
 ./projectA/internal/math/lcp.d
 ./projectA/internal/math/optimize.d
 ./projectA/gui/main.d

 $ find projectB
 ./projectB/app/render/gfx.d
 ./projectB/app/render/algo.d
 ./projectB/app/physics/math/algo.d
 ./projectB/app/physics/math/lcp.d
 ./projectB/app/physics/math/optimize.d
 ./projectB/main.d

 The directory "math" is shared between projects. It actually is 
 an external/submodule. So it has a standalone existence as a 
 library, and might one day be used by projectC.
 (In the context of this issue, I'm using separate compilation).

 I'd like to be able to write, in projectA's main:

 import internal.math.optimize;

 This requires me to add, at the beginning of "optimize.d" file, 
 this module definition:
 module internal.math.optimize;

 However, this "optimize.d" file is shared between projects, now 
 it's becoming specific to projectA.

 How am I supposed to share code between projects then?



Would this work?

1. pick a single module name like

     module math.optimize;

2. import that module with:

     import math.optimize;

3. put this module in a hierarchy like that:

     math/optimize.d

4. pass -I<directory-containing-math-directory> to the compiler

However it may clutter your module namespace a bit more.
Mar 01 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
On Wednesday, 2 March 2016 at 00:16:57 UTC, Guillaume Piolat 
wrote:

 Would this work?

 1. pick a single module name like

     module math.optimize;

 2. import that module with:

     import math.optimize;

 3. put this module in a hierarchy like that:

     math/optimize.d

 4. pass -I<directory-containing-math-directory> to the compiler

 However it may clutter your module namespace a bit more.


Yeah, this is what I going to do: flat hierarchy, like one 'src' 
directory and one 'extra' directory in the project root, globally 
unique package names, passing -Isrc and -Iextra to the compiler, 
and ... module declaration directives.
I was trying to avoid them, but maybe it's not possible.

I still think that software entities shouldn't depend upon their 
own name, or their absolute location in the top-level namespace.

This is true for classes, functions/methods, variables, 
structures ... maybe this rule becomes invalid when third-party 
packages enter a project?
Mar 01 2016
next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 2 March 2016 at 06:40:53 UTC, Sebastien Alaiwan 
wrote:


 Yeah, this is what I going to do: flat hierarchy, like one 
 'src' directory and one 'extra' directory in the project root, 
 globally unique package names, passing -Isrc and -Iextra to the 
 compiler, and ... module declaration directives.
 I was trying to avoid them, but maybe it's not possible.


Every module you create will have a module name whether you use a 
module declaration or not. If you don't use one, the compiler 
uses the file name by default, but *only* the file name. The path 
on the file system is in no way taken into account (I suppose it 
could be by adding a flag to the compiler, but it seems to me 
that would needlessly complicate what is a really simple and easy 
to use system). A module with no package name is in the default 
package, i.e. given src/foo.d, using the following declaration:

module foo;

Is the same as using none at all. If you are content with keeping 
all of your modules in one or more top-level directory and use no 
packages, then you can leave out the module declarations and just 
pass each top-level directory to the compiler with -I.


 I still think that software entities shouldn't depend upon 
 their own name, or their absolute location in the top-level 
 namespace.

 This is true for classes, functions/methods, variables, 
 structures ... maybe this rule becomes invalid when third-party 
 packages enter a project?


The module names and package names are mangled into the symbol 
names in the binary. They also serve as namespaces to distinguish 
conflicting symbols in code. So yes, all of the symbols in a 
module are dependent on their location in the namespace. It's 
baked into the language.

I'm curious what sort of system you have in mind. How does the 
current system not work for you and what would you prefer to see?
Mar 02 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
On Wednesday, 2 March 2016 at 08:50:50 UTC, Mike Parker wrote:

 I'm curious what sort of system you have in mind. How does the 
 current system not work for you and what would you prefer to 
 see?


First, you must know that I've been a C++ programmer for way too 
much time ;
and that my way of thinking about modules has probably been - 
badly - shaped accordingly :-).

However, it feels wrong that modules (foo, bar) inside the same 
package ("pkg") need to repeat the "absolute" package name when 
referencing each other, I mean "import pkg.foo" instead of 
"import foo". Not because of the extra typing, but because of the 
dependency it introduces. But as I said, maybe my vision is wrong.

I'm going to answer your first question, I hope you don't get 
bored before the end ;-)

First some generic stuff:
- The libraries I write are source-only, and I never install them 
system-wide, as many projects might depend on different revisions 
of one library.
- I only work with separate compilation, using gdc (Windows-dmd 
produces OMF, which is a dealbreaker ; but I use rdmd a lot for 
small programs).
- I often mix C code with D, sometimes, C++ code. I compile them 
using gcc/g++.
- My build system is composed of simple dependency-aware 
non-recursive makefiles (BTW, this is a great way to keep your 
code decoupled, otherwise you might not get feedback that you're 
making a dependency mess soon enough!)

This is the best way I found to be able to checkout one project, 
type 'make',
and have everything build properly, wether I'm doing it on 
GNU/Linux or on Windows (MSYS).

Everything is under version control using GNU Bazaar (which 
basically is to git what D is to C++).

I have a small library, which I call "lib_algo", containing 
semi-related utilities (containers, topological sort, etc.) which 
I use in many projects.

$ cd myLibs
$ find lib_algo -name "*.d" -or -name "*.mk" -or name "Makefile"
lib_algo/options.d
lib_algo/misc.d
lib_algo/pointer.d
lib_algo/set.d
lib_algo/queue.d
lib_algo/algebraic.d
lib_algo/vector.d
lib_algo/bresenham.d
lib_algo/topological.d
lib_algo/fix_array.d
lib_algo/test.d
lib_algo/pool.d
lib_algo/stack.d
lib_algo/list.d
lib_algo/array2d.d
lib_algo/project.mk
lib_algo/Makefile
(I know, there's some overlap with Phobos today)

The project.mk simply contains the list of source files of the 
library.
The test.d file actually contains a main function, which I use to 
run manual tests of the library functionalities. It's not 
included in the project.mk list. And it obviously must import the 
files needing to be tested.

With this flat directory structure, the only way test.d can 
import files is by saying, "import stack;", not  "import 
lib_algo.stack".
(adding ".." to the list of import directories might do the 
trick, but this would definitely feel wrong).

Now, let's see a project using this "lib_algo", I call it 
"myGame".

$ cd myProjects
$ find <files relevant to this discussion>
myGame/lib_algo/options.d
myGame/lib_algo/....
myGame/lib_algo/array2d.d
myGame/lib_algo/project.mk
myGame/lib_algo/Makefile
myGame/src/main.d
myGame/Makefile


"lib_algo" simply is an external (aka "git submodule"), pointing 
to a specific revision of the repository "lib_algo".
The top-level "Makefile" of the projects includes 
"lib_algo/project.mk", and all is well, I can compose the list of 
source files to compile without having to rely on 
"convenience/intermediate" libraries.

Now, if "src/main.d" wants to import "array2d.d", currently, I 
must write "import array2d.d", and add "-I myGame/lib_algo" to my 
compiler command line.
I said in a previous post that this would not scale.
That's because with this scheme, obviously, I can't have two 
modules having the same name, even if they belong to different 
libraries.
Let's say I have "lib_algo/utils.d" and "lib_geom/utils.d", if 
"main.d" says "import utils.d", it's ambiguous.

Clearly, "main.d" needs to say "import lib_algo.utils". However, 
if I do this,
I must add a module declaration in "lib_algo/utils.d", and I must 
change
"lib_algo/test.d" so it says "import lib_algo.utils".

This is where it begins to feel clumsy. Why should lib_algo need 
to be
modified in order to resolve an ambiguity happening in "myGame" ?

Moreover, we saw earlier that this modification of the import 
done by
"lib_algo/test.d" had consequences on the directory structure of 
lib_algo.
In order to be able to build the test executable for lib_algo, I 
now need to
have the following structure:
$ find lib_algo
lib_algo/lib_algo/options.d
lib_algo/lib_algo/....
lib_algo/lib_algo/array2d.d
lib_algo/test.d
(test.d can be put inside the inner "lib_algo")

Am I the only one to find this counterintuitive - and probably 
wrong?
What am I missing there?

(And please, don't tell me that I need to switch to 
dub+git+dmd+D_source_files_only ; I like my tools to be 
composable)

Thanks for reading!
Mar 02 2016
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 02.03.2016 21:40, Sebastien Alaiwan wrote:

 - I only work with separate compilation, using gdc (Windows-dmd produces
 OMF, which is a dealbreaker ; but I use rdmd a lot for small programs).


dmd gives you COFF with -m64 or -m32mscoff (-m32 is the default is => OMF).

[...]

 Clearly, "main.d" needs to say "import lib_algo.utils". However, if I do
 this,
 I must add a module declaration in "lib_algo/utils.d", and I must change
 "lib_algo/test.d" so it says "import lib_algo.utils".


That's the way you're supposed to do it, yes. One source file <-> one 
module name.


 This is where it begins to feel clumsy. Why should lib_algo need to be
 modified in order to resolve an ambiguity happening in "myGame" ?


I don't know the exact reasons why the module system works like it does. 
Maybe you're on to something, maybe not. Anyway, here are some 
problems/surprises I see with your alternative idea (which basically 
means using relative file paths, if I get you right):

* You need to be aware of the directory structure when reading/writing 
code, as `import foo;` may mean different things depending on where you 
are. And you don't have a module declaration to tell you where you are.

* You're not able to copy imports from one file to another anymore.

* When moving a module, you have to update all its imports. You can 
introduce a subtle bug when you miss one, because the old import may 
still be valid but have a different meaning.

* The name of a module depends on the current working directory of the 
compiler run. You can't change directories between compiler calls and 
link the object files together.

* You can deliberately or accidentally import one source file under two 
different module names. Two modules from one source file is weird, 
especially when it happens accidentally.

* Can't just throw source files from different packages at the compiler 
anymore without thinking about their relative locations or the -I switch.

[...]

 Moreover, we saw earlier that this modification of the import done by
 "lib_algo/test.d" had consequences on the directory structure of lib_algo.
 In order to be able to build the test executable for lib_algo, I now
 need to
 have the following structure:
 $ find lib_algo
 lib_algo/lib_algo/options.d
 lib_algo/lib_algo/....
 lib_algo/lib_algo/array2d.d
 lib_algo/test.d
 (test.d can be put inside the inner "lib_algo")


Or, you know: `-I..` :P But changing the directory structure is probably 
nicer.


 Am I the only one to find this counterintuitive - and probably wrong?
 What am I missing there?


I don't think your alternative is obviously superior. The current way of 
having exactly one fixed name per module makes it easy to see what's 
going on at any point. You have to pay for that by typing a bit more, I 
guess.
Mar 02 2016
parent reply Sebastien Alaiwan <ace17 free.fr> writes:
Hi guys,

thanks a lot for your answers!

On Wednesday, 2 March 2016 at 22:42:18 UTC, ag0aep6g wrote:

 On 02.03.2016 21:40, Sebastien Alaiwan wrote:

 - I only work with separate compilation, using gdc 
 (Windows-dmd produces OMF, which is a dealbreaker ; but I use 
 rdmd a lot for small programs).


 dmd gives you COFF with -m64 or -m32mscoff (-m32 is the default 
 is => OMF).


Thanks for the tip, I didn't know that.
I have actually deeper reasons for not using dmd (seemless 
cross-compilation, e.g "make CROSS_COMPILE=i686-w64-mingw32" or 
"make CROSS_COMPILE=arm-linux-gnueabi" without any special case 
in the Makefile ; and the way its unusual command line 
(-oftarget) conflicts with MSYS's path conversion, etc. ).


 I don't know the exact reasons why the module system works like 
 it does. Maybe you're on to something, maybe not. Anyway, here 
 are some problems/surprises I see with your alternative idea 
 (which basically means using relative file paths, if I get you 
 right):


Yes, I think we can say that.

I 100% agree with all the limitations you listed ; which makes a 
forced absolute-import system appear to be a lot more comfortable!

There's also one big limitation I see that you didn't list:
* If you're importing lib_a and lib_b both depending on lib_c, 
you need them to expect lib_c to be at the same path/namespace 
location. The current module system guarantees this, because all 
package names are global.

(And linking with precompiled binaries might become a nightmare)

However, using global package names means you're relying on the 
library providers to avoid conflicts, in their package names, 
although they might not even know each other. Which basically 
implies to always put your name / company name in your package 
name, like "import alaiwan.lib_algo.array2d;", or "import 
apple.lib_algo.array2d".
Or rename my "lib_algo" to something meaningless / less common, 
like "import diamond.array2d" (e.g Derelict, Pegged, imaged, 
etc.).

If, unfortunately, I happen to run into a conflict, i.e my 
project uses two unrelated libraries sharing a name for their 
top-level namespace, there's no way out for me, right?
(Except relying on shared objects to isolate symbols)


 I don't think your alternative is obviously superior.


Me neither :-) I'm just sharing my doubts about the current 
system!


 The current way of having exactly one fixed name per module 
 makes it easy to see what's going on at any point. You have to 
 pay for that by typing a bit more, I guess.


Yes, I guess you're right.

I'm going to follow your advice and Mike's for my projects, this 
will allow me to see things more clearly. Anyway, thanks again to 
all of you for your answers!
Mar 02 2016
next sibling parent reply Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Thursday, 3 March 2016 at 06:53:33 UTC, Sebastien Alaiwan 
wrote:

 Hi guys,

 thanks a lot for your answers!

 On Wednesday, 2 March 2016 at 22:42:18 UTC, ag0aep6g wrote:

 On 02.03.2016 21:40, Sebastien Alaiwan wrote:

 - I only work with separate compilation, using gdc 
 (Windows-dmd produces OMF, which is a dealbreaker ; but I use 
 rdmd a lot for small programs).


 dmd gives you COFF with -m64 or -m32mscoff (-m32 is the 
 default is => OMF).


 Thanks for the tip, I didn't know that.
 I have actually deeper reasons for not using dmd (seemless 
 cross-compilation, e.g "make CROSS_COMPILE=i686-w64-mingw32" or 
 "make CROSS_COMPILE=arm-linux-gnueabi" without any special case 
 in the Makefile ; and the way its unusual command line 
 (-oftarget) conflicts with MSYS's path conversion, etc. ).


 I don't know the exact reasons why the module system works 
 like it does. Maybe you're on to something, maybe not. Anyway, 
 here are some problems/surprises I see with your alternative 
 idea (which basically means using relative file paths, if I 
 get you right):


 Yes, I think we can say that.

 I 100% agree with all the limitations you listed ; which makes 
 a forced absolute-import system appear to be a lot more 
 comfortable!

 There's also one big limitation I see that you didn't list:
 * If you're importing lib_a and lib_b both depending on lib_c, 
 you need them to expect lib_c to be at the same path/namespace 
 location. The current module system guarantees this, because 
 all package names are global.

 (And linking with precompiled binaries might become a nightmare)

 However, using global package names means you're relying on the 
 library providers to avoid conflicts, in their package names, 
 although they might not even know each other. Which basically 
 implies to always put your name / company name in your package 
 name, like "import alaiwan.lib_algo.array2d;", or "import 
 apple.lib_algo.array2d".
 Or rename my "lib_algo" to something meaningless / less common, 
 like "import diamond.array2d" (e.g Derelict, Pegged, imaged, 
 etc.).

 If, unfortunately, I happen to run into a conflict, i.e my 
 project uses two unrelated libraries sharing a name for their 
 top-level namespace, there's no way out for me, right?
 (Except relying on shared objects to isolate symbols)


See modules doc page off dlang.org.
import io = std.stdio;
void main()
{ io.writeln("hello!"); // ok, calls std.stdio.writeln
std.stdio.writeln("hello!"); // error, std is undefined
writeln("hello!"); // error, writeln is undefined
}

Also static import and scoped imports.
Mar 03 2016
parent Sebastien Alaiwan <ace17 free.fr> writes:
On Thursday, 3 March 2016 at 08:39:51 UTC, Laeeth Isharc wrote:

 On Thursday, 3 March 2016 at 06:53:33 UTC, Sebastien Alaiwan

 If, unfortunately, I happen to run into a conflict, i.e my 
 project uses two unrelated libraries sharing a name for their 
 top-level namespace, there's no way out for me, right?
 (Except relying on shared objects to isolate symbols)


 See modules doc page off dlang.org.
 import io = std.stdio;
 void main()
 { io.writeln("hello!"); // ok, calls std.stdio.writeln
 std.stdio.writeln("hello!"); // error, std is undefined
 writeln("hello!"); // error, writeln is undefined
 }


Thansk, but I was talking about module name collisions, e.g if my 
project uses two unrelated libraries, both - sloppily - defining 
a module called "geom.utils".

Can this be worked-around using renamed imports?

How is "import myUtils = geom.utils" going to know which of both 
"geom.utils" I'm talking about?
As far as I understand, _module_ renaming at import is rather a 
tool for defining shorthands, than for disambiguation.

I agree that it might not be a problem in practice - just wanting 
to make sure I'm not missing something.
Mar 03 2016
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On 03.03.2016 07:53, Sebastien Alaiwan wrote:

 However, using global package names means you're relying on the library
 providers to avoid conflicts, in their package names, although they
 might not even know each other. Which basically implies to always put
 your name / company name in your package name, like "import
 alaiwan.lib_algo.array2d;", or "import apple.lib_algo.array2d".
 Or rename my "lib_algo" to something meaningless / less common, like
 "import diamond.array2d" (e.g Derelict, Pegged, imaged, etc.).


Yeah, that's a thing. For published libraries one should choose a unique 
name.

I think reverse domain name notation [1] is a convention for exactly 
that. As far as I know, not many D projects do that, though, if any at 
all. But it would be a way out if collisions become a problem.

Also, dub/code.dlang.org plays a role here, as people may assume that a 
name is free as long as it's not on the dub list. This works well when 
dub is indeed the one place to go for libraries. If it isn't, then 
things may get messy.


 If, unfortunately, I happen to run into a conflict, i.e my project uses
 two unrelated libraries sharing a name for their top-level namespace,
 there's no way out for me, right?
 (Except relying on shared objects to isolate symbols)


Yeah, as far as I can tell, you would have to change the name of one of 
the libraries, and update all related module declarations and imports.


[1] https://en.wikipedia.org/wiki/Reverse_domain_name_notation
Mar 03 2016
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 3 March 2016 at 20:05:03 UTC, ag0aep6g wrote:

 [1] https://en.wikipedia.org/wiki/Reverse_domain_name_notation


LOL at the code section of that page.
Mar 03 2016
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 2 March 2016 at 20:40:39 UTC, Sebastien Alaiwan 
wrote:


 However, it feels wrong that modules (foo, bar) inside the same 
 package ("pkg") need to repeat the "absolute" package name when 
 referencing each other, I mean "import pkg.foo" instead of 
 "import foo". Not because of the extra typing, but because of 
 the dependency it introduces. But as I said, maybe my vision is 
 wrong.


The package name is part of the module name. If you use 
thislib.somemod and thatlib.somemod, how would you disambiguate 
them without the package name? Even in C and C++, libraries are 
often distributed with header files in a sub directory, where the 
top-level directory is intended to be added to the include path 
and not the subdirecotry, otherwise two includes with the same 
name conflict. Some libraries go further and prefix their header 
files with a namespace (like sdl_foo.h, sdl_bar.h), because they 
get no help from the compiler with this. Making the package name 
part of the fully qualified module name helps minimize the 
likelihood of such conflicts. It also gives a natural namespace 
for conflicting symbols in two modules named foo. You can use 
thispack.somemod.foo and thatpack.somemod.foo to disambiguate.



 - I only work with separate compilation, using gdc (Windows-dmd 
 produces OMF, which is a dealbreaker ; but I use rdmd a lot for 
 small programs).


-m64 or -m32mscoff


 - I often mix C code with D, sometimes, C++ code. I compile 
 them using gcc/g++.


Assuming MinGW, the GDC is definitely your best bet here. The 
flags above will get DMD to produce COFF for the MS linker, but 
MinGW COFF and MS COFF are not always compatible.



 With this flat directory structure, the only way test.d can 
 import files is by saying, "import stack;", not  "import 
 lib_algo.stack".
 (adding ".." to the list of import directories might do the 
 trick, but this would definitely feel wrong).


It is wrong. You should add ../mylibs :)



 Now, let's see a project using this "lib_algo", I call it 
 "myGame".

 $ cd myProjects
 $ find <files relevant to this discussion>
 myGame/lib_algo/options.d
 myGame/lib_algo/....
 myGame/lib_algo/array2d.d
 myGame/lib_algo/project.mk
 myGame/lib_algo/Makefile
 myGame/src/main.d
 myGame/Makefile


 "lib_algo" simply is an external (aka "git submodule"), 
 pointing to a specific revision of the repository "lib_algo".
 The top-level "Makefile" of the projects includes 
 "lib_algo/project.mk", and all is well, I can compose the list 
 of source files to compile without having to rely on 
 "convenience/intermediate" libraries.

 Now, if "src/main.d" wants to import "array2d.d", currently, I 
 must write "import array2d.d", and add "-I myGame/lib_algo" to 
 my compiler command line.
 I said in a previous post that this would not scale.
 That's because with this scheme, obviously, I can't have two 
 modules having the same name, even if they belong to different 
 libraries.
 Let's say I have "lib_algo/utils.d" and "lib_geom/utils.d", if 
 "main.d" says "import utils.d", it's ambiguous.


Assuming that all of the libraries you are going to use are set 
up the same way, then you can simply pass -ImyGame. Then you can 
use lib_algo.utils, lib_geom.utils, and so on. What's the problem 
with that?


 Clearly, "main.d" needs to say "import lib_algo.utils". 
 However, if I do this,
 I must add a module declaration in "lib_algo/utils.d", and I 
 must change
 "lib_algo/test.d" so it says "import lib_algo.utils".

 This is where it begins to feel clumsy. Why should lib_algo 
 need to be
 modified in order to resolve an ambiguity happening in "myGame" 
 ?


You shouldn't have to "change" it. This is how it should be set 
up from the beginning. The idea is that you decide on a module 
structure as soon as you start your project and then you stick to 
it.

The module declaration is necessary when using packages because 
otherwise DMD has no idea how far up the path to go to create a 
default package name. That's why the it uses only the file name 
when no module declaration is present. The package name is the 
solution to *prevent* ambiguities, not to resolve them after the 
fact. If you don't want lib_algo/util.d and lib_geom/util.d to 
conflict, then the use both lib_algo and lib_geom as package 
names in module declarations. How else would you propose 
resolving the ambiguity?


 Moreover, we saw earlier that this modification of the import 
 done by
 "lib_algo/test.d" had consequences on the directory structure 
 of lib_algo.
 In order to be able to build the test executable for lib_algo, 
 I now need to
 have the following structure:
 $ find lib_algo
 lib_algo/lib_algo/options.d
 lib_algo/lib_algo/....
 lib_algo/lib_algo/array2d.d
 lib_algo/test.d
 (test.d can be put inside the inner "lib_algo")

 Am I the only one to find this counterintuitive - and probably 
 wrong?
 What am I missing there?


No, you don't have to set it up that way. If your top-level 
directory is always named the same as your top-level package 
name, then you can just use its parent directory.

mylibs/lib_algo/*.d
mylibs/lib_geom/*.d

dmd -Imylibs main.d

Done. Does not the same issue exist in C++? You either have to 
keep all of your headers in a single directory or pass multiple 
directories on the command line with -I.

What I used to do before I started using DUB exclusively was 
model the system I used for C and C++ projects. For every new 
project I created, I had created an imports subdirectory in the 
project, then ran a script (written in D) to copy over the source 
tree of the specific version of each library I wanted to use. 
Then I added -Iimport on the command line when compiling. Done.

The package and module system is very intuitive to me and has 
been from the beginning, so I'm still not quite sure what it is 
about it that bothers you. The two major issues you bring up 
here, the need to import package.module and how to structure your 
projects, are easily resolved. Am I misunderstanding something?
Mar 02 2016
parent Mike Parker <aldacron gmail.com> writes:
On Thursday, 3 March 2016 at 01:47:08 UTC, Mike Parker wrote:


 The package and module system is very intuitive to me and has 
 been from the beginning, so I'm still not quite sure what it is 
 about it that bothers you. The two major issues you bring up 
 here, the need to import package.module and how to structure 
 your projects, are easily resolved. Am I misunderstanding 
 something?


I should also say that I get the impression that you aren't 
considering the package name as part of the module name. Maybe 
that's the root of your problem. As I said in a previous post, 
your desire to use the default package for everything so that you 
can simply import foo is perfectly fine. The price of doing that, 
as you point out here, is that you can easily run into conflicts. 
Packages are the solution to avoiding them and package names are 
part of the module name, such that module foo is no longer just 
foo, but mypack.foo. Without a module declaration, the compiler 
would need some other way to know that 
C:\\src\libs\mylib\mypack\foo.d is actually mypack.foo and not 
c.src.lib.mylib.mypack.foo (which again, is why it only uses the 
file name when no module declaration is given). I can't think of 
a better way to do it than putting it right there in the source 
file. Using a command line switch would not be scalable.
Mar 02 2016
prev sibling parent Guillaume Piolat <contact gam3sfrommars.fr> writes:
On Wednesday, 2 March 2016 at 06:40:53 UTC, Sebastien Alaiwan 
wrote:

 Yeah, this is what I going to do: flat hierarchy, like one 
 'src' directory and one 'extra' directory in the project root, 
 globally unique package names, passing -Isrc and -Iextra to the 
 compiler, and ... module declaration directives.
 I was trying to avoid them, but maybe it's not possible.


TBH I'm not sure what module directives do now that you mention 
them.
Mar 02 2016