digitalmars.D - Header-Only Library - pragma(root)
- Walter Bright (28/28) Jun 14 2020 There is much misunderstanding in the "What does 'inline' mean?" thread.
- Tove (3/14) Jun 14 2020 I like this feature.
- Max Samukha (5/7) Jun 14 2020 We have -i already for automatic inclusion of imported modules:
- =?UTF-8?Q?Ali_=c3=87ehreli?= (10/15) Jun 14 2020 -i would mean "assume everything has pragma(auto_add)".
- Max Samukha (4/20) Jun 14 2020 Yes. However, you can specify patterns as strict as matching
- Walter Bright (3/17) Jun 14 2020 I don't know what -i does exactly, or why it was added.
- Stanislav Blinov (2/5) Jun 14 2020
- Mathias LANG (4/21) Jun 14 2020 https://github.com/dlang/dmd/pull/7099
- Walter Bright (7/9) Jun 14 2020 Thank you. I see I was involved in the discussion.
- Stefan Koch (6/16) Jun 14 2020 I agree with everything except for the name.
- Stefan Koch (6/21) Jun 14 2020 I take it back.
- Adam D. Ruppe (3/5) Jun 14 2020 That's not true. -i just works with almost any D code. It is
- Andrei Alexandrescu (3/9) Jun 14 2020 What code wouldn't it work with? Would be good to document those. "How
- Adam D. Ruppe (38/39) Jun 14 2020 It isn't so much code as it is some build paradigms.
- Mathias LANG (12/26) Jun 16 2020 On the other hand, you rely on the person checking out the
- Adam D. Ruppe (19/21) Jun 16 2020 Which is required in ALL cases by the dub approach, you just
- Mathias LANG (24/44) Jun 16 2020 Not quite, only `-I` is required, not `-mv`, which is fair IMO,
- Martin Tschierschke (3/7) Jun 17 2020 +1 !
- Petar Kirov [ZombineDev] (59/68) Jun 16 2020 The '-i' flag is the symmetric opposite of '-I'. '-I' expects
- Petar Kirov [ZombineDev] (6/9) Jun 16 2020 I meant:
- Walter Bright (5/11) Jun 14 2020 If you're building a project using separate compilation, I doubt you'd w...
- Adam D. Ruppe (5/8) Jun 14 2020 If you're using separate compilation, you won't pass -i.
- Adam D. Ruppe (5/6) Jun 14 2020 -i is the most exciting thing that happened in the compiler from
- Seb (10/16) Jun 14 2020 I fully agree, but there's a very sad side-story to it:
- Adam D. Ruppe (5/6) Jun 14 2020 Indeed, rund is a practical piece of work that actually adds
- Paolo Invernizzi (5/11) Jun 14 2020 +1000
- novice2 (2/6) Jun 16 2020 +100500
- Johannes Pfau (33/37) Jun 14 2020 If it really does only that, it'll lead to duplicate symbol errors as
- Stefan Koch (2/9) Jun 14 2020 Proof of concept here: https://github.com/dlang/dmd/pull/11272
There is much misunderstanding in the "What does 'inline' mean?" thread. I've slowly come to realize that it's not about inlining at all, it's about a desire for "header-only" libraries. To use a header-only library, you only have to import the module for it. This works today in D if you also add the imported module to the command line. But, of course, it's good to make it simpler. So, I propose the addition of `pragma(root);` which looks like: --- electron.di --- module electron; pragma(root); // header-only library! int orbit() { return 3; } --- atom.d --- module atom; import core.stdc.stdio; import electron; void main() { printf("%d orbits\n", orbit()); } -------------- To compile, just: dmd atom and the pragma(root) will cause the compiler to behave as if you'd typed: dmd atom electron Not only does this make for header-only libraries, but if you have a project that is split up into several modules, it becomes super convenient to compile it if the other modules are all annotated with pragma(root). Note that there's no "inline" anywhere. Inlining will remain about inlining, and nothing else.
Jun 14 2020
On Sunday, 14 June 2020 at 08:36:50 UTC, Walter Bright wrote:There is much misunderstanding in the "What does 'inline' mean?" thread. I've slowly come to realize that it's not about inlining at all, it's about a desire for "header-only" libraries. To use a header-only library, you only have to import the module for it. This works today in D if you also add the imported module to the command line. But, of course, it's good to make it simpler. So, I propose the addition of `pragma(root);` which looks like: --- electron.di --- module electron; pragma(root); // header-only library!I like this feature. Could *.di always be root?
Jun 14 2020
On Sunday, 14 June 2020 at 08:36:50 UTC, Walter Bright wrote:To compile, just: dmd atomWe have -i already for automatic inclusion of imported modules: dmd atom -i=. What does the pragma add to this? I think Manu was asking for something different.
Jun 14 2020
On 6/14/20 2:33 AM, Max Samukha wrote:We have -i already for automatic inclusion of imported modules: dmd atom -i=. What does the pragma add to this?-i would mean "assume everything has pragma(auto_add)". Note how I could not say "root". :) Where does it come from? Why root? Most programmers wouldn't understand it at all. "Header" wouldn't mean anything to anybody unless they are coming from C or C++. And you said "automatic inclusion". Perhaps pragma(auto_include) but then it would confuse C and C++ programmers because they "include" header files. Tough naming situation... :) Ali
Jun 14 2020
On Sunday, 14 June 2020 at 09:39:52 UTC, Ali Çehreli wrote:On 6/14/20 2:33 AM, Max Samukha wrote:Yes. However, you can specify patterns as strict as matching single modules. Anyway, I can see the difference.We have -i already for automatic inclusion of imported modules: dmd atom -i=. What does the pragma add to this?-i would mean "assume everything has pragma(auto_add)".Note how I could not say "root". :) Where does it come from? Why root? Most programmers wouldn't understand it at all. "Header" wouldn't mean anything to anybody unless they are coming from C or C++. And you said "automatic inclusion". Perhaps pragma(auto_include) but then it would confuse C and C++ programmers because they "include" header files. Tough naming situation... :)pragma(static) ))Ali
Jun 14 2020
On 6/14/2020 2:33 AM, Max Samukha wrote:On Sunday, 14 June 2020 at 08:36:50 UTC, Walter Bright wrote:I don't know what -i does exactly, or why it was added. https://dlang.org/dmd-windows.html#switch-iTo compile, just: dmd atomWe have -i already for automatic inclusion of imported modules: dmd atom -i=. What does the pragma add to this? I think Manu was asking for something different.
Jun 14 2020
On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:I don't know what -i does exactly, or why it was added. https://dlang.org/dmd-windows.html#switch-ihttps://dlang.org/dmd-windows.html#switch-i[.<
Jun 14 2020
On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:On 6/14/2020 2:33 AM, Max Samukha wrote:https://github.com/dlang/dmd/pull/7099 It's essentially putting rdmd in the compiler because rdmd was considered too slow.On Sunday, 14 June 2020 at 08:36:50 UTC, Walter Bright wrote:I don't know what -i does exactly, or why it was added. https://dlang.org/dmd-windows.html#switch-iTo compile, just: dmd atomWe have -i already for automatic inclusion of imported modules: dmd atom -i=. What does the pragma add to this? I think Manu was asking for something different.
Jun 14 2020
On 6/14/2020 2:52 AM, Mathias LANG wrote:https://github.com/dlang/dmd/pull/7099 It's essentially putting rdmd in the compiler because rdmd was considered too slow.Thank you. I see I was involved in the discussion. The issue is then should the implementer of the module decide if it is header-only (i.e. pragma(root)) or the caller (-i). It would probably be better as the implementer, as a header-only library has to be designed for it. It also means the user needn't need to set the switches, he can just compile.
Jun 14 2020
On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:On 6/14/2020 2:52 AM, Mathias LANG wrote:I agree with everything except for the name. I same sure there are better names than root. Perhaps extern(static) ? Also this needs to be either restricted to leaf functions, or draw in callees as well.https://github.com/dlang/dmd/pull/7099 It's essentially putting rdmd in the compiler because rdmd was considered too slow.Thank you. I see I was involved in the discussion. The issue is then should the implementer of the module decide if it is header-only (i.e. pragma(root)) or the caller (-i). It would probably be better as the implementer, as a header-only library has to be designed for it. It also means the user needn't need to set the switches, he can just compile.
Jun 14 2020
On Sunday, 14 June 2020 at 10:35:51 UTC, Stefan Koch wrote:On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:I take it back. I thought this was per_symbol rather than per_module. I would be on-board with doing this per symbol. And infact I am implementing it right now. To see where the challenges are.On 6/14/2020 2:52 AM, Mathias LANG wrote:I agree with everything except for the name.https://github.com/dlang/dmd/pull/7099 It's essentially putting rdmd in the compiler because rdmd was considered too slow.Thank you. I see I was involved in the discussion. The issue is then should the implementer of the module decide if it is header-only (i.e. pragma(root)) or the caller (-i). It would probably be better as the implementer, as a header-only library has to be designed for it. It also means the user needn't need to set the switches, he can just compile.
Jun 14 2020
On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:It would probably be better as the implementer, as a header-only library has to be designed for it.That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
Jun 14 2020
On 6/14/20 11:36 AM, Adam D. Ruppe wrote:On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:What code wouldn't it work with? Would be good to document those. "How to write a header-only D library". Blog post!It would probably be better as the implementer, as a header-only library has to be designed for it.That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
Jun 14 2020
On Sunday, 14 June 2020 at 16:05:39 UTC, Andrei Alexandrescu wrote:What code wouldn't it work with?It isn't so much code as it is some build paradigms. If you are forced to do separate compilation (like if dmd is eating 30 GB of RAM...), you can't enjoy dmd -i just working. But you can still use it in individual pieces then link the result together normally, so it doesn't get in your way, it just doesn't help as much. If you do it wrong in these cases, you are liable to get duplicate symbol problems with a diamond dependency (two packages compiled separately that both include a common third package). All manageable and -i can still help in these cases with appropriate inclusion/exclusion patterns, just -i's biggest strength is when you are doing an all-at-once build. Ditto with non-module based dependency versions or compilation flags, since those again bring you back to separate compilation. The other semi-tricky thing is dub's default source layout doesn't follow the convention -i expects due to having the extra source/ directory. So you need to add it with -Iproject/source each time. Or you can automatically rearrange files from a package upon downloading it (just read the `module` declaration and sort it into directories from that under a common `lib` top-level thing. This is actually what I do on my computer and why my arsd repo has most its files at top-level - if you clone the repo in your lib dir, the files are now right at `arsd/cgi.d` etc... which automatically works with the compiler. Zero extra config needed, it all just works. I get a lot of complaints about that layout but this is an objective advantage I don't think a lot of people really realize!) I've been tempted before to write a competing package manager that does stuff like this all automatically. Imagine you just write some D and `import foo.bar;` then the compiler (or, more specifically, the helper program that works with the compiler) tells you "module foo.bar is not installed but can be found in package `footech`, install now? (y/n)". And if you made the major version part of the module names instead of git tags, that'd all magically just work too. I'd be kinda fun and very efficient thanks to dmd -i.
Jun 14 2020
On Monday, 15 June 2020 at 00:55:59 UTC, Adam D. Ruppe wrote:The other semi-tricky thing is dub's default source layout doesn't follow the convention -i expects due to having the extra source/ directory. So you need to add it with -Iproject/source each time. Or you can automatically rearrange files from a package upon downloading it (just read the `module` declaration and sort it into directories from that under a common `lib` top-level thing. This is actually what I do on my computer and why my arsd repo has most its files at top-level - if you clone the repo in your lib dir, the files are now right at `arsd/cgi.d` etc... which automatically works with the compiler. Zero extra config needed, it all just works. I get a lot of complaints about that layout but this is an objective advantage I don't think a lot of people really realize!)On the other hand, you rely on the person checking out the repository with a specific directory name, which is IMO much worse than the dub approach. If I check it out as "arsd-v1.1.1", suddenly it breaks, unless I use two DMD flags to solve the situation (-I and -mv). Additionally, many people want to separate their code and their diet files / config file / scripts / whatever and having the project's source file at the root make this impossible. So yeah I'll take the dub approach over this any day of the week, I just would like dub to generate source/projectname/ and not just source/
Jun 16 2020
On Tuesday, 16 June 2020 at 13:27:00 UTC, Mathias LANG wrote:If I check it out as "arsd-v1.1.1", suddenly it breaks, unless I use two DMD flags to solve the situation (-I and -mv).Which is required in ALL cases by the dub approach, you just don't notice it as much because `dub build` sets all that based on the settings in the dub config file (or their conventional defaults). But you lose something there: optional modules are compiled in by dub (unless you go through the config process to separate it), whereas with dmd -i they would not be processed at all. It is also impossible to mix versions of dependencies with the dub approach; you can rename that directory to -v1.1.1 but put in a -v1.2.1 next to it and you just have trouble (which is why dub resolves it down to just the one compatible version, or fails if it can't find one...) Maybe one of these days I'll actually write my competing package manager and demo all this stuff (you wouldn't even have to change your layouts btw since it is easy to auto-scan packages for D modules and rearrange them as needed; my doc site does exactly this). But since I don't use packages it isn't a big priority of mine.
Jun 16 2020
On Wednesday, 17 June 2020 at 02:51:20 UTC, Adam D. Ruppe wrote:On Tuesday, 16 June 2020 at 13:27:00 UTC, Mathias LANG wrote:Not quite, only `-I` is required, not `-mv`, which is fair IMO, and expected from anyone familiar with C[++].If I check it out as "arsd-v1.1.1", suddenly it breaks, unless I use two DMD flags to solve the situation (-I and -mv).Which is required in ALL cases by the dub approach, you just don't notice it as much because `dub build` sets all that based on the settings in the dub config file (or their conventional defaults).But you lose something there: optional modules are compiled in by dub (unless you go through the config process to separate it), whereas with dmd -i they would not be processed at all.I totally agree that the current situation on that regard is terrible, although that's more of a dub limitation than something inherent to the hierarchy. `dub` has a concept of `mainSourceFile` and I hope one day it'll use this to drive compilation instead of selecting all files. I have a somewhat related bug report: https://github.com/dlang/dub/issues/1778It is also impossible to mix versions of dependencies with the dub approach; you can rename that directory to -v1.1.1 but put in a -v1.2.1 next to it and you just have trouble (which is why dub resolves it down to just the one compatible version, or fails if it can't find one...)I don't quite see how that relates to the discussion. I was merely stating that relying on the external name of the directory was much worse than having the user pass `-Iproject_name/source` by command line. Nonetheless, I don't see why you would ever want to mix two semver-incompatible versions of the same library. I get that it can happen due to transitive dependencies, but there's really no way to make it foolproof for non-trivial cases, and I think everyone's time and energy are best spent avoiding this situation than catering for it.Maybe one of these days I'll actually write my competing package manager and demo all this stuff (you wouldn't even have to change your layouts btw since it is easy to auto-scan packages for D modules and rearrange them as needed; my doc site does exactly this).Contributions to `dub` are welcome, will be immediately available to users, and will benefit the community much more than yet another package manager (we have `dub`, `dud`, `reggae`, and probably a plethora of other tools).
Jun 16 2020
On Wednesday, 17 June 2020 at 03:21:30 UTC, Mathias LANG wrote: [...]Contributions to `dub` are welcome, will be immediately available to users, and will benefit the community much more than yet another package manager (we have `dub`, `dud`, `reggae`, and probably a plethora of other tools).+1 !
Jun 17 2020
On Sunday, 14 June 2020 at 16:05:39 UTC, Andrei Alexandrescu wrote:On 6/14/20 11:36 AM, Adam D. Ruppe wrote:The '-i' flag is the symmetric opposite of '-I'. '-I' expects either header-only libraries, or separate compilation, while '-i' expects neither. What Walter is suggesting (marking modules with 'pragma (root);' to make them header-only mainly serves the '-I' flag. The only advantage 'pragma (root);' may have to '-i' is a compile-time guarantee that modules intended to be header-only actually don't impose any link-time dependencies. The main use case of the `-i` flag is to compile (not just import) modules. However the killer feature of '-i' is that while '-I' a is blunt tool that just adds all modules under a dir to the import path, '-i' leverages D's import system and lazily compiles only the modules that are imported starting from the root set of modules. For example, giving the following project structure: project/ ├── lib1 │ ├── mod1.d │ ├── ... │ └── modn.d ├── lib2 │ ├── mod1.d │ ├── ... │ └── modn.d └── main.d Previously, in order to compile it, you could do the following: find lib1/ lib2/ -type f -name '*.d' | xargs dmd -betterC main.d However, this has the disadvantage of compiling stuff that main.d doesn't need. Say main.d was this: extern (C) int main(int argc, const char** argv) { import lib1.mod1 : fun1a; return fun1a(argc, argc + 1); } The only function that ever needs to be compiled is fun1a. However the command above will end up compiling the whole world (every module under lib1/). With the '-i' flag we can compile main.d like so: dmd -betterC -i=lib1 -i=lib2 main.d Now, set of modules that would be compiled equals precisely the set of modules imported from main.d - in this case only lib1/mod1.d. The only deficiency currently is that it doesn't take advantage of selective imports - each imported module is compiled as a whole, even if you only need to compile the imported symbols. This problem is not just an implementation deficiency though, as whether something is compiled and whether it exists is somewhat conflated and has impact on D's meta programming (DbI and all that). I think the best way to address this issue is at the root - we need to define precise semantics over what lazy compilation means. I have been meaning to write a DIP that addresses this issue for a while, based on the idea of an export attribute (orthogonal to public/protected/package/private access modifiers) and export inference. You can see a sketch of what I have in mind in these posts: https://forum.dlang.org/post/xbllqrpvflazfpowizwj forum.dlang.org https://forum.dlang.org/post/nvvgrdlucajshjngdjlj forum.dlang.orgOn Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:What code wouldn't it work with? Would be good to document those. "How to write a header-only D library". Blog post!It would probably be better as the implementer, as a header-only library has to be designed for it.That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
Jun 16 2020
On Tuesday, 16 June 2020 at 07:06:16 UTC, Petar Kirov [ZombineDev] wrote:The '-i' flag is the symmetric opposite of '-I'. '-I' expects either header-only libraries, or separate compilation, while '-i' expects neither.I meant: The '-i' flag is the symmetric opposite of '-I'. '-I' requires either header-only libraries, or separate compilation, while '-i' *requires neither*.
Jun 16 2020
On 6/14/2020 8:36 AM, Adam D. Ruppe wrote:On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:If you're building a project using separate compilation, I doubt you'd want an import with large functions in it being compiled over and over into multiple object files. Also, you probably don't want dependencies leaking into the imported code.It would probably be better as the implementer, as a header-only library has to be designed for it.That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
Jun 14 2020
On Monday, 15 June 2020 at 00:17:42 UTC, Walter Bright wrote:If you're building a project using separate compilation, I doubt you'd want an import with large functions in it being compiled over and over into multiple object files.If you're using separate compilation, you won't pass -i. You might pass -i=-foo.bar though; the optional pattern part of the argument is pretty flexible to pick and choose what you do and do not want.
Jun 14 2020
On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:I don't know what -i does exactly, or why it was added.-i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D. Don't touch it!
Jun 14 2020
On Sunday, 14 June 2020 at 15:31:05 UTC, Adam D. Ruppe wrote:On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:I fully agree, but there's a very sad side-story to it: https://github.com/dlang/tools/pull/271 https://github.com/dlang/tools/pull/290 tl;dr: rund [1] (a modern variant of rdmd without the compile twice bug + useful features like including flags inside the file header) was a result of this revert debacle. At this point rund is vastly superior to rdmd and there's almost zero talk about it (both: the rdmd debacle + rund). [1] https://github.com/dragon-lang/rundI don't know what -i does exactly, or why it was added.-i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D. Don't touch it!
Jun 14 2020
On Sunday, 14 June 2020 at 16:19:54 UTC, Seb wrote:I fully agree, but there's a very sad side-story to it:Indeed, rund is a practical piece of work that actually adds value over dmd -i for some cases. Why was that rdmd change reverted? Is it just because the other compiler versions in repos didn't support -i?
Jun 14 2020
On Sunday, 14 June 2020 at 15:31:05 UTC, Adam D. Ruppe wrote:On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:+1000 That flags is the workhorse of all our builds in the company: don't touch it .... please /PaoloI don't know what -i does exactly, or why it was added.-i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D. Don't touch it!
Jun 14 2020
On Sunday, 14 June 2020 at 15:31:05 UTC, Adam D. Ruppe wrote:-i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D. Don't touch it!+100500
Jun 16 2020
Am Sun, 14 Jun 2020 01:36:50 -0700 schrieb Walter Bright:and the pragma(root) will cause the compiler to behave as if you'd typed: dmd atom electronIf it really does only that, it'll lead to duplicate symbol errors as soon as electron introces any symbols which do not have weak linkage. This includes global variables, but that may be expected not to work in header-only libraries. This problem however also occurs because of initZ, TypeInfo and other special symbol generated for structs and classes. As mentioned in the other post, even if you don't declare structs or classes, right now this does not work because of duplicate ModuleInfo. But those are implementation issues. Regarding the feature: I don't really like the name "root". I also wonder whether it would not be better to unify all "emit to object file" use cases into one pragma: // CTFE only function (no codegen) pragma(emit, never) void foo() {} // can be used at runtime (codegen) but never output a symbol pragma(emit, inliner) // Only once, in the defining module / CU pragma(emit, normal) // Once into each into CUs importing this. Not into defining CU pragma(emit, imports) // Emit into all importing and defining CU pragma(emit, all) It should be noted that due to D's name mangling, whenever symbols are emitted multiple times (imports, all) they have to be weak and will be merged by linker, so you never get multiple copies. If a static inline effect is desired (emit multiple copies with independent addresses), we'd need pragma(emit, all_copy) and pragma(import_copy). However, I'm not sure if this should be part of pragma(emit), or an additional orthogonal pragma. -- Johannes
Jun 14 2020
On Sunday, 14 June 2020 at 16:42:36 UTC, Johannes Pfau wrote:Am Sun, 14 Jun 2020 01:36:50 -0700 schrieb Walter Bright:Proof of concept here: https://github.com/dlang/dmd/pull/11272[...]If it really does only that, it'll lead to duplicate symbol errors as soon as electron introces any symbols which do not have weak linkage. This includes global variables, but that may be expected not to work in header-only libraries. [...]
Jun 14 2020