www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Our template emission strategy is broken

reply David Nadlinger <code klickverbot.at> writes:
Hi all,

Kenji and Walter have been working on improving the template 
emission strategy during the last couple of releases, i.e. 
whether a template instance is emitted to a given object file or 
not. Nevertheless, I've been continually hearing complaints from 
various people with large D code bases (our commercial users and 
some of the more complex open source projects) that they have 
problems compiling their code doing anything else than an 
all-at-once build.

This of course detracts from what we like to cite as one of D's 
key advantages, compilation speed, because you cannot exploit the 
many cores of your build machine anymore, and the 
edit-compile-debug cycle is slowed down because you have to build 
the full application every time. But this turns into a severe 
problem as soon as your compilation does not fit into RAM 
anymore, which happens quite quickly when using D's advanced 
features – see for example Liran Zvibel's DConf talk, where he 
describes that they needed machines with more than 100 GB of RAM 
in order to build the Weka code base.

In any case, I hope you agree that fixing these kinds of issues 
that prevent D code from being compiled all is strategically 
important for us, since they are a deciding factor in driving 
widespread adoption. Sadly, the template problems didn't seem to 
receive a lot of attention beyond quick fixes recently, possibly 
because they don't occur as readily for smaller projects and if 
so, are easier to work around.

With all that out of my system, I'd like to point your attention 
to a particular template instantiation issue I managed to reduce 
recently when working with the folks at Weka. It seems like there 
is a fundamental oversight in the way the code tries to elide 
repeat code generation of instances – or, of course, I'm just 
missing something:

https://issues.dlang.org/show_bug.cgi?id=15318

This is one particular issue that makes writing "idiomatic D" 
(which is rather template-heavy) in large code bases hard, but 
likely not the only one. In fact, I already know about another 
issue where function-local imports cause semantic analysis not to 
be properly run on template instances, but I'm still struggling 
to find a minimal example for the problem. My hope would be that 
we can clean up this mess soon and replace the current ad-hoc 
patchwork with a more principled approach. In the process, we 
should also make sure that all of our assumptions [1] about the 
compilation process are clearly stated in the documentation, as 
that frequently leads to confusion among newcomers.

Best,
David



[1] That is: All imported modules must also be compiled into the 
executable; incremental compilation is only guaranteed to work if 
precisely the same subsets of modules are compiled every time.
Nov 11 2015
next sibling parent Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

 Hi all,

 Nevertheless, I've been continually hearing complaints from 
 various people with large D code bases (our commercial users 
 and some of the more complex open source projects) that they 
 have problems compiling their code doing anything else than an 
 all-at-once build.


The same here, we are not able to compile two of our products 
without an all-at-once build.

Actually we are plagued also by that:

https://issues.dlang.org/show_bug.cgi?id=14517

That's a real pain...

---
Paolo
Nov 11 2015
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 11 Nov 2015 13:08:06 +0000
schrieb David Nadlinger <code klickverbot.at>:


 Hi all,
=20
 Kenji and Walter have been working on improving the template=20
 emission strategy during the last couple of releases, i.e.=20
 whether a template instance is emitted to a given object file or=20
 not. Nevertheless, I've been continually hearing complaints from=20
 various people with large D code bases (our commercial users and=20
 some of the more complex open source projects) that they have=20
 problems compiling their code doing anything else than an=20
 all-at-once build.
=20
 This of course detracts from what we like to cite as one of D's=20
 key advantages, compilation speed, because you cannot exploit the=20
 many cores of your build machine anymore, and the=20
 edit-compile-debug cycle is slowed down because you have to build=20
 the full application every time. But this turns into a severe=20
 problem as soon as your compilation does not fit into RAM=20
 anymore, which happens quite quickly when using D's advanced=20
 features =E2=80=93=C2=A0see for example Liran Zvibel's DConf talk, where =


he=20

 describes that they needed machines with more than 100 GB of RAM=20
 in order to build the Weka code base.
=20
 In any case, I hope you agree that fixing these kinds of issues=20
 that prevent D code from being compiled all is strategically=20
 important for us, since they are a deciding factor in driving=20
 widespread adoption. Sadly, the template problems didn't seem to=20
 receive a lot of attention beyond quick fixes recently, possibly=20
 because they don't occur as readily for smaller projects and if=20
 so, are easier to work around.
=20
 With all that out of my system, I'd like to point your attention=20
 to a particular template instantiation issue I managed to reduce=20
 recently when working with the folks at Weka. It seems like there=20
 is a fundamental oversight in the way the code tries to elide=20
 repeat code generation of instances =E2=80=93 or, of course, I'm just=20
 missing something:
=20
 https://issues.dlang.org/show_bug.cgi?id=3D15318
=20
 This is one particular issue that makes writing "idiomatic D"=20
 (which is rather template-heavy) in large code bases hard, but=20
 likely not the only one. In fact, I already know about another=20
 issue where function-local imports cause semantic analysis not to=20
 be properly run on template instances, but I'm still struggling=20
 to find a minimal example for the problem. My hope would be that=20
 we can clean up this mess soon and replace the current ad-hoc=20
 patchwork with a more principled approach. In the process, we=20
 should also make sure that all of our assumptions [1] about the=20
 compilation process are clearly stated in the documentation, as=20
 that frequently leads to confusion among newcomers.
=20
 Best,
 David
=20
=20
=20
 [1] That is: All imported modules must also be compiled into the=20
 executable; incremental compilation is only guaranteed to work if=20
 precisely the same subsets of modules are compiled every time.



http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com
Nov 11 2015
parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 11 Nov 2015 14:44:39 +0100
schrieb Johannes Pfau <nospam example.com>:


 Am Wed, 11 Nov 2015 13:08:06 +0000
 schrieb David Nadlinger <code klickverbot.at>:
=20

 Hi all,
=20
 Kenji and Walter have been working on improving the template=20
 emission strategy during the last couple of releases, i.e.=20
 whether a template instance is emitted to a given object file or=20
 not. Nevertheless, I've been continually hearing complaints from=20
 various people with large D code bases (our commercial users and=20
 some of the more complex open source projects) that they have=20
 problems compiling their code doing anything else than an=20
 all-at-once build.
=20
 This of course detracts from what we like to cite as one of D's=20
 key advantages, compilation speed, because you cannot exploit the=20
 many cores of your build machine anymore, and the=20
 edit-compile-debug cycle is slowed down because you have to build=20
 the full application every time. But this turns into a severe=20
 problem as soon as your compilation does not fit into RAM=20
 anymore, which happens quite quickly when using D's advanced=20
 features =E2=80=93=C2=A0see for example Liran Zvibel's DConf talk, wher=




e he=20

 describes that they needed machines with more than 100 GB of RAM=20
 in order to build the Weka code base.




A quick workaround could be enabling the GC for DDMD. IIRC I read
somewhere on github that the segfaulting code was actually rewritten
now and the GC might just work.


=20
 In any case, I hope you agree that fixing these kinds of issues=20
 that prevent D code from being compiled all is strategically=20
 important for us, since they are a deciding factor in driving=20
 widespread adoption. Sadly, the template problems didn't seem to=20
 receive a lot of attention beyond quick fixes recently, possibly=20
 because they don't occur as readily for smaller projects and if=20
 so, are easier to work around.
=20
 With all that out of my system, I'd like to point your attention=20
 to a particular template instantiation issue I managed to reduce=20
 recently when working with the folks at Weka. It seems like there=20
 is a fundamental oversight in the way the code tries to elide=20
 repeat code generation of instances =E2=80=93 or, of course, I'm just=20
 missing something:
=20
 https://issues.dlang.org/show_bug.cgi?id=3D15318
=20
 This is one particular issue that makes writing "idiomatic D"=20
 (which is rather template-heavy) in large code bases hard, but=20
 likely not the only one. In fact, I already know about another=20
 issue where function-local imports cause semantic analysis not to=20
 be properly run on template instances, but I'm still struggling=20
 to find a minimal example for the problem. My hope would be that=20
 we can clean up this mess soon and replace the current ad-hoc=20
 patchwork with a more principled approach. In the process, we=20
 should also make sure that all of our assumptions [1] about the=20
 compilation process are clearly stated in the documentation, as=20
 that frequently leads to confusion among newcomers.
=20
 Best,
 David
=20
=20
=20
 [1] That is: All imported modules must also be compiled into the=20
 executable; incremental compilation is only guaranteed to work if=20
 precisely the same subsets of modules are compiled every time. =20


=20
=20
 http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com
=20


Sorry, sent the wrong message ;-)

Ellery Newcomer recently reported a template emission bug where
templates are emitted twice:
http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com=20

I think we should really fix these issues, working separate compilation
is very important.
Nov 11 2015
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:

 I think we should really fix these issues, working separate 
 compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design 
(b/c all dependencies get imported over and over again, just like 
headers).
We should endorse compiling small (rather independent) packages 
to static/shared libraries.
Nov 11 2015
next sibling parent reply Dicebot <public dicebot.lv> writes:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:

 I think we should really fix these issues, working separate 
 compilation is very important.


 No, it's not. Separate compilation is C++-ishly slow by design 
 (b/c all dependencies get imported over and over again, just 
 like headers).
 We should endorse compiling small (rather independent) packages 
 to static/shared libraries.


Note that many of template emission issues that affect "plain" 
separate compilation are likely to affect package-based static 
libraties as well, thus in context of this thread distinction is 
not very important.
Nov 11 2015
parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 14:06:43 UTC, Dicebot wrote:

 Note that many of template emission issues that affect "plain" 
 separate compilation are likely to affect package-based static 
 libraties as well, thus in context of this thread distinction 
 is not very important.


Yes, you can get similar issues with package libraries, but only 
if the packages are mutually dependent on each other. Think of 
that for a moment, no package manager allows you to have cycles 
in your dependencies.

The case where you do compile packages with circular dependencies 
seems to mostly come from artificially splitting packages to deal 
with OOM, and we better address that at it's root.

In general you save many troubles with a library b/c you always 
link with all objects that were compiled (so it's only important 
that a template was emitted but not into which object).
Nov 11 2015
next sibling parent reply Dicebot <public dicebot.lv> writes:
On Wednesday, 11 November 2015 at 14:23:16 UTC, Martin Nowak 
wrote:

 On Wednesday, 11 November 2015 at 14:06:43 UTC, Dicebot wrote:

 Note that many of template emission issues that affect "plain" 
 separate compilation are likely to affect package-based static 
 libraties as well, thus in context of this thread distinction 
 is not very important.


 Yes, you can get similar issues with package libraries, but 
 only if the packages are mutually dependent on each other. 
 Think of that for a moment, no package manager allows you to 
 have cycles in your dependencies.

 The case where you do compile packages with circular 
 dependencies seems to mostly come from artificially splitting 
 packages to deal with OOM, and we better address that at it's 
 root.

 In general you save many troubles with a library b/c you always 
 link with all objects that were compiled (so it's only 
 important that a template was emitted but not into which 
 object).


I was referring to D packages, not dub packages.
Nov 11 2015
parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 14:26:28 UTC, Dicebot wrote:

 I was referring to D packages, not dub packages.


Not much of a difference, compiling something separately that has 
mutual dependencies is flawed but can be made working to some 
extend.
Nov 11 2015
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 11 Nov 2015 14:23:15 +0000
schrieb Martin Nowak <code dawg.eu>:


 Think of that for a moment, no package manager allows you to
 have cycles in your dependencies.


There are package managers that allow packages to mutually
depend on each other. Order is established by qualifying the
dependency similar to how you declare an owning and a weak
reference in a parent/child pointer relation.
E.g. You must install DMD before druntime/Phobos, but DMD is
only usable after druntime/Phobos has also been installed.

-- 
Marco
Nov 16 2015
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, 16 November 2015 at 09:05:16 UTC, Marco Leise wrote:

 Am Wed, 11 Nov 2015 14:23:15 +0000
 schrieb Martin Nowak <code dawg.eu>:


 Think of that for a moment, no package manager allows you to 
 have cycles in your dependencies.


 There are package managers that allow packages to mutually
 depend on each other. Order is established by qualifying the
 dependency similar to how you declare an owning and a weak
 reference in a parent/child pointer relation.


LOL. Yeah. I expect that 99+% of Linux boxes contain mutually 
dependent packages. Circular dependencies make setting up Gentoo 
that much more entertaining (e.g. IIRC, OpenSSL and Cyrus SASL 
depend on each other normally). At least some dependencies can be 
eliminated with binary packages, because they're only build 
dependencies, but not all of them are that way, unfortunately.

- Jonathan M Davis
Nov 16 2015
prev sibling next sibling parent reply Daniel Kozak via Digitalmars-d <digitalmars-d puremagic.com> writes:
V Wed, 11 Nov 2015 13:56:50 +0000
Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> napsáno:


 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:

 I think we should really fix these issues, working separate 
 compilation is very important.  


 
 No, it's not. Separate compilation is C++-ishly slow by design 
 (b/c all dependencies get imported over and over again, just like 
 headers).
 We should endorse compiling small (rather independent) packages 
 to static/shared libraries.
 


This isn't a solution it's a workaround.
It seems to me like "It's not a bug it's a feature"
Nov 11 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/15 9:14 AM, Daniel Kozak via Digitalmars-d wrote:

 V Wed, 11 Nov 2015 13:56:50 +0000
 Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> napsáno:


 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau
 wrote:

 I think we should really fix these issues, working separate
 compilation is very important.


 No, it's not. Separate compilation is C++-ishly slow by design
 (b/c all dependencies get imported over and over again, just like
 headers).
 We should endorse compiling small (rather independent) packages
 to static/shared libraries.


 This isn't a solution it's a workaround.
 It seems to me like "It's not a bug it's a feature"


Rust, Haskell, and Go have similar approaches if that helps. -- Andrei
Nov 11 2015
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 11 Nov 2015 13:56:50 +0000
schrieb Martin Nowak <code dawg.eu>:


 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:

 I think we should really fix these issues, working separate 
 compilation is very important.  


 
 No, it's not. Separate compilation is C++-ishly slow by design 
 (b/c all dependencies get imported over and over again, just like 
 headers).
 We should endorse compiling small (rather independent) packages 
 to static/shared libraries.
 


And where do you draw the line? Do we expect ARM users to have 4GB of
RAM to compile phobos? Is 512MB enough? 64 MB? Even huge C++ codebases
such as GCC compile with 128MB memory.

I use docker hub to build docker images. One Dockerfile compiles one
small D tool which depends on vibe.d for JSON parsing. Without
separate compilation the build fails as docker hub kills the build
process, out of memory...
Nov 11 2015
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
wrote:

 And where do you draw the line? Do we expect ARM users to have 
 4GB of RAM to compile phobos? Is 512MB enough? 64 MB? Even huge 
 C++ codebases such as GCC compile with 128MB memory.


OK let's clarify the terminology first.

separate compilation: -c single_source.d
incremental compilation: -c source_a.d source_b.d (as well as -c 
-ofsingle.o source_a.d source_b.d)
library compilation: -lib all_pkg_sources

Separate compilation works fine (and predictable) but is slow 
(due to reparsing), you do need to link all objects from all your 
sources though.
Choosing this strategy makes sense to mitigate OOM issues.

Incremental compilation doesn't work b/c templates are easily 
emitted into different objects, causing all sorts of linker 
errors. It works somewhat when you delete all objects and 
recompile all of them.

Library compilation works and recompiles a whole library whenever 
something changes.
It's fast b/c modules/semantic can be reused by the compiler.
Choose this strategy unless your running into OOM issues.
Nov 11 2015
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 11 Nov 2015 14:31:53 +0000
schrieb Martin Nowak <code dawg.eu>:


 On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
 wrote:

 And where do you draw the line? Do we expect ARM users to have 
 4GB of RAM to compile phobos? Is 512MB enough? 64 MB? Even huge 
 C++ codebases such as GCC compile with 128MB memory.  


 
 OK let's clarify the terminology first.
 
 separate compilation: -c single_source.d
 incremental compilation: -c source_a.d source_b.d (as well as -c 
 -ofsingle.o source_a.d source_b.d)
 library compilation: -lib all_pkg_sources
 
 Separate compilation works fine (and predictable) but is slow 
 (due to reparsing), you do need to link all objects from all your 
 sources though.
 Choosing this strategy makes sense to mitigate OOM issues.
 


OK, that makes sense. I thought there were some issues with separate
compilation as well, but I'm not sure.

Anyway, I meant working _separate_ compilation is important, incremental
compilations is much less important for me. GDC doesn't even support
multiobj style compilation.

BTW: shouldn't incremental compilation with one single output object
have exactly the same result as compiling into one static library? So I
guess we already avoid most of these problems in gdc by simply not
offering a multiobj mode ;-)
Nov 11 2015
parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 14:50:27 UTC, Johannes Pfau 
wrote:

 Anyway, I meant working _separate_ compilation is important, 
 incremental compilations is much less important for me. GDC 
 doesn't even support multiobj style compilation.


The particular issue I mentioned in the original post (that will 
hit GDC as well, by the way, if/as soon as you are using 
TemplateInstance::needsCodegen in your glue layer) has nothing to 
do with "multiobj" compilation.


 BTW: shouldn't incremental compilation with one single output 
 object have exactly the same result as compiling into one 
 static library?


Yes, and both are broken.

  — David
Nov 11 2015
prev sibling next sibling parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 14:31:54 UTC, Martin Nowak 
wrote:

 OK let's clarify the terminology first.

 separate compilation: -c single_source.d
 incremental compilation: -c source_a.d source_b.d (as well as 
 -c -ofsingle.o source_a.d source_b.d)
 library compilation: -lib all_pkg_sources

 Separate compilation works fine (and predictable) but is slow 
 (due to reparsing), you do need to link all objects from all 
 your sources though.
 Choosing this strategy makes sense to mitigate OOM issues.

 Incremental compilation doesn't work b/c templates are easily 
 emitted into different objects, causing all sorts of linker 
 errors. It works somewhat when you delete all objects and 
 recompile all of them.

 Library compilation works and recompiles a whole library 
 whenever something changes.
 It's fast b/c modules/semantic can be reused by the compiler.
 Choose this strategy unless your running into OOM issues.


That terminology seems a bit misleading for discussing the 
problem at hand. There is nothing incremental about compiling 
per-package with a single .o file, and it is in fact rather 
similar to what you call "library compilation". As noted on 
Bugzilla, the particular template instantiation bug that was 
plaguing Weka occurs with "-lib" just the same as it does with 
"-c -osingleobj.o".

  — David
Nov 11 2015
prev sibling parent Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Wednesday, 11 November 2015 at 14:31:54 UTC, Martin Nowak 
wrote:

 On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
 wrote:

 Separate compilation works fine (and predictable) but is slow 
 (due to reparsing), you do need to link all objects from all 
 your sources though.


That's not the experience we have in my company: we are not able 
to compile and link project right now, using separate compilation.

 From dmd 2.066 up to 2.069.
---
Paolo
Nov 11 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/15 9:15 AM, Johannes Pfau wrote:

 Am Wed, 11 Nov 2015 13:56:50 +0000
 schrieb Martin Nowak <code dawg.eu>:


 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau
 wrote:

 I think we should really fix these issues, working separate
 compilation is very important.


 No, it's not. Separate compilation is C++-ishly slow by design
 (b/c all dependencies get imported over and over again, just like
 headers).
 We should endorse compiling small (rather independent) packages
 to static/shared libraries.


 And where do you draw the line? Do we expect ARM users to have 4GB of
 RAM to compile phobos? Is 512MB enough? 64 MB? Even huge C++ codebases
 such as GCC compile with 128MB memory.


Large codebases are best organized to have more packages, not larger 
packages.

Phobos takes about 600MB max to build.


Andrei
Nov 11 2015
parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 15:07:15 UTC, Andrei 
Alexandrescu wrote:

 Phobos takes about 600MB max to build.


When was the last time you did "dmd -unittest -main $(find std 
-name '*.d')"? On current master, dmd takes just below 9 GiB to 
complete the build.

Also, consider that many of the features in Phobos – apart from 
the underlying molasses consisting of std.traits and friends – 
are not really widely used internally. This is not surprising, 
considering that large parts of std.* are mostly a grab-bag of 
orthogonal stuff, and the unit tests are, as they should, fairly 
isolated. For example, how many Phobos modules actually use 
std.regex internally, let alone its compile-time variants (which 
we are quick to advertise)? What about range "pipelines" longer 
than the one or two stages you'd typically write in a unit test?

Apologies for maybe being a bit too confrontational about this, 
but I don't think anybody is helped by dismissing entire classes 
of problems just because they tend not to be so apparent from a 
library writer's perspective. If you were to, say, visit the 
folks at Weka, you might be surprised to see how easy it is to do 
the latter without even realizing so.

  — David
Nov 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 12:13 PM, David Nadlinger wrote:

 On Wednesday, 11 November 2015 at 15:07:15 UTC, Andrei Alexandrescu wrote:

 Phobos takes about 600MB max to build.


 When was the last time you did "dmd -unittest -main $(find std -name
 '*.d')"? On current master, dmd takes just below 9 GiB to complete the
 build.


"Then don't do that". Seriously, I agree reducing memory consumed is 
necessary but there are always extreme cases to construct.


 Also, consider that many of the features in Phobos – apart from the
 underlying molasses consisting of std.traits and friends – are not
 really widely used internally. This is not surprising, considering that
 large parts of std.* are mostly a grab-bag of orthogonal stuff, and the
 unit tests are, as they should, fairly isolated. For example, how many
 Phobos modules actually use std.regex internally, let alone its
 compile-time variants (which we are quick to advertise)? What about
 range "pipelines" longer than the one or two stages you'd typically
 write in a unit test?


My perception is Phobos uses its own dog food intensively, which also 
leads to the criticisms that everything in Phobos depends on everything 
else. That said, clearly the library + unittests themselves are not a 
large project using the library.


 Apologies for maybe being a bit too confrontational about this,


There is agreement that a problem exists. No need to get overly carried 
over this; I recall last time you got really upset, you said something 
that was so incredibly offensive, Walter and I had to hold a pow-wow and 
decide to simply ignore it and just give you time to calm down. There's 
no need or time for all that. We're on the same boat.


 but I
 don't think anybody is helped by dismissing entire classes of problems
 just because they tend not to be so apparent from a library writer's
 perspective. If you were to, say, visit the folks at Weka, you might be
 surprised to see how easy it is to do the latter without even realizing so.


Agreed. Glad you're helping them out!


Andrei
Nov 11 2015
parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 18:02:01 UTC, Andrei 
Alexandrescu wrote:

 On 11/11/2015 12:13 PM, David Nadlinger wrote:

 When was the last time you did "dmd -unittest -main $(find std 
 -name
 '*.d')"? On current master, dmd takes just below 9 GiB to 
 complete the
 build.


 "Then don't do that". Seriously, I agree reducing memory 
 consumed is necessary but there are always extreme cases to 
 construct.


Sure. Let me point out, though, that this very thread is about a 
bug making that impossible. ;)


 There is agreement that a problem exists.


I'm glad that there is now.


 I recall last time you got really upset, you said something 
 that was so incredibly offensive, Walter and I had to hold a 
 pow-wow and decide to simply ignore it and just give you time 
 to calm down.


I've had to do the same quite a few times with either of you. 
What specific occasion are you thinking of? Maybe just call me 
out on it right there, right then next time (even privately, if 
you prefer so)?


 Agreed. Glad you're helping them out!


Well, part of the reason I wanted to make sure that this topic 
makes it near the center of our collective attention again is 
precisely that I won't be able to spend any more time (at least 
no significant amount) working with them anytime soon. They are 
doing some quite remarkable things with D, but unfortunately 
their computers are not quantum enough for me.

  — David
Nov 11 2015
prev sibling parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 11.11.2015 um 15:15 schrieb Johannes Pfau:

 I use docker hub to build docker images. One Dockerfile compiles one
 small D tool which depends on vibe.d for JSON parsing. Without
 separate compilation the build fails as docker hub kills the build
 process, out of memory...


Sorry for OT: the latest 0.7.27-alpha.1 of vibe.d is now separated into 
sub packages ("vibe-d:data" in particular contains only JSON/BSON and 
the serialization mechanics).
Nov 11 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/15 8:56 AM, Martin Nowak wrote:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau wrote:

 I think we should really fix these issues, working separate
 compilation is very important.


 No, it's not. Separate compilation is C++-ishly slow by design (b/c all
 dependencies get imported over and over again, just like headers).
 We should endorse compiling small (rather independent) packages to
 static/shared libraries.


Yes, compiling package-at-a-time should be the mode endorsed by the core 
community. Liran Zvibel and I discussed this privately following his 
DConf 2015 talk, and I recall he mentioned the package-at-a-time solved 
essentially all of their build problems.

It would be fantastic if somebody could write a wiki/dlang.org article 
about this with details on how to do it and measurements. One prime 
example is Phobos itself, for which conversion to package-at-a-time 
build (http://forum.dlang.org/thread/mkvpgn$2fg3$2 digitalmars.com) has 
led to dramatic improvements in memory consumed and build times 
(https://github.com/D-Programming-Language/phobos/pull/3379).


Andrei
Nov 11 2015
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:

 Yes, compiling package-at-a-time should be the mode endorsed by 
 the core community. Liran Zvibel and I discussed this privately 
 following his DConf 2015 talk, and I recall he mentioned the 
 package-at-a-time solved essentially all of their build 
 problems.


They must have encountered more since then since their build has 
had a lot of difficulty. I tried to hack around the problems they 
had using the various tricks I've accumulated (btw you can hit 
several of these problems without necessarily having a huge 
codebase, though reproducing them in a reduced test is extremely 
difficult) and failed. They tried throwing more hardware at it 
and that didn't work either with newer compiler versions. David 
has been working on fixing the fundamental problems and surely, 
if he succeeded completely already, he wouldn't have made these 
posts.

We, as a community, need to start believing people when they say 
they are having troubles instead of blaming the customer for 
doing it wrong.
Nov 11 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/15 10:12 AM, Adam D. Ruppe wrote:

 We, as a community, need to start believing people when they say they
 are having troubles instead of blaming the customer for doing it wrong.


Total agreement with that! -- Andrei
Nov 11 2015
prev sibling parent Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Wednesday, 11 November 2015 at 15:12:16 UTC, Adam D. Ruppe 
wrote:

 We, as a community, need to start believing people when they 
 say they are having troubles instead of blaming the customer 
 for doing it wrong.


+10_000

---
Paolo
Nov 11 2015
prev sibling next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:

 Yes, compiling package-at-a-time should be the mode endorsed by 
 the core community. Liran Zvibel and I discussed this privately 
 following his DConf 2015 talk, and I recall he mentioned the 
 package-at-a-time solved essentially all of their build 
 problems.


If you have a look at the bug report I linked in the original 
post, you'll find that this case occurs precisely with 
per-package compilation. Yes, there is a single cyclic 
inter-package dependency hidden somewhere deep inside the 
dependency graph, but getting rid of it after the fact would have 
either required re-structuring large parts of a ˜10^5 LOC code 
base, or would have again involved creating logical packages so 
large that the memory inefficiencies (CTFE is the biggest problem 
here) and run-time scaling issues (e.g. O(n^2) behavior in the 
overall number of templates) in the front-end become a problem 
again.

Of course, many of the problems could have probably been avoided 
if there was an iron-clad rule that the module dependency graph 
must remain acyclic from the beginning of development (at least 
at the level of units of compilation). But how could they have 
known how bad it would get otherwise? I don't think this is 
reflected in our documentation anywhere, at least not in a 
prominent place.

  — David
Nov 11 2015
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 17:19:31 UTC, David Nadlinger 
wrote:

 Yes, there is a single cyclic inter-package dependency hidden
 somewhere deep inside the dependency graph, but getting rid
 of it after the fact would  have either required re-structuring
 large parts of a ˜10^5 LOC code base, or would have again
 involved creating logical


Understood. FWIW it'd be more of a help to dig down and fix that 
particular bug sooner rather than going back to come up with a 
new template emission design.

Cross-Posting my Bugzilla comment here.
https://issues.dlang.org/show_bug.cgi?id=15318#c1

This seems to be a case where the second rule of this issue 14431 
fix doesn't work.
http://forum.dlang.org/post/mailman.697.1440962414.13986.digitalmars-d-bugs puremagic.com


 If a template is instantiated in non-root module, compiler 
 usually does not have to put it in object file. But if a 
 template is instantiated in both of root and non-root modules 
 which mutually import each other, it needs to placed in objfile.


===

In both of the compilations there is a root and a non-root module 
instantiating bar!5. Now unfortunately the compiler decides both 
times that the non-root module should do it.
This problem would be solved by my proposal to define a global 
order for who is responsible to instantiate a template, by 
choosing the module with the lexicographically smaller module 
name.
https://github.com/D-Programming-Language/dmd/pull/4384#discussion_r29910422
https://issues.dlang.org/show_bug.cgi?id=14431#c12

This would establish a stable order between B and C, and no 
matter how you compile them, B gets to instantiate the template.

===

An intermediate workaround for your problem is to use the 
-allinst switch, even though it slows down compilation a lot. You 
can also compile each module separately in which case both B and 
C get the instance (in general this is even slower than -allinst).

===


 packages so large that the memory inefficiencies (CTFE is the 
 biggest problem here) and run-time scaling issues (e.g. O(n^2) 
 behavior in the overall number of templates) in the front-end 
 become a problem again.


There shouldn't be anything quadratic left for template 
instantiation in the frontend.
Nov 11 2015
parent David Nadlinger <code klickverbot.at> writes:
On Thursday, 12 November 2015 at 02:13:04 UTC, Martin Nowak wrote:

 There shouldn't be anything quadratic left for template 
 instantiation in the frontend.


Maybe not for a single instantiation, but constructing the module 
member list is overall still quadratic in the number of members 
(which might be quite a lot if you heavily use templates, because 
all the instances get added first, whether they make it to 
codegen or not). See: 
https://issues.dlang.org/show_bug.cgi?id=15323.

  — David
Nov 12 2015
prev sibling next sibling parent James Hofmann <jhofmann 321f.net> writes:
On Wednesday, 11 November 2015 at 17:19:31 UTC, David Nadlinger 
wrote:

 Of course, many of the problems could have probably been 
 avoided if there was an iron-clad rule that the module 
 dependency graph must remain acyclic from the beginning of 
 development (at least at the level of units of compilation). 
 But how could they have known how bad it would get otherwise? I 
 don't think this is reflected in our documentation anywhere, at 
 least not in a prominent place.

  — David


There is some literature about whether this kind of rule, 
enforced at compile-time, can benefit software architecture in 

(Answer: probably yes - there's a relationship between cyclical 
dependencies and other accidental coupling; you can make some 
case for it both theoretically and in statistics)



Looking at what D does now, at least according to "The D 
Programming Language" (2010), it tries to allow any ordering but 
throws an exception at runtime when ambiguious cases are 
detected. So there's already some precedent to avoid cyclical 
dependency simply to avoid those errors. The coupling argument 
and the compile-times argument just add more urgency to it.


looked, but I think that position was softening towards 
"optional" due to some community pressure. I don't think anything 
makes this architecture style actually impractical and it might 
help to have the compiler warn towards it - although the same 
kind of community pressure is going to arise if it did do so. 
After all, nobody likes to be told that they are writing Bad Code 
:)
Nov 11 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 9:19 AM, David Nadlinger wrote:

 Of course, many of the problems could have probably been avoided if there was
an
 iron-clad rule that the module dependency graph must remain acyclic from the
 beginning of development (at least at the level of units of compilation).


In hindsight, I would have made D enforce that (Go does).


 But how could they have known how bad it would get otherwise?


I hadn't anticipated that code would inevitably be constructed so that every 
module imports every other module.
Nov 12 2015
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:

 On 11/11/2015 9:19 AM, David Nadlinger wrote:

 Of course, many of the problems could have probably been 
 avoided if there was an
 iron-clad rule that the module dependency graph must remain 
 acyclic from the
 beginning of development (at least at the level of units of 
 compilation).


 In hindsight, I would have made D enforce that (Go does).


That's impossible because of the way we do mixin. You can have a 
template that do mixin without creating loop because then you 
pass it down identifiers that won't resolve.

That has been a giant pain in various situations in project I 
work on.
Nov 12 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2015 2:58 PM, deadalnix wrote:

 You can have a template that
 do mixin without creating loop because then you pass it down identifiers that
 won't resolve.


My brain seg faulted trying to parse that.
Nov 12 2015
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 13 November 2015 at 00:58:57 UTC, Walter Bright wrote:

 On 11/12/2015 2:58 PM, deadalnix wrote:

 You can have a template that
 do mixin without creating loop because then you pass it down 
 identifiers that
 won't resolve.


 My brain seg faulted trying to parse that.


module a;

struct S(E) {
     import std.bitfield;
     mixin(bitfield!(
         bool, "flag", 1,
         T, "myT", SizeOf!T,
         uint, "", 32 - SizeOf!T - 1,
     ));
}

module b;

enum MyEnum { Val1, Val2, Val3 }

import a;
alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined

Now you end up having to tie stupid node into your code, like 
adding all user of library X as import in library X. Which sucks, 
but is forced by the language design.
Nov 12 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2015 5:14 PM, deadalnix wrote:

 module a;

 struct S(E) {
      import std.bitfield;
      mixin(bitfield!(
          bool, "flag", 1,
          T, "myT", SizeOf!T,
          uint, "", 32 - SizeOf!T - 1,
      ));
 }

 module b;

 enum MyEnum { Val1, Val2, Val3 }

 import a;
 alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined

 Now you end up having to tie stupid node into your code, like adding all user
of
 library X as import in library X. Which sucks, but is forced by the language
 design.


The trouble there is that E is turned into MyEnum in the mixin string, and 
MyEnum would be unknown to module a.

You can file an enhancement request on this in bugzilla if you like.
Nov 12 2015
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 13 November 2015 at 04:03:23 UTC, Walter Bright wrote:

 On 11/12/2015 5:14 PM, deadalnix wrote:

 module a;

 struct S(E) {
      import std.bitfield;
      mixin(bitfield!(
          bool, "flag", 1,
          T, "myT", SizeOf!T,
          uint, "", 32 - SizeOf!T - 1,
      ));
 }

 module b;

 enum MyEnum { Val1, Val2, Val3 }

 import a;
 alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined

 Now you end up having to tie stupid node into your code, like 
 adding all user of
 library X as import in library X. Which sucks, but is forced 
 by the language
 design.


 The trouble there is that E is turned into MyEnum in the mixin 
 string, and MyEnum would be unknown to module a.

 You can file an enhancement request on this in bugzilla if you 
 like.


I know what the problem is. Hence you need to import a in b and b 
in a. There is not really anyway around this because this is 
intrinsically how mixin works.
Nov 12 2015
parent Dicebot <public dicebot.lv> writes:
 I know what the problem is. Hence you need to import a in b and 
 b in a. There is not really anyway around this because this is 
 intrinsically how mixin works.


It could be all much easier if resolving aliases was controllable 
in generic code, most importantly in regards of .stringof
Nov 13 2015
prev sibling next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:

 On 11/11/2015 9:19 AM, David Nadlinger wrote:

 Of course, many of the problems could have probably been 
 avoided if there was an
 iron-clad rule that the module dependency graph must remain 
 acyclic from the
 beginning of development (at least at the level of units of 
 compilation).


 In hindsight, I would have made D enforce that (Go does).


Would there be any gain from adding an optional compiler warning 
for it?
Nov 12 2015
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Thursday, 12 November 2015 at 23:06:01 UTC, rsw0x wrote:

 On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
 wrote:

 On 11/11/2015 9:19 AM, David Nadlinger wrote:

 Of course, many of the problems could have probably been 
 avoided if there was an
 iron-clad rule that the module dependency graph must remain 
 acyclic from the
 beginning of development (at least at the level of units of 
 compilation).


 In hindsight, I would have made D enforce that (Go does).


 Would there be any gain from adding an optional compiler 
 warning for it?


It can be implemented as an external tool with the -deps compiler 
switch.
Nov 12 2015
parent reply Stefan <dl ng.rocks> writes:
On Thursday, 12 November 2015 at 23:08:57 UTC, Jakob Ovrum wrote:


 [...dependency check...]





 It can be implemented as an external tool with the -deps 
 compiler switch.


there is depend [1].

It warns about cycles and unintended dependencies if you specify 
the target dependencies.
It helps us to maintain our commercial code base. src as well as 
unittest.


[1]
https://github.com/funkwerk/depend
Nov 13 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/13/2015 07:51 AM, Stefan wrote:

 On Thursday, 12 November 2015 at 23:08:57 UTC, Jakob Ovrum wrote:


 [...dependency check...]





 It can be implemented as an external tool with the -deps compiler switch.


 there is depend [1].

 It warns about cycles and unintended dependencies if you specify the
 target dependencies.
 It helps us to maintain our commercial code base. src as well as unittest.


 [1]
 https://github.com/funkwerk/depend


Hi Stefan, it was great to meet in Stuttgart!

This is a simple but very useful tool, it ought to be made part of the 
standard distribution. I just created 
https://issues.dlang.org/show_bug.cgi?id=15337, in summary:

1. Relicensing the work under Boost

2. Make the tool easier to use, e.g. have it run dmd automatically if 
needed etc.

3. Integrate the tool within the tools/ repository and make it part of 
the standard dmd distribution.

Good tooling has been a huge asset for Go - we could take a page from 
their book.

Stefan, Dragos - do you guys think you could take this to completion?


Thanks,

Andrei
Nov 14 2015
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:

 I hadn't anticipated that code would inevitably be constructed 
 so that every module imports every other module.


One day, I was cleaning my house and found all kinds of the cat's 
stuff under the couch. I wondered: why does the cat love shoving 
her stuff down there?

Then I realized the answer: she randomly bats it around the whole 
floor. If it lands out in the open, she knocks it somewhere else. 
When it randomly ends up under the couch though, she can no 
longer reach it, so that is where it stays. Eventually, 
everything will accumulate there.


I guess this is just a long winded way of saying that if 
something is even a little useful and possible, it will end up 
being used until it reaches some point where we can't change it 
anymore...
Nov 12 2015
prev sibling parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:

 Liran Zvibel […] mentioned the package-at-a-time solved 
 essentially all of their build problems.


They were stuck at 2.066 until very recently, though, because of 
issues very similar to the one discussed here. Right now, they 
are using LDC 2.068.2 with a couple of local workarounds for e.g. 
the issue from the original post that are not really general 
enough to go into upstream. The idea would be to return to an 
unpatched frontend as soon as possible, though.

  — David
Nov 11 2015
prev sibling next sibling parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:

 No, it's not. Separate compilation is C++-ishly slow by design 
 (b/c all dependencies get imported over and over again, just 
 like headers).


To a certain extent this is not true. As long as the amount 
repeated work is not too big, the ability to parallelize the 
actual semantic3/glue/codegen parts in a separate compilation 
setting still leads to a considerable speedup. All-at-once 
compilation forces you to throw away 31 of the 32 cores in your 
nice heavy duty development box.

(Yes, in theory the different parts of the compilation process 
could be parallelized even for an all-at-once build, but by the 
time we reach the level of separation/well-definedness of the 
internal state required for this, all the by-package compilation 
problems would have probably disappeared rather automatically.)

  — David
Nov 11 2015
prev sibling next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:

 I think we should really fix these issues, working separate 
 compilation is very important.


 No, it's not. Separate compilation is C++-ishly slow by design 
 (b/c all dependencies get imported over and over again, just 
 like headers).
 We should endorse compiling small (rather independent) packages 
 to static/shared libraries.


This would be great if cross-module inlining actually worked. 
pragma(inline,true) doesn't even work across modules afaik
Nov 11 2015
parent deadalnix <deadalnix gmail.com> writes:
On Wednesday, 11 November 2015 at 18:06:22 UTC, rsw0x wrote:

 On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
 wrote:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:

 I think we should really fix these issues, working separate 
 compilation is very important.


 No, it's not. Separate compilation is C++-ishly slow by design 
 (b/c all dependencies get imported over and over again, just 
 like headers).
 We should endorse compiling small (rather independent) 
 packages to static/shared libraries.


 This would be great if cross-module inlining actually worked. 
 pragma(inline,true) doesn't even work across modules afaik


It does if you compile by package with something else than DMD, 
because DMD does inlining in some very weird ways.
Nov 11 2015
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:

 I think we should really fix these issues, working separate 
 compilation is very important.


 No, it's not. Separate compilation is C++-ishly slow by design 
 (b/c all dependencies get imported over and over again, just 
 like headers).
 We should endorse compiling small (rather independent) packages 
 to static/shared libraries.


But compiling by package isn't slow by design. Also, it depends 
on the modules you're compiling. If, like me, you have all your 
unit tests in separate files from the implementation, it's faster 
when only changing the test to recompile that one file and relink 
than compiling the whole program again.

So... it depends.

Atila
Nov 13 2015
prev sibling next sibling parent reply Robert burner Schadek <rburners gmail.com> writes:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:

 A quick workaround could be enabling the GC for DDMD. IIRC I 
 read somewhere on github that the segfaulting code was actually 
 rewritten now and the GC might just work.


I just run make -f posix.mak unittest -j10 on phobos after 
removing GC.disable() from ddmd. It worked just fine. Everything 
build everything passed the tests.

(the time program told 1:41.25 for nogc and 1:49.80 with gc. This 
is with running all the tests.)

Shouldn't we at least add a command line switch to dmd to 
activate the GC.
Nov 11 2015
next sibling parent Robert burner Schadek <rburners gmail.com> writes:
druntime also works with GC activated
Nov 11 2015
prev sibling next sibling parent rsw0x <anonymous anonymous.com> writes:
On Wednesday, 11 November 2015 at 23:44:04 UTC, Robert burner 
Schadek wrote:

 On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
 wrote:

 A quick workaround could be enabling the GC for DDMD. IIRC I 
 read somewhere on github that the segfaulting code was 
 actually rewritten now and the GC might just work.


 I just run make -f posix.mak unittest -j10 on phobos after 
 removing GC.disable() from ddmd. It worked just fine. 
 Everything build everything passed the tests.

 (the time program told 1:41.25 for nogc and 1:49.80 with gc. 
 This is with running all the tests.)

 Shouldn't we at least add a command line switch to dmd to 
 activate the GC.


if it works, shouldn't the runtime 
CLI(http://dlang.org/changelog/2.067.0.html#gc-options) just be 
used to manually deactivate it if needed?
Nov 11 2015
prev sibling parent reply Daniel Murphy <yebbliesnospam gmail.com> writes:
On 12/11/2015 10:44 AM, Robert burner Schadek wrote:

 I just run make -f posix.mak unittest -j10 on phobos after removing
 GC.disable() from ddmd. It worked just fine. Everything build everything
 passed the tests.

 (the time program told 1:41.25 for nogc and 1:49.80 with gc. This is
 with running all the tests.)

 Shouldn't we at least add a command line switch to dmd to activate the GC.


You also need to modify root/rmem.d to actually use the GC as the allocator.
Nov 12 2015
parent reply Robert burner Schadek <rburners gmail.com> writes:
On Friday, 13 November 2015 at 02:50:07 UTC, Daniel Murphy wrote:

 You also need to modify root/rmem.d to actually use the GC as 
 the allocator.


I should have known that it couldn't be that simple. Anyway, 
after doing so. Building druntime and phobos die with a segfault, 
but all dmd tests pass, except runnable/arrayop.d

Furthermore, I had to create a function called:

     extern (C) void* allocmemory(size_t m_size) {
         return GC.malloc(m_size);
     }
Nov 13 2015
parent Daniel Murphy <yebbliesnospam gmail.com> writes:
On 13/11/2015 8:26 PM, Robert burner Schadek wrote:

 On Friday, 13 November 2015 at 02:50:07 UTC, Daniel Murphy wrote:

 You also need to modify root/rmem.d to actually use the GC as the
 allocator.


 I should have known that it couldn't be that simple. Anyway, after doing
 so. Building druntime and phobos die with a segfault, but all dmd tests
 pass, except runnable/arrayop.d

 Furthermore, I had to create a function called:

      extern (C) void* allocmemory(size_t m_size) {
          return GC.malloc(m_size);
      }


Yep, that sounds about right.  It's probably just some malloced memory 
being used to store GC pointers, somewhere in there.
Nov 13 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 5:47 AM, Johannes Pfau wrote:

 Ellery Newcomer recently reported a template emission bug where
 templates are emitted twice:
 http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com


Please file bug reports in bugzilla. Filing them in the n.g. means they will 
most likely get overlooked.
Nov 12 2015
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 12 Nov 2015 15:15:24 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:


 On 11/11/2015 5:47 AM, Johannes Pfau wrote:

 Ellery Newcomer recently reported a template emission bug where
 templates are emitted twice:
 http://forum.dlang.org/thread/n1omke$1bh5$1 digitalmars.com  


 
 Please file bug reports in bugzilla. Filing them in the n.g. means
 they will most likely get overlooked.
 


Ellery filed a bug report and I've posted a reduced test case:

https://issues.dlang.org/show_bug.cgi?id=15324
Nov 13 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/13/2015 9:23 AM, Johannes Pfau wrote:

 Ellery filed a bug report and I've posted a reduced test case:

 https://issues.dlang.org/show_bug.cgi?id=15324


Thank you!
Nov 13 2015
parent deadalnix <deadalnix gmail.com> writes:
On Saturday, 14 November 2015 at 03:43:38 UTC, Walter Bright 
wrote:

 On 11/13/2015 9:23 AM, Johannes Pfau wrote:

 Ellery filed a bug report and I've posted a reduced test case:

 https://issues.dlang.org/show_bug.cgi?id=15324


 Thank you!


https://issues.dlang.org/show_bug.cgi?id=15333

Also, thanks to whoever was able to reduce it more than I.
Nov 16 2015
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

We're well aware of the template problems and FWIW the most 
important step is to simplify the current implementation (not the 
rules).
http://forum.dlang.org/post/55F07E92.7040104 dawg.eu
Once that is done and we have a clearer understanding of what the 
problems are, we could rediscuss the emission rules.

Ultimatively the decision to instantiate/emit as few templates as 
possible sped up compilation a lot but basically means -c with 
multiple objects will never work reliably.
Compiling separate objects is a C++-idiom anyhow and doesn't make 
too much sense for D. Instead separating your codebase into 
packages with little interdependencies and always recompiling 
those packages when something changes should be the preferred way.
Nov 11 2015
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

Let's please make sure that we come out of this discussion with 
some concrete actions to work on. Ideally you'd fill our backlog 
with some stories, even if they are just research stories.
Nov 11 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 08:08 AM, David Nadlinger wrote:
[snip]

Regarding the top-level issue. Walter and I agree it's an important 
problem, and also that plugging holes as they start leaking (which is 
what we've been doing so far) is not going to work well in the long haul.

A redesign of template instantiation is necessary, and Walter needs to 
be fully involved in it. However, please give it time. Walter is 
currently working full time on catching C++ exceptions from D code, and 
as we all know the best way of getting many things done is to do one 
thing at a given time and do it fully. It should take him at least two 
weeks' time to get there.

This is a good time to collect cases known to be troublesome and to 
discuss high-level approaches.


Thanks,

Andrei
Nov 11 2015
next sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 11/11/2015 08:08 AM, David Nadlinger wrote:
 [snip]

 Regarding the top-level issue. Walter and I agree it's an important
 problem, and also that plugging holes as they start leaking (which is what
 we've been doing so far) is not going to work well in the long haul.

 A redesign of template instantiation is necessary, and Walter needs to be
 fully involved in it. However, please give it time. Walter is currently
 working full time on catching C++ exceptions from D code, and as we all
 know the best way of getting many things done is to do one thing at a given
 time and do it fully. It should take him at least two weeks' time to get
 there.


Please no.  I've done my fair share of investigating this in libunwind, and
it just isn't worth it.
Nov 11 2015
next sibling parent David Nadlinger <code klickverbot.at> writes:
On Wednesday, 11 November 2015 at 17:19:51 UTC, Iain Buclaw wrote:

 Please no.  I've done my fair share of investigating this in 
 libunwind, and it just isn't worth it.


It isn't worth it, or you couldn't figure out how to? ;P

In other words, what I'm asking is: Yes, all the lifetime 
management depends crucially on the C++ standard library 
internals. But given that we do this anyway (name mangling, 
etc.), how would you decide that the effort is not worth it?

  — David
Nov 11 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/11/2015 12:19 PM, Iain Buclaw via Digitalmars-d wrote:

 On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On 11/11/2015 08:08 AM, David Nadlinger wrote:
     [snip]

     Regarding the top-level issue. Walter and I agree it's an important
     problem, and also that plugging holes as they start leaking (which
     is what we've been doing so far) is not going to work well in the
     long haul.

     A redesign of template instantiation is necessary, and Walter needs
     to be fully involved in it. However, please give it time. Walter is
     currently working full time on catching C++ exceptions from D code,
     and as we all know the best way of getting many things done is to do
     one thing at a given time and do it fully. It should take him at
     least two weeks' time to get there.


 Please no.  I've done my fair share of investigating this in libunwind,
 and it just isn't worth it.


Could you please provide Walter more detail? What's not worth it - 
catching C++ exceptions from D code? -- Andrei
Nov 11 2015
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 November 2015 at 19:02, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 11/11/2015 12:19 PM, Iain Buclaw via Digitalmars-d wrote:


 On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On 11/11/2015 08:08 AM, David Nadlinger wrote:
     [snip]

     Regarding the top-level issue. Walter and I agree it's an important
     problem, and also that plugging holes as they start leaking (which
     is what we've been doing so far) is not going to work well in the
     long haul.

     A redesign of template instantiation is necessary, and Walter needs
     to be fully involved in it. However, please give it time. Walter is
     currently working full time on catching C++ exceptions from D code,
     and as we all know the best way of getting many things done is to do
     one thing at a given time and do it fully. It should take him at
     least two weeks' time to get there.


 Please no.  I've done my fair share of investigating this in libunwind,
 and it just isn't worth it.


 Could you please provide Walter more detail? What's not worth it -
 catching C++ exceptions from D code? -- Andrei


The main problem for seamless support is having some way to generate the
C++ typeinfo in D to allow work across boundaries.  However there is
nothing stopping catch-all style exceptions from working.

Consider the following desired example (this works today with minimal
library changes in gdc's libunwind library):
*****************************
#include <iostream>

int compute(int a, int b)
{
  if (b == 0)
    throw "Division by 0 in C++!";

  return a / b;
}

void cpp_main()
{
  try {
     std::cout << compute(1, 0);
  } catch (...) {
    std::cout << "Unknown exception caught in C++, rethrowing..." << "\n";
    throw;
  }
}
*****************************

Then put in bindings and main in D.
*****************************
import std.stdio;

extern(C++) void cpp_main();

void main()
{
    try {
        cpp_main();
    }
    catch {
        writeln("Unknown exception caught in D, exiting...");
    }
}
*****************************

In our unwind library, we can put in a copy of the C++ unwind
implementation:
*****************************
static if (GNU_ARM_EABI_Unwinder)
{
  const _Unwind_Exception_Class __gxx_exception_class
  = ['G', 'N', 'U', 'C', 'C', '+', '+', '\0'];
}
else
{
  const _Unwind_Exception_Class __gxx_exception_class =
0x474e5543432b2b00UL;
}

// Structure of a C++ exception, represented as a C structure...
// See unwind-cxx.h for the full definition.
struct __cxa_exception
{
  void *exceptionType;
  void function(void *) exceptionDestructor;
  void function() unexpectedHandler;
  void function() terminateHandler;
  __cxa_exception *nextException;
  int handlerCount;

  static if (GNU_ARM_EABI_Unwinder)
  {
    __cxa_exception* nextPropagatingException;
    int propagationCount;
  }
  else
  {
    int handlerSwitchValue;
    const ubyte *actionRecord;
    const ubyte *languageSpecificData;
    _Unwind_Ptr catchTemp;
    void *adjustedPtr;
  }

  _Unwind_Exception unwindHeader;
}
*****************************

Ignoring the main guts of gdc's unwind implementation (should be very
similar in ldc and sdc by coincidence), let's instead focus on the exact
part where we try to match the exception object, you'd have something like
the following:
*****************************
if (ar_filter > 0)
  {
    // Positive filter values are handlers.
    void *catch_type = get_ttype_entry (&info, ar_filter);

    if (!foreign_exception)
      {
        if (_d_isbaseof (xh.object.classinfo, cast(ClassInfo)catch_type))
          saw_handler = true;
      }
    else
      {
        // == Case A ==
        // Null catch type is a catch-all handler; we can catch foreign
exceptions with this.
        if (catch_type is null)
          saw_handler = true;

        else if (ue_header.exception_class == __gxx_exception_class)
          {
            // == Case B ==
            // Typeinfo are directly compared, which might not be correct
if they aren't merged.
            void *except_typeinfo = ((cast(__cxa_exception *)(ue_header +
1)) - 1).exceptionType;

            if (catch_type == except_typeinfo)
              saw_handler = true;

            // == Case C ==
            // Typeid are compared between catch type and some specially
crafted C++ Exception class.
            if (cast(ClassInfo)catch_type is typeid(UnknownCPPException))
              saw_handler = true;
          }
      }
  }
*****************************

Now let's discuss the three cases:

- Case A:
A `null` catch type never happens in D code, because `catch { }` is always
re-written to `catch(Throwable) { }` (this might have changed since 2.066,
but I doubt it).  So any kind of `catch(...)` equivalent in D would make
this is a valid candidate for catching not just C++ exceptions, but also
exceptions in any language that use libunwind for EH (Ada, Go, Java,
etc...).  Bonus, we don't need to know anything about C++ exception objects
or typeinfo to achieve this.

In the other cases, checking the exception class is simple enough,
libunwind provides a field for that, and just requires to declare it
somewhere on the D bindings side.

- Case B:
Explicitly try to match a C++ exception object.  Copying the
__cxa_exception struct from unwind-cxx.h, takes advantage of ABI
compatibility - though we must take care that they are kept in sync (not
that it ever changes in an incompatible way).  Also from the C++ unwind
library, is one small trick to get the C++ typeinfo pointer.  Where this
falls short is that ``catch_type == except_typeinfo`` will always be false
unless we expose some way to generate C++ typeinfo in D.  *Maybe* this
could be achieved by generating an pointer to an `extern(C++)` symbol to
the typeinfo - just need to get mangling right.  But this is the ugliest
thing we can possibly do.

- Case C:
An alternative to the catch all handler in case A, but instead test whether
the catch we are inspecting matches some explicit object (derived from
`Throwable` to be compatible with the existing compiler constraints).
Again, this can be extended for other languages (UnknownAdaException,
UnknownGoException, UnknownJavaException, etc...) - however, unlike case A,
we need to copy the exception class signature of other languages into the D
bindings.

With the possible exception being in case B, there is no way to return the
foreign language object from EH to the catch handler.  Using case C and
storing the caught exception object to a variable `catch
(UnknownCPPException e)` will always result with the object being equal to
`null`.

*****************************

Coming back round full circle to the original example, I've implemented
case C but instead using `Throwable` to make `catch { }` succeed in
catching C++ exceptions.

Compilation and Result:
$ gdc dcode.d cxxcode.cc
$ ./a.out
Unknown exception caught in C++, rethrowing...
Unknown exception caught in D, exiting...

Even throwing in D, and catching in C++ works thanks to C++'s existing
support for foreign language exceptions and `catch(...)`.  So moving
`compute` into D, we have the following.
*****************************
extern(C++) int compute(int a, int b)
{
  if (b == 0) {
    writeln("Exception thrown in D...");
    throw new Exception("Division by 0 in D!");
  }

  return a / b;
}
*****************************

Compilation and Result:
$ gdc dcode.d cxxcode.cc
$ ./a.out
Exception thrown in D...
Unknown exception caught in C++, rethrowing...
Unknown exception caught in D, exiting...


Andrei - Hope this answers your question.

--
Regards
Iain
Nov 11 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:

 The main problem for seamless support is having some way to generate the C++
 typeinfo in D to allow work across boundaries.


I suspect that is unnecessary.

The whole point of D's C++ interface is to interface with C++ code. Defining
C++ 
classes in D code that does not link to any C++ code that also defines those 
classes won't be supported. I.e. we can reply on the C++ code generating the 
necessary C++ typeinfo. The generated D code only has to link to it, i.e. 
generate an extern reference to the name of the typeinfo.

Similarly, we never need to throw a C++ exception from D code. All we need to
do 
is call a library function written in C++ that throws it for us.

What D does need to do is support catching an std::exception*, or a class 
derived from std::exception. I believe we are on solid ground in not supporting 
any other C++ types being thrown.

Nor do I believe we need to catch C++ objects with catch(...). Catching C++ 
objects will need an explicit catch(std.exception*).

Over time, perhaps it may become apparent that we do need to support more kinds 
of C++ types being thrown. But it is not necessary to get this lead brick 
airborne, and we shouldn't waste time worrying about it.
Nov 11 2015
next sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 Nov 2015 7:15 am, "Walter Bright via Digitalmars-d" <
digitalmars-d puremagic.com> wrote:

 On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:

 The main problem for seamless support is having some way to generate the




C++

 typeinfo in D to allow work across boundaries.



 I suspect that is unnecessary.

 The whole point of D's C++ interface is to interface with C++ code.


Defining C++ classes in D code that does not link to any C++ code that also
defines those classes won't be supported. I.e. we can reply on the C++ code
generating the necessary C++ typeinfo. The generated D code only has to
link to it, i.e. generate an extern reference to the name of the typeinfo.

 Similarly, we never need to throw a C++ exception from D code. All we


need to do is call a library function written in C++ that throws it for us.


Yes, never throw a C++ exception from D, even if we manage to intercept the
object.


 What D does need to do is support catching an std::exception*, or a class


derived from std::exception. I believe we are on solid ground in not
supporting any other C++ types being thrown.

 Nor do I believe we need to catch C++ objects with catch(...). Catching


C++ objects will need an explicit catch(std.exception*).


Well, as demonstrated, that is the one thing that can be supported now for
free.  Either by allowing C++-style catch-all or using language specific
catch-all class matching.


 Over time, perhaps it may become apparent that we do need to support more


kinds of C++ types being thrown. But it is not necessary to get this lead
brick airborne, and we shouldn't waste time worrying about it.


Any type could be caught, again I demonstrated this, but I would have more
confidence if g++ generated the typeinfo bindings for us using
pragma(mangle) to some D-friendly symbol.  Rather than us pushing in more
frail mangling support from our end.

--
Regards,
Iain



 Nov 11 2015



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/11/2015 11:45 PM, Iain Buclaw via Digitalmars-d wrote:

  > Nor do I believe we need to catch C++ objects with catch(...). Catching C++
 objects will need an explicit catch(std.exception*).
  >

 Well, as demonstrated, that is the one thing that can be supported now for
 free.  Either by allowing C++-style catch-all or using language specific
 catch-all class matching.


I don't agree with implementing things just because we can. There has to be a 
good case for it. I cannot see a case for catching ints, strings, etc. My 
understanding is that this is more or less acknowledged by the C++ community, 
and std::exceptions are used.



 Any type could be caught, again I demonstrated this, but I would have more
 confidence if g++ generated the typeinfo bindings for us using pragma(mangle)
to
 some D-friendly symbol.  Rather than us pushing in more frail mangling support
 from our end.


Having dealt with mangling issues for decades, I'm not overly concerned about
it.



 Nov 12 2015



prev sibling next sibling parent  Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 12 Nov 2015 7:15 am, "Walter Bright via Digitalmars-d" <
digitalmars-d puremagic.com> wrote:

 On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:

 The main problem for seamless support is having some way to generate the




C++

 typeinfo in D to allow work across boundaries.



 I suspect that is unnecessary.

 The whole point of D's C++ interface is to interface with C++ code.


Defining C++ classes in D code that does not link to any C++ code that also
defines those classes won't be supported. I.e. we can reply on the C++ code
generating the necessary C++ typeinfo. The generated D code only has to
link to it, i.e. generate an extern reference to the name of the typeinfo.


It is a small concern to me that C++ catch handlers work by calling some
support library code, If I recall correctly both at the start and end of
the handler.  These calls may change the state of C++'s language specific
unwind library, such as adjust any reference counters, mark the thrown
object as having been caught.

Maybe less of a concern now than last year.  But I do wonder what effect it
will incur on a long running program.

--
Regards,
Iain



 Nov 11 2015



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/2015 01:15 AM, Walter Bright wrote:

 What D does need to do is support catching an std::exception*, or a
 class derived from std::exception. I believe we are on solid ground in
 not supporting any other C++ types being thrown.

 Nor do I believe we need to catch C++ objects with catch(...). Catching
 C++ objects will need an explicit catch(std.exception*).


I notice you use a star - did you back off from the idea of defining 
C++'s std::exception as an extern(C++) class (as opposed to an 
extern(C++) struct)? -- Andrei



 Nov 12 2015



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/12/2015 5:38 AM, Andrei Alexandrescu wrote:

 I notice you use a star - did you back off from the idea of defining C++'s
 std::exception as an extern(C++) class (as opposed to an extern(C++) struct)?
--



I just wanted to emphasize receiving it by reference rather than by value.



 Nov 12 2015



prev sibling  parent reply deadalnix <deadalnix gmail.com>  writes:


On Wednesday, 11 November 2015 at 17:19:51 UTC, Iain Buclaw wrote:

 Please no.  I've done my fair share of investigating this in 
 libunwind, and it just isn't worth it.


Ok I know Walter won't listen, and I wouldn't be as radical as 
Iain, but I concur. There are thing that can be done to improve 
C++ and D interrop with Exception. For instance, SDC makes sure 
that C++ exception unwind properly through D and vice versa, but 
don't allow to catch across language.

There is a reason for that. To be able to catch C++ exceptions, 
you need to duplicate the whole RTTI mechanism from C++ + runtime 
support for it + hook oneself in the C++ runtime to signal 
catching and rethrowing.

This is definitively not impossible, but the cost benefit ratio 
is such as I wouldn't prioritize this. There are definitively 
more way to have impact without summoning a Golem when it comes 
to C++ interrop.



 Nov 11 2015



 next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/11/2015 07:45 PM, deadalnix wrote:

 This is definitively not impossible, but the cost benefit ratio is such
 as I wouldn't prioritize this.


The way we see it is the benefit is all C++ interop. You can't really do 
any meaningful interop whatsoever if you can't catch std::exception. -- 
Andrei



 Nov 11 2015



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/11/2015 4:45 PM, deadalnix wrote:

 Ok I know Walter won't listen,


Does anyone else hear a buzzing sound? Or is my hearing aid on the fritz again?



 Nov 11 2015



prev sibling next sibling parent reply David Nadlinger <code klickverbot.at>  writes:


On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei 
Alexandrescu wrote:

 Walter is currently working full time on catching C++ 
 exceptions from D code […]


It would have sure been interesting to know about this, but I 
couldn't find anything to that effect on the mailing lists or on 
Trello – where Martin would certainly wish it could be found.

There are quite a few people around here who very familiar with 
what is required to make that happen (DWARF/libunwind based EH, 
etc…), and although I'm certainly not going to tell Walter how to 
do his job, it seems rather pointless for him to go through the 
process of rediscovering all this again on his own.

  — David



 Nov 11 2015



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/11/2015 12:25 PM, David Nadlinger wrote:

 On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei Alexandrescu wrote:

 Walter is currently working full time on catching C++ exceptions from
 D code […]


 It would have sure been interesting to know about this, but I couldn't
 find anything to that effect on the mailing lists or on Trello – where
 Martin would certainly wish it could be found.

 There are quite a few people around here who very familiar with what is
 required to make that happen (DWARF/libunwind based EH, etc…), and
 although I'm certainly not going to tell Walter how to do his job, it
 seems rather pointless for him to go through the process of
 rediscovering all this again on his own.


That's great. We have a history of initiatives that nobody followed and 
endless debates without work getting done; also this particular issue 
has been up in the air and spoken about for a good while. So we figured 
if Walter doesn't sit down and do it, nobody will.

I'll leave it up to him to reach out to you and/or this forum in this 
matter.


Andrei



 Nov 11 2015



  parent reply David Nadlinger <code klickverbot.at>  writes:


On Wednesday, 11 November 2015 at 18:05:46 UTC, Andrei 
Alexandrescu wrote:

 That's great. We have a history of initiatives that nobody 
 followed and endless debates without work getting done; also 
 this particular issue has been up in the air and spoken about 
 for a good while. So we figured if Walter doesn't sit down and 
 do it, nobody will.


I couldn't agree more.

What I tried to point out is merely that between Amaury, Kai, 
Martin, Iain, Dan, Joakim and me there is quite a bit of working 
knowledge regarding the internals of the various forms of the 
libunwind-based C++ exception handling mechanism, and I'm sure 
most of us would be happy to help out with any specific questions 
and design decisions (Walter of course being the authority on his 
own EH mechanism, which DMD will however have to transition away 
from on Win64 and Linux/… to transparently interface with C++).

  — David



 Nov 11 2015



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/11/2015 12:07 PM, David Nadlinger wrote:

 On Wednesday, 11 November 2015 at 18:05:46 UTC, Andrei Alexandrescu wrote:

 That's great. We have a history of initiatives that nobody followed and
 endless debates without work getting done; also this particular issue has been
 up in the air and spoken about for a good while. So we figured if Walter
 doesn't sit down and do it, nobody will.


 I couldn't agree more.

 What I tried to point out is merely that between Amaury, Kai, Martin, Iain,
Dan,
 Joakim and me there is quite a bit of working knowledge regarding the internals
 of the various forms of the libunwind-based C++ exception handling mechanism,
 and I'm sure most of us would be happy to help out with any specific questions
 and design decisions (Walter of course being the authority on his own EH
 mechanism, which DMD will however have to transition away from on Win64 and
 Linux/… to transparently interface with C++).

   — David


I appreciate the offer. I'll start a new thread on it.



 Nov 11 2015



prev sibling  parent reply Martin Nowak <code dawg.eu>  writes:


On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei 
Alexandrescu wrote:

 A redesign of template instantiation is necessary, and Walter 
 needs to be fully involved in it. However, please give it time.


I'd still opt to first refactor the existing code into something 
that's understandable before starting to redesign the whole 
topic. Maybe the design is already fine but buried by complex to 
understand control flow. Also just redesigning this on a blank 
sheet will easily miss important edge cases. And last but not 
least if we can't come up with a better design we'd still have a 
better codebase.



 Nov 11 2015



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/11/2015 06:06 PM, Martin Nowak wrote:

 On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei Alexandrescu wrote:

 A redesign of template instantiation is necessary, and Walter needs to
 be fully involved in it. However, please give it time.


 I'd still opt to first refactor the existing code into something that's
 understandable before starting to redesign the whole topic. Maybe the
 design is already fine but buried by complex to understand control flow.
 Also just redesigning this on a blank sheet will easily miss important
 edge cases. And last but not least if we can't come up with a better
 design we'd still have a better codebase.


We have to come up with a better design. Not coming with a better design 
is not an option. Also there is ample evidence that the months-long 
process of patching things as problems are found does not work. -- Andrei



 Nov 11 2015



prev sibling next sibling parent  Atila Neves <atila.neves gmail.com>  writes:


On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

 Hi all,

 Kenji and Walter have been working on improving the template 
 emission strategy during the last couple of releases, i.e. 
 whether a template instance is emitted to a given object file 
 or not. Nevertheless, I've been continually hearing complaints 
 from various people with large D code bases (our commercial 
 users and some of the more complex open source projects) that 
 they have problems compiling their code doing anything else 
 than an all-at-once build.

 [...]


I've had problems compiling part of my code as a static library 
and linking to a file that used a template mixin that in turn did 
compile-time reflection. The mangling was different between the 
two and I had linker errors. It was weird and frustrating, and I 
had to go back to compiling all at once, which is slower.

Atila



 Nov 13 2015



prev sibling  parent  Tofu Ninja <emmons0 purdue.edu>  writes:


On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

 ...


Sorry I haven't read this whole thread so maybe this has been 
mentioned before or not. I am not even sure if its really related 
to this topic, but it seemed related to me so I figured I would 
just post it here rather than make a new thread.

ModuleInfo.localClasses doesn't list template class instances. I 
just ran into this(like 5 min ago) and thought it was odd. I 
expected it too list template instances as well so I thought it 
might be a bug, not really sure. Thankfully I don't really need 
it to list template instances of classes so it not affecting my 
project, but I could see it being a problem for others.



 Nov 16 2015