www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Make DMD emit C++ .h files same as .di files

reply Manu <turkeyman gmail.com> writes:
I've been talking about this for years, and logged this a while back:
https://issues.dlang.org/show_bug.cgi?id=19579

Is there anyone interested in or knows how to do this?
It would really be super valuable; this idea would motivate
inter-language projects that typically go C++-fist-with-a-D-binding to
work the other way.

Creating a pressure to write D-code first because the binding part is
maintained automatically when you compile is potentially significant;
gets programmers thinking and writing D code as first-class.

Idea would be same as emitting .di files, but in this case only
extern(C)/extern(C++) declarations would be emit to a .h file.

I could use this to create some nice demos in my office which I think
would get a few people talking and excited.
Feb 24
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/19 6:24 PM, Manu wrote:
 I've been talking about this for years, and logged this a while back:
 https://issues.dlang.org/show_bug.cgi?id=19579
 
 Is there anyone interested in or knows how to do this?
 It would really be super valuable; this idea would motivate
 inter-language projects that typically go C++-fist-with-a-D-binding to
 work the other way.
 
 Creating a pressure to write D-code first because the binding part is
 maintained automatically when you compile is potentially significant;
 gets programmers thinking and writing D code as first-class.
 
 Idea would be same as emitting .di files, but in this case only
 extern(C)/extern(C++) declarations would be emit to a .h file.
 
 I could use this to create some nice demos in my office which I think
 would get a few people talking and excited.
Yes that would be fantastic, and something I'd advocated for a long time as well. It would be in fact a prime application of the relatively recent ability to use dmd as a library. No need to modify the compiler, just walk the AST and output the header. Should be a couple hundred lines for good effect. Andrei
Feb 24
parent reply Manu <turkeyman gmail.com> writes:
On Sun, Feb 24, 2019 at 4:05 PM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 2/24/19 6:24 PM, Manu wrote:
 I've been talking about this for years, and logged this a while back:
 https://issues.dlang.org/show_bug.cgi?id=19579

 Is there anyone interested in or knows how to do this?
 It would really be super valuable; this idea would motivate
 inter-language projects that typically go C++-fist-with-a-D-binding to
 work the other way.

 Creating a pressure to write D-code first because the binding part is
 maintained automatically when you compile is potentially significant;
 gets programmers thinking and writing D code as first-class.

 Idea would be same as emitting .di files, but in this case only
 extern(C)/extern(C++) declarations would be emit to a .h file.

 I could use this to create some nice demos in my office which I think
 would get a few people talking and excited.
Yes that would be fantastic, and something I'd advocated for a long time as well. It would be in fact a prime application of the relatively recent ability to use dmd as a library. No need to modify the compiler, just walk the AST and output the header. Should be a couple hundred lines for good effect.
Why wouldn't you do it in the same pass as the .di output?
Feb 24
parent reply Jacob Carlborg <doob me.com> writes:
On 2019-02-25 02:04, Manu wrote:

 Why wouldn't you do it in the same pass as the .di output?
* Separation of concerns * Simplifying the compiler ("simplifying" is not the correct description, rather avoid making the compiler more complex) I think the .di generation should be a separate tool as well. -- /Jacob Carlborg
Feb 25
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 25 February 2019 at 10:20:34 UTC, Jacob Carlborg wrote:
 On 2019-02-25 02:04, Manu wrote:

 Why wouldn't you do it in the same pass as the .di output?
* Separation of concerns * Simplifying the compiler ("simplifying" is not the correct description, rather avoid making the compiler more complex) I think the .di generation should be a separate tool as well.
I don't think so. h and .di generation have to be in sync with the AST. The easiest way to guarantee that it is is by having it in the compiler test-suite. Therefore it should be part of the main compiler. Also it removes the need of having to add another application to your build.
Feb 25
prev sibling next sibling parent Guillaume Piolat <contact spam.org> writes:
On Monday, 25 February 2019 at 10:20:34 UTC, Jacob Carlborg wrote:
 On 2019-02-25 02:04, Manu wrote:

 Why wouldn't you do it in the same pass as the .di output?
* Separation of concerns * Simplifying the compiler ("simplifying" is not the correct description, rather avoid making the compiler more complex) I think the .di generation should be a separate tool as well.
+1 Also: is there anyone else that need this? Because it could well be a fringe need (like .di generation)
Feb 25
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2/25/19 5:20 AM, Jacob Carlborg wrote:
 On 2019-02-25 02:04, Manu wrote:
 
 Why wouldn't you do it in the same pass as the .di output?
* Separation of concerns * Simplifying the compiler ("simplifying" is not the correct description, rather avoid making the compiler more complex)
Indeed so. There's also the network effect of tooling. Integrating within the compiler would be like the proverbial "giving someone a fish", whereas framing it as a tool that can be the first inspiring many others is akin to "teaching fishing".
Feb 25
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Mon, Feb 25, 2019 at 10:10 AM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 2/25/19 5:20 AM, Jacob Carlborg wrote:
 On 2019-02-25 02:04, Manu wrote:

 Why wouldn't you do it in the same pass as the .di output?
* Separation of concerns
Are we planning to remove .di output?
 * Simplifying the compiler ("simplifying" is not the correct
 description, rather avoid making the compiler more complex)
It seems theoretically very simple to me; whatever the .di code looks like, I can imagine a filter for isExternCorCPP() on candidate nodes when walking the AST. Seems like a pretty simple tweak of the existing code... but I haven't looked at it. I suspect 1 line in the AST walk code, and 99% of the job, a big ugly block that emits a C++ declaration instead of the D declaration?
 Indeed so. There's also the network effect of tooling. Integrating
 within the compiler would be like the proverbial "giving someone a
 fish", whereas framing it as a tool that can be the first inspiring many
 others is akin to "teaching fishing".
That sounds nice, but it's bollocks though; give me dtoh, i'm about 95% less likely to use it. It's easy to add a flag to the command line of our hyper-complex build, but reworking custom tooling into it, not so much. I'm not a build engineer, and I have no idea how I'd wire a second pass to each source compile if I wanted to. Tell me how to wire that into VS? How do I wite that into XCode? How do I express that in the scripts that emit those project formats, and also makefiles and ninja? How do I express that the outputs (which are .h files) are correctly expressed as inputs of dependent .cpp compile steps? At best, it would take me hours or days of implementing that comprehensive solution, it might not be possible in all build environments (XCode is a disaster), and I will never spare that time. Give me dtoh, you give me problems, not solutions. Certainly it *could* be a separate tool, but your argument that it's more enabling as a separate tool is the opposite of the truth. At best, it'll just waste more precious CI time and complicate our build.
Feb 25
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 2/25/19 2:04 PM, Manu wrote:
 On Mon, Feb 25, 2019 at 10:10 AM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 2/25/19 5:20 AM, Jacob Carlborg wrote:
 On 2019-02-25 02:04, Manu wrote:

 Why wouldn't you do it in the same pass as the .di output?
* Separation of concerns
Are we planning to remove .di output?
 * Simplifying the compiler ("simplifying" is not the correct
 description, rather avoid making the compiler more complex)
It seems theoretically very simple to me; whatever the .di code looks like, I can imagine a filter for isExternCorCPP() on candidate nodes when walking the AST. Seems like a pretty simple tweak of the existing code... but I haven't looked at it. I suspect 1 line in the AST walk code, and 99% of the job, a big ugly block that emits a C++ declaration instead of the D declaration?
 Indeed so. There's also the network effect of tooling. Integrating
 within the compiler would be like the proverbial "giving someone a
 fish", whereas framing it as a tool that can be the first inspiring many
 others is akin to "teaching fishing".
That sounds nice, but it's bollocks though; give me dtoh, i'm about 95% less likely to use it. It's easy to add a flag to the command line of our hyper-complex build, but reworking custom tooling into it, not so much.
More like dog's ones, right? :o) There are indeed arguments going either way. The point is a universe of tools can be built based on the compiler as a library, of which only a minority should be realistically integrated within the compiler itself. That said, I'd take such work in either form! Andrei
Feb 25
parent Manu <turkeyman gmail.com> writes:
On Mon, Feb 25, 2019 at 12:25 PM Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 2/25/19 2:04 PM, Manu wrote:
 On Mon, Feb 25, 2019 at 10:10 AM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 2/25/19 5:20 AM, Jacob Carlborg wrote:
 On 2019-02-25 02:04, Manu wrote:

 Why wouldn't you do it in the same pass as the .di output?
* Separation of concerns
Are we planning to remove .di output?
 * Simplifying the compiler ("simplifying" is not the correct
 description, rather avoid making the compiler more complex)
It seems theoretically very simple to me; whatever the .di code looks like, I can imagine a filter for isExternCorCPP() on candidate nodes when walking the AST. Seems like a pretty simple tweak of the existing code... but I haven't looked at it. I suspect 1 line in the AST walk code, and 99% of the job, a big ugly block that emits a C++ declaration instead of the D declaration?
 Indeed so. There's also the network effect of tooling. Integrating
 within the compiler would be like the proverbial "giving someone a
 fish", whereas framing it as a tool that can be the first inspiring many
 others is akin to "teaching fishing".
That sounds nice, but it's bollocks though; give me dtoh, i'm about 95% less likely to use it. It's easy to add a flag to the command line of our hyper-complex build, but reworking custom tooling into it, not so much.
More like dog's ones, right? :o) There are indeed arguments going either way. The point is a universe of tools can be built based on the compiler as a library, of which only a minority should be realistically integrated within the compiler itself.
Right, but in this case, the technology is *already* in the compiler. I'm not suggesting a large new development, just a filter on the output of the existing pass with a bit of a re-format. That form would be so much more readily useful.
 That said, I'd take such work in either form!
Perhaps. But I'd like to strongly encourage a form that's useful to me as best as I can... otherwise it's just a nice talking point and still no practical solution.
Feb 25
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Feb 25, 2019 at 11:04:56AM -0800, Manu via Digitalmars-d wrote:
 On Mon, Feb 25, 2019 at 10:10 AM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
[...]
 Indeed so. There's also the network effect of tooling. Integrating
 within the compiler would be like the proverbial "giving someone a
 fish", whereas framing it as a tool that can be the first inspiring
 many others is akin to "teaching fishing".
That sounds nice, but it's bollocks though; give me dtoh, i'm about 95% less likely to use it. It's easy to add a flag to the command line of our hyper-complex build, but reworking custom tooling into it, not so much. I'm not a build engineer, and I have no idea how I'd wire a second pass to each source compile if I wanted to. Tell me how to wire that into VS? How do I wite that into XCode? How do I express that in the scripts that emit those project formats, and also makefiles and ninja? How do I express that the outputs (which are .h files) are correctly expressed as inputs of dependent .cpp compile steps?
[...] <off-topic rant> This is a perfect example of what has gone completely wrong in the world of build systems. Too many assumptions and poor designs over an extremely simple and straightforward dependency graph walk algorithm, that turn something that ought to be trivial to implement into a gargantuan task that requires a dedicated job title like "build engineer". It's completely insane, yet people accept it as a fact of life. It boggles the mind. </off-topic rant> T -- Those who don't understand Unix are condemned to reinvent it, poorly.
Feb 25
next sibling parent reply Rubn <where is.this> writes:
On Monday, 25 February 2019 at 19:28:54 UTC, H. S. Teoh wrote:
 On Mon, Feb 25, 2019 at 11:04:56AM -0800, Manu via 
 Digitalmars-d wrote:
 On Mon, Feb 25, 2019 at 10:10 AM Andrei Alexandrescu via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
[...]
 Indeed so. There's also the network effect of tooling. 
 Integrating within the compiler would be like the proverbial 
 "giving someone a fish", whereas framing it as a tool that 
 can be the first inspiring many others is akin to "teaching 
 fishing".
That sounds nice, but it's bollocks though; give me dtoh, i'm about 95% less likely to use it. It's easy to add a flag to the command line of our hyper-complex build, but reworking custom tooling into it, not so much. I'm not a build engineer, and I have no idea how I'd wire a second pass to each source compile if I wanted to. Tell me how to wire that into VS? How do I wite that into XCode? How do I express that in the scripts that emit those project formats, and also makefiles and ninja? How do I express that the outputs (which are .h files) are correctly expressed as inputs of dependent .cpp compile steps?
[...] <off-topic rant> This is a perfect example of what has gone completely wrong in the world of build systems. Too many assumptions and poor designs over an extremely simple and straightforward dependency graph walk algorithm, that turn something that ought to be trivial to implement into a gargantuan task that requires a dedicated job title like "build engineer". It's completely insane, yet people accept it as a fact of life. It boggles the mind. </off-topic rant> T
I don't think it is as simple as you make it seem. Especially when you need to start adding components that need to be build that isn't source code. Add in different different operating systems to that. Each have different requirements and how do you not make assumptions? You have to implement something in some way, you can't just not implement it. Doing so you weigh the benefits and drawbacks of certain implementations. That means a build system may not be suitable for all circumstances and it is impossible for it to. If you are building something simple it is very easy to say that build-systems are over complicated. If you don't have to worry about not rebuilding files that don't have to, then it becomes extremely simple, my <50 line build script is all I need but then that becomes more complicated if I don't want to rebuild files that don't need to. That is especially bad for something like D where you can import any where and with mixins that can import as well using from!"std.stdio". It's easy to say build-systems are overly complicated until you actually work on a big project.
Feb 25
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Feb 25, 2019 at 10:14:18PM +0000, Rubn via Digitalmars-d wrote:
 On Monday, 25 February 2019 at 19:28:54 UTC, H. S. Teoh wrote:
[...]
 <off-topic rant>
 This is a perfect example of what has gone completely wrong in the world
 of build systems. Too many assumptions and poor designs over an
 extremely simple and straightforward dependency graph walk algorithm,
 that turn something that ought to be trivial to implement into a
 gargantuan task that requires a dedicated job title like "build
 engineer".  It's completely insane, yet people accept it as a fact of
 life. It boggles the mind.
 </off-topic rant>
[...]
 I don't think it is as simple as you make it seem. Especially when you
 need to start adding components that need to be build that isn't
 source code.
It's very simple. The build description is essentially a DAG whose nodes represent files (well, any product, really, but let's say files for a concrete example), and whose edges represent commands that transform input files into output files. All the build system has to do is to do a topological walk of this DAG, and execute the commands associated with each edge to derive the output from the input. This is all that's needed. The rest are all fluff. The basic problem with today's build systems is that they impose arbitrary assumptions on top of this simple DAG. For example, all input nodes are arbitrarily restricted to source code files, or in some bad cases, source code of some specific language or set of languages. Then they arbitrarily limit edges to be only compiler invocations and/or linker invocations. So the result is that if you have an input file that isn't source code, or if the output file requires invoking something other than a compiler/linker, then the build system doesn't support it and you're left out in the cold. Worse yet, many "modern" build systems assume a fixed depth of paths in the graph, i.e., you can only compile source files into binaries, you cannot compile a subset of source files into an auxiliary utility that in turn generates new source files that are then compiled into an executable. So automatic code generation is ruled out, preprocessing is ruled out, etc., unless you shoehorn all of that into the compiler invocation, which is a ridiculous idea. None of these restrictions are necessary, and they only needlessly limit what you can do with your build system. I understand that these assumptions are primarily to simplify the build description, e.g., by inferring dependencies so that you don't have to specify edges and nodes yourself (which is obviously impractical for large projects). But these additional niceties ought to be implemented as a SEPARATE layer on top of the topological walk, and the user should not be arbitrarily prevented from directly accessing the DAG description. The way so many build systems are designed is that either you have to do everything manually, like makefiles, which everybody hates, or the hood is welded shut and you can only do what the authors decide that you should be able to do and nothing else. [...]
 It's easy to say build-systems are overly complicated until you
 actually work on a big project.
You seem to think that I'm talking out of an ivory tower. I assure you I know what I'm talking about. I have written actual build systems that do things like this: - Compile a subset of source files into a utility; - Run said utility to transform certain input data files into source code; - Compile the generated source code into executables; - Run said executables on other data files to transform the data into PovRay scene files; - Run PovRay to produce images; - Run post-processing utilities on said images to crop / reborder them; - Run another utility to convert these images into animations; - Install these animations into a target directory. - Compile another set of source files into a different utility; - Run said utility on input files to transform them to PHP input files; - Run php-cli to generate HTML from said input files; - Install said HTML files into a target directory. - Run a network utility to retrieve the history of a specific log file and pipe it through a filter to extract a list of dates. - Run a utility to transform said dates into a gnuplot input file for generating a graph; - Run gnuplot to create the graph; - Run postprocessing image utilities to touch up the image; - Install the result into the target directory. None of the above are baked-in rules. The user is fully capable of specifying whatever transformation he wants on whatever inputs he wants to produce whatever output he wants. No straitjackets, no stupid hacks to work around stupid build system limitations. Tell it how you want your inputs to be transformed into outputs, and it handles the rest for you. Furthermore, the build system is incremental: if I modify any of the above input files, it automatically runs the necessary commands to derive the updated output files AND NOTHING ELSE (i.e., it does not needlessly re-derive stuff that hasn't changed). Better yet, if any of the intermediate output files are identical to the previous outputs, the build stops right there and does not needlessly recreate other outputs down the line. The build system is also reliable: running the build in a dirty workspace produces identical products as running the build in a fresh checkout. I never have to worry about doing the equivalent of 'make clean; make', which is a stupid thing to have to do in 2019. I have a workspace that hasn't been "cleaned" for months, and running the build on it produces exactly the same outputs as a fresh checkout. There's more I can say, but basically, this is the power that having direct access to the DAG can give you. In this day and age, it's inexcusable not to be able to do this. Any build system that cannot do all of the above is a crippled build system that I will not use, because life is far too short to waste fighting with your build system rather than getting things done. T -- English has the lovely word "defenestrate", meaning "to execute by throwing someone out a window", or more recently "to remove Windows from a computer and replace it with something useful". :-) -- John Cowan
Feb 25
next sibling parent reply ted <nospam example.org> writes:
On 26/2/19 9:25 am, H. S. Teoh wrote:
 On Mon, Feb 25, 2019 at 10:14:18PM +0000, Rubn via Digitalmars-d wrote:
 On Monday, 25 February 2019 at 19:28:54 UTC, H. S. Teoh wrote:
[...]
 <off-topic rant>
 This is a perfect example of what has gone completely wrong in the world
 of build systems. Too many assumptions and poor designs over an
 extremely simple and straightforward dependency graph walk algorithm,
 that turn something that ought to be trivial to implement into a
 gargantuan task that requires a dedicated job title like "build
 engineer".  It's completely insane, yet people accept it as a fact of
 life. It boggles the mind.
 </off-topic rant>
[...]
 I don't think it is as simple as you make it seem. Especially when you
 need to start adding components that need to be build that isn't
 source code.
It's very simple. The build description is essentially a DAG whose nodes represent files (well, any product, really, but let's say files for a concrete example), and whose edges represent commands that transform input files into output files. All the build system has to do is to do a topological walk of this DAG, and execute the commands associated with each edge to derive the output from the input. This is all that's needed. The rest are all fluff. The basic problem with today's build systems is that they impose arbitrary assumptions on top of this simple DAG. For example, all input nodes are arbitrarily restricted to source code files, or in some bad cases, source code of some specific language or set of languages. Then they arbitrarily limit edges to be only compiler invocations and/or linker invocations. So the result is that if you have an input file that isn't source code, or if the output file requires invoking something other than a compiler/linker, then the build system doesn't support it and you're left out in the cold. Worse yet, many "modern" build systems assume a fixed depth of paths in the graph, i.e., you can only compile source files into binaries, you cannot compile a subset of source files into an auxiliary utility that in turn generates new source files that are then compiled into an executable. So automatic code generation is ruled out, preprocessing is ruled out, etc., unless you shoehorn all of that into the compiler invocation, which is a ridiculous idea. None of these restrictions are necessary, and they only needlessly limit what you can do with your build system. I understand that these assumptions are primarily to simplify the build description, e.g., by inferring dependencies so that you don't have to specify edges and nodes yourself (which is obviously impractical for large projects). But these additional niceties ought to be implemented as a SEPARATE layer on top of the topological walk, and the user should not be arbitrarily prevented from directly accessing the DAG description. The way so many build systems are designed is that either you have to do everything manually, like makefiles, which everybody hates, or the hood is welded shut and you can only do what the authors decide that you should be able to do and nothing else. [...]
 It's easy to say build-systems are overly complicated until you
 actually work on a big project.
You seem to think that I'm talking out of an ivory tower. I assure you I know what I'm talking about. I have written actual build systems that do things like this: - Compile a subset of source files into a utility; - Run said utility to transform certain input data files into source code; - Compile the generated source code into executables; - Run said executables on other data files to transform the data into PovRay scene files; - Run PovRay to produce images; - Run post-processing utilities on said images to crop / reborder them; - Run another utility to convert these images into animations; - Install these animations into a target directory. - Compile another set of source files into a different utility; - Run said utility on input files to transform them to PHP input files; - Run php-cli to generate HTML from said input files; - Install said HTML files into a target directory. - Run a network utility to retrieve the history of a specific log file and pipe it through a filter to extract a list of dates. - Run a utility to transform said dates into a gnuplot input file for generating a graph; - Run gnuplot to create the graph; - Run postprocessing image utilities to touch up the image; - Install the result into the target directory. None of the above are baked-in rules. The user is fully capable of specifying whatever transformation he wants on whatever inputs he wants to produce whatever output he wants. No straitjackets, no stupid hacks to work around stupid build system limitations. Tell it how you want your inputs to be transformed into outputs, and it handles the rest for you. Furthermore, the build system is incremental: if I modify any of the above input files, it automatically runs the necessary commands to derive the updated output files AND NOTHING ELSE (i.e., it does not needlessly re-derive stuff that hasn't changed). Better yet, if any of the intermediate output files are identical to the previous outputs, the build stops right there and does not needlessly recreate other outputs down the line. The build system is also reliable: running the build in a dirty workspace produces identical products as running the build in a fresh checkout. I never have to worry about doing the equivalent of 'make clean; make', which is a stupid thing to have to do in 2019. I have a workspace that hasn't been "cleaned" for months, and running the build on it produces exactly the same outputs as a fresh checkout. There's more I can say, but basically, this is the power that having direct access to the DAG can give you. In this day and age, it's inexcusable not to be able to do this. Any build system that cannot do all of the above is a crippled build system that I will not use, because life is far too short to waste fighting with your build system rather than getting things done. T
I'd be interested in your thoughts on https://github.com/GrahamStJack/bottom-up-build We use it here (commercial environment with deliverables into defence & commercial customers) (it was created as a response to poor existing build tools). It is rigid on preventing circularities, and deals with code-generation as part of its build cycle. It currently deals with a mixed C++/D codebase of well over 1/2 million lines. It is agnostic to the tool chain - that's part of the configuration - we just use c++ (clang & gcc), and D (dmd & ldc). Also allows codebase to be split amongst multiple repositories. --ted
Feb 25
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Feb 26, 2019 at 11:44:28AM +1030, ted via Digitalmars-d wrote:
[...]
 I'd be interested in your thoughts on
 https://github.com/GrahamStJack/bottom-up-build
 
 We use it here (commercial environment with deliverables into defence
 & commercial customers) (it was created as a response to poor existing
 build tools). It is rigid on preventing circularities, and deals with
 code-generation as part of its build cycle. It currently deals with a
 mixed C++/D codebase of well over 1/2 million lines. It is agnostic to
 the tool chain - that's part of the configuration - we just use c++
 (clang & gcc), and D (dmd & ldc). Also allows codebase to be split
 amongst multiple repositories.
[...] Sorry didn't get around to this until now. At first glance, it looks like a kind of make variant -- sorry if this, or any of the following comments, is wrong, as I only briefly skimmed over the example and some of the bub source code. It seems to follow make's model of global variables for things like compile flags and so forth. Which is convenient, but could lead to messiness in large enough projects -- i.e., it's not clear to me at first glance how to handle compiling source code with different compile options, or how to generate multiple binaries (e.g., native compile + cross compile) from the same set of sources. Perhaps encapsulating these variables in configuration objects (e.g. SCons' "environments") might help with this. Not scanning for dependencies until a target needs to be built is quite a clever idea. I'll have to keep that in mind if I ever decide to invent my own build system. :-D Splitting a codebase across multiple repositories is also a nice idea. I wonder if it could be taken further: in the form of a build graph export into some kind of standard format, that potentially other build tools can import and build with equivalent results. That's the kind of "holy grail" of build tools that I envision these days, a way to unify all the divergent build tools out there and make it possible to integrate projects across different build systems. There seems to be baked-in rules for generating executables, static / dynamic libraries, etc., and it appears that some special handling is done in these cases. It's not clear to me how to achieve equivalent functionality just from the various bub configuration files if the user had to, for example, support a new programming language that needed specialized scanning for dependencies, or how to handle multi-input, multi-output transformations like the Java compiler (does bub support building Java?). Would it be necessary to modify bub source code in order to extend it to handle such cases, or is it already possible with bub.cfg? I took a quick look at bub.planner.doPlanning, and it appears to me that it has to scan the entire source tree in order to determine what file(s) have changed. Which would make the cost of a build proportional to the size of the workspace, rather than the size of the changeset. Is this correct? If so, I'd suggest taking a look at Tup (http://gittup.org/tup/), which uses modern OS features (inotify / the Windows equivalent which I can't remember) to detect which sources have changed, and correspondingly which subgraph of the entire project's DAG is pertinent at the next build command -- the rest of the DAG is skipped, which in a large project can greatly improve turnaround build times. Also, are partial builds supported? T -- Programming is not just an act of telling a computer what to do: it is also an act of telling other programmers what you wished the computer to do. Both are important, and the latter deserves care. -- Andrew Morton
Feb 27
parent ted <nospam example.org> writes:
I'm not the writer of bub, so I'll try and answer as best I can.






On 28/2/19 2:51 am, H. S. Teoh wrote:
 On Tue, Feb 26, 2019 at 11:44:28AM +1030, ted via Digitalmars-d wrote:
 [...]
 I'd be interested in your thoughts on
 https://github.com/GrahamStJack/bottom-up-build

 We use it here (commercial environment with deliverables into defence
 & commercial customers) (it was created as a response to poor existing
 build tools). It is rigid on preventing circularities, and deals with
 code-generation as part of its build cycle. It currently deals with a
 mixed C++/D codebase of well over 1/2 million lines. It is agnostic to
 the tool chain - that's part of the configuration - we just use c++
 (clang & gcc), and D (dmd & ldc). Also allows codebase to be split
 amongst multiple repositories.
[...] Sorry didn't get around to this until now. At first glance, it looks like a kind of make variant -- sorry if this, or any of the following comments, is wrong, as I only briefly skimmed over the example and some of the bub source code. It seems to follow make's model of global variables for things like compile flags and so forth. Which is convenient, but could lead to messiness in large enough projects -- i.e., it's not clear to me at first glance how to handle compiling source code with different compile options, or how to generate multiple binaries (e.g., native compile + cross compile) from the same set of sources. Perhaps encapsulating these variables in configuration objects (e.g. SCons' "environments") might help with this.
This is handled by having multiple bub.cfg files. As the build directory is located elsewhere and the build is performed in that directory, the setup for each build directory is done once - so this has worked well in practice.
 
 Not scanning for dependencies until a target needs to be built is quite
 a clever idea. I'll have to keep that in mind if I ever decide to invent
 my own build system. :-D
 
 Splitting a codebase across multiple repositories is also a nice idea. I
 wonder if it could be taken further: in the form of a build graph export
 into some kind of standard format, that potentially other build tools
 can import and build with equivalent results. That's the kind of "holy
 grail" of build tools that I envision these days, a way to unify all the
 divergent build tools out there and make it possible to integrate
 projects across different build systems.
I know this build graph is available - I believe it can be output from a commandline switch - so getting it into a standard format would be quite easy.
 
 There seems to be baked-in rules for generating executables, static /
 dynamic libraries, etc., and it appears that some special handling is
 done in these cases.  It's not clear to me how to achieve equivalent
 functionality just from the various bub configuration files if the user
 had to, for example, support a new programming language that needed
 specialized scanning for dependencies, or how to handle multi-input,
 multi-output transformations like the Java compiler (does bub support
 building Java?).  Would it be necessary to modify bub source code in
 order to extend it to handle such cases, or is it already possible with
 bub.cfg?
It is a product of the environment in which it is used. i.e. there has been no push for anything else at this stage. I'm fairly sure that Graham would take suggestions like this on board.
 
 I took a quick look at bub.planner.doPlanning, and it appears to me that
 it has to scan the entire source tree in order to determine what file(s)
 have changed. Which would make the cost of a build proportional to the
 size of the workspace, rather than the size of the changeset.  Is this
 correct?  If so, I'd suggest taking a look at Tup
 (http://gittup.org/tup/), which uses modern OS features (inotify / the
 Windows equivalent which I can't remember) to detect which sources have
 changed, and correspondingly which subgraph of the entire project's DAG
 is pertinent at the next build command -- the rest of the DAG is
 skipped, which in a large project can greatly improve turnaround build
 times.
On our codebase, this step takes less than a second, so it hasn't been an issue.
 
 Also, are partial builds supported?
By partial builds, if you mean that it only builds the files that are affected by the code change just made? In that case yes. If you mean the equivalent of a makefile target where a subset of files is defined, then no. (Again, there is no call for this within our environment)
 
 
 T
 
--ted
Feb 28
prev sibling parent reply Rubn <where is.this> writes:
On Monday, 25 February 2019 at 22:55:18 UTC, H. S. Teoh wrote:
 On Mon, Feb 25, 2019 at 10:14:18PM +0000, Rubn via 
 Digitalmars-d wrote:
 On Monday, 25 February 2019 at 19:28:54 UTC, H. S. Teoh wrote:
[...]
 <off-topic rant>
 This is a perfect example of what has gone completely wrong 
 in the world
 of build systems. Too many assumptions and poor designs over 
 an
 extremely simple and straightforward dependency graph walk 
 algorithm,
 that turn something that ought to be trivial to implement 
 into a
 gargantuan task that requires a dedicated job title like 
 "build
 engineer".  It's completely insane, yet people accept it as 
 a fact of
 life. It boggles the mind.
 </off-topic rant>
[...]
 I don't think it is as simple as you make it seem. Especially 
 when you need to start adding components that need to be build 
 that isn't source code.
It's very simple. The build description is essentially a DAG whose nodes represent files (well, any product, really, but let's say files for a concrete example), and whose edges represent commands that transform input files into output files. All the build system has to do is to do a topological walk of this DAG, and execute the commands associated with each edge to derive the output from the input. This is all that's needed. The rest are all fluff. The basic problem with today's build systems is that they impose arbitrary assumptions on top of this simple DAG. For example, all input nodes are arbitrarily restricted to source code files, or in some bad cases, source code of some specific language or set of languages. Then they arbitrarily limit edges to be only compiler invocations and/or linker invocations. So the result is that if you have an input file that isn't source code, or if the output file requires invoking something other than a compiler/linker, then the build system doesn't support it and you're left out in the cold. Worse yet, many "modern" build systems assume a fixed depth of paths in the graph, i.e., you can only compile source files into binaries, you cannot compile a subset of source files into an auxiliary utility that in turn generates new source files that are then compiled into an executable. So automatic code generation is ruled out, preprocessing is ruled out, etc., unless you shoehorn all of that into the compiler invocation, which is a ridiculous idea.
What build systems are you talking about here? I mean I can search for programs that do certain things and I'll most definitely find more subpar ones than any spectacular ones. Especially if they are free. So we are on the same page on which build systems you are referring to.
 None of these restrictions are necessary, and they only 
 needlessly limit what you can do with your build system.

 I understand that these assumptions are primarily to simplify 
 the build description, e.g., by inferring dependencies so that 
 you don't have to specify edges and nodes yourself (which is 
 obviously impractical for large projects).  But these 
 additional niceties ought to be implemented as a SEPARATE layer 
 on top of the topological walk, and the user should not be 
 arbitrarily prevented from directly accessing the DAG 
 description.  The way so many build systems are designed is 
 that either you have to do everything manually, like makefiles, 
 which everybody hates, or the hood is welded shut and you can 
 only do what the authors decide that you should be able to do 
 and nothing else.


 [...]
 It's easy to say build-systems are overly complicated until 
 you actually work on a big project.
You seem to think that I'm talking out of an ivory tower. I assure you I know what I'm talking about. I have written actual build systems that do things like this: - Compile a subset of source files into a utility; - Run said utility to transform certain input data files into source code; - Compile the generated source code into executables; - Run said executables on other data files to transform the data into PovRay scene files; - Run PovRay to produce images; - Run post-processing utilities on said images to crop / reborder them; - Run another utility to convert these images into animations; - Install these animations into a target directory. - Compile another set of source files into a different utility; - Run said utility on input files to transform them to PHP input files; - Run php-cli to generate HTML from said input files; - Install said HTML files into a target directory. - Run a network utility to retrieve the history of a specific log file and pipe it through a filter to extract a list of dates. - Run a utility to transform said dates into a gnuplot input file for generating a graph; - Run gnuplot to create the graph; - Run postprocessing image utilities to touch up the image; - Install the result into the target directory.
Yes doing all those things isn't all that difficult, it really is just a matter of calling a different program to generate the file. The difficulty of build systems comes in when you have an extremely large project that takes a long time to build.
 None of the above are baked-in rules. The user is fully capable 
 of specifying whatever transformation he wants on whatever 
 inputs he wants to produce whatever output he wants.  No 
 straitjackets, no stupid hacks to work around stupid build 
 system limitations. Tell it how you want your inputs to be 
 transformed into outputs, and it handles the rest for you.

 Furthermore, the build system is incremental: if I modify any 
 of the above input files, it automatically runs the necessary 
 commands to derive the updated output files AND NOTHING ELSE 
 (i.e., it does not needlessly re-derive stuff that hasn't 
 changed).  Better yet, if any of the intermediate output files 
 are identical to the previous outputs, the build stops right 
 there and does not needlessly recreate other outputs down the 
 line.

 The build system is also reliable: running the build in a dirty 
 workspace produces identical products as running the build in a 
 fresh checkout.  I never have to worry about doing the 
 equivalent of 'make clean; make', which is a stupid thing to 
 have to do in 2019. I have a workspace that hasn't been 
 "cleaned" for months, and running the build on it produces 
 exactly the same outputs as a fresh checkout.
It really depends on what you are building. Working on DMD I don't have to do a clean, doing a bisect though I effective have to do a clean every new commit.
 There's more I can say, but basically, this is the power that 
 having direct access to the DAG can give you.  In this day and 
 age, it's inexcusable not to be able to do this.

 Any build system that cannot do all of the above is a crippled 
 build system that I will not use, because life is far too short 
 to waste fighting with your build system rather than getting 
 things done.


 T
The build systems I've used can do all that, the problem is about functionality so much as the ease of achieving that functionality. I just use a script, don't need a build system but doing a fully build of my project only takes 10 seconds so I have that luxury.
Feb 25
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Feb 26, 2019 at 01:33:50AM +0000, Rubn via Digitalmars-d wrote:
 On Monday, 25 February 2019 at 22:55:18 UTC, H. S. Teoh wrote:
[...]
 What build systems are you talking about here? I mean I can search for
 programs that do certain things and I'll most definitely find more
 subpar ones than any spectacular ones. Especially if they are free. So
 we are on the same page on which build systems you are referring to.
SCons is free, and does all of what I described and more. It's not perfect, of course. But it's miles better than, say, make -- for its unreliability and the tendency for makefiles to become unreadably complex and unmaintainable. Or dub, for forcing you to work a certain way and unable to express things like multi-stage builds or non-compilation tasks. [...]
 - Compile a subset of source files into a utility;
 
 - Run said utility to transform certain input data files into source
   code;
 
 - Compile the generated source code into executables;
 
 - Run said executables on other data files to transform the data into
   PovRay scene files;
 
 - Run PovRay to produce images;
 
 - Run post-processing utilities on said images to crop / reborder them;
 
 - Run another utility to convert these images into animations;
 
 - Install these animations into a target directory.
[...]
 Yes doing all those things isn't all that difficult, it really is just
 a matter of calling a different program to generate the file.
And yet the above build is not expressible in dub.
 The difficulty of build systems comes in when you have an extremely
 large project that takes a long time to build.
The above steps are part of a project I have whose full build takes about 5-6 hours. But while working on it, the build turnaround time is about 10-15 seconds (and that's only because SCons didn't get one thing right: that build time should be proportional to changeset size, rather than the size of the entire workspace -- otherwise it would be more like 3-4 seconds). *That's* what I call a sane build system. [...]
 It really depends on what you are building. Working on DMD I don't
 have to do a clean, doing a bisect though I effective have to do a
 clean every new commit.
Well exactly, that's the stupidity of it. You always have to 'make clean', "just to be sure", even if "most of the time" it works. It's 2019, and algorithms for reliable builds have been known for at least a decade or more, yet we're still stuck in the dark ages of "occasionally I have to run make clean, and maybe I should do it right now 'cos I'm not sure if this bug is caused by out-of-sync object files or if it's a real bug". Can you imagine how ridiculous it would be with the above 5-6 hour build script, if I had built that project out of makefiles? I would get absolutely nothing done at all if every once in a while I have to `make clean` "just to be sure". Thankfully, SCons is sane enough that I don't have to rerun the entire build for months on end -- actually, I never had to do it. Even when there were big changes that cause almost the whole thing to rebuild, it was SCons that figured out that it had to rebuild everything; I never had to tell it to. Every time I build, no matter what state the workspace was in, it would always update everything correctly. I can even `git checkout <branch>` all over the place, and it doesn't lose track of how to update all relevant targets. I never have to hold its hand to get it to do the right thing, it Just Works(tm). *That's* what I call a sane system. (In spite of said SCons warts.)
 There's more I can say, but basically, this is the power that having
 direct access to the DAG can give you.  In this day and age, it's
 inexcusable not to be able to do this.
 
 Any build system that cannot do all of the above is a crippled build
 system that I will not use, because life is far too short to waste
 fighting with your build system rather than getting things done.
[...]
 The build systems I've used can do all that, the problem is about
 functionality so much as the ease of achieving that functionality.
Well, yes. That's why I repeatedly say, a proper design should empower the user. Easy things should be easy, and hard things should be possible. It shouldn't be the case that easy things are hard (e.g. Manu's "if I have to run an extra step before compilation, I have to bend backwards and recite gibberish in encrypted Reverse Klingon to get make to do the right thing"), and hard things are either outright impossible, or practically impossible because it's so onerous you might as well not bother trying.
 I just use a script, don't need a build system but doing a fully build
 of my project only takes 10 seconds so I have that luxury.
As I said, my website project takes about 5-6 hours for a full, clean build. Anything less than a sane build system -- or a mostly-sane one (SCons does have its warts like I said) -- is simply not even worth my consideration. Life is too short to have to take 6-hour coffee breaks every other day just because make is too dumb to produce reliable builds. T -- We are in class, we are supposed to be learning, we have a teacher... Is it too much that I expect him to teach me??? -- RL
Feb 25
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Mon, Feb 25, 2019 at 2:55 PM H. S. Teoh via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Mon, Feb 25, 2019 at 10:14:18PM +0000, Rubn via Digitalmars-d wrote:
 On Monday, 25 February 2019 at 19:28:54 UTC, H. S. Teoh wrote:
[...]
 <off-topic rant>
 This is a perfect example of what has gone completely wrong in the world
 of build systems. Too many assumptions and poor designs over an
 extremely simple and straightforward dependency graph walk algorithm,
 that turn something that ought to be trivial to implement into a
 gargantuan task that requires a dedicated job title like "build
 engineer".  It's completely insane, yet people accept it as a fact of
 life. It boggles the mind.
 </off-topic rant>
[...]
 I don't think it is as simple as you make it seem. Especially when you
 need to start adding components that need to be build that isn't
 source code.
It's very simple. The build description is essentially a DAG whose nodes represent files (well, any product, really, but let's say files for a concrete example), and whose edges represent commands that transform input files into output files. All the build system has to do is to do a topological walk of this DAG, and execute the commands associated with each edge to derive the output from the input.
You don't know the edges of the DAG until AFTER you run the compiler (ie, discovering imports/#includes, etc from the source code) You also want to run the build with all 64 cores in your machine. File B's build depends on file A's build output, but it can't know that until after it attempts (and fails) to build B... How do you resolve this tension? There's no 'simple' solution to this problem that I'm aware of. You start to address this with higher-level structure, and that is not a 'simple DAG' anymore. Now... whatever solution you concluded; express that in make, ninja, MSBuild, .xcodeproj...
Feb 25
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Feb 25, 2019 at 05:24:00PM -0800, Manu via Digitalmars-d wrote:
 On Mon, Feb 25, 2019 at 2:55 PM H. S. Teoh via Digitalmars-d
[...]
 It's very simple. The build description is essentially a DAG whose
 nodes represent files (well, any product, really, but let's say
 files for a concrete example), and whose edges represent commands
 that transform input files into output files. All the build system
 has to do is to do a topological walk of this DAG, and execute the
 commands associated with each edge to derive the output from the
 input.
You don't know the edges of the DAG until AFTER you run the compiler (ie, discovering imports/#includes, etc from the source code)
Yes, that's what scanners are for. There can be standard scanners for common languages like C, C++, Java, C#, etc.. I didn't say that you have to write DAG nodes and edges by hand. But my point is that prebaked automatic scanning rules of this sort should not *exclude* you from directly adding your own DAG nodes and edges. Build systems like SCons offer an interface for building your own scanners, for example.
 You also want to run the build with all 64 cores in your machine.
Build systems like SCons offer parallel building out-of-the-box, and require no additional user intervention. That's proper design. Makefiles require special care when writing rules in order not to break, and you (last time I checked) have to explicitly specify which rules are parallelizable. That's bad design.
 File B's build depends on file A's build output, but it can't know
 that until after it attempts (and fails) to build B...
 
 How do you resolve this tension?
There is no tension. You just do a topological walk on the DAG and run the steps in order. If a step fails, all subsequent steps related to that target are aborted. (Any other products that didn't fail may still continue in that case.) Parallelization works by identifying DAG nodes that aren't dependent on each other and running them in parallel. A proper build system handles this automatically without user intervention. Unless you're talking about altering the DAG as you go -- SCons *does* in fact handle this case. You just have to sequence your build steps such that any new products/targets that are introduced don't invalidate prior steps. A topological walk usually already solves this problem, as long as you don't ask for impossible things like building target A also adds a new dependency to unrelated target B. In the normal case, building A adds dependency to downstream target C (which depends on A), but that's no problem because the topological walk guarantees A is built before C, and by then, we already know of the new dependency and can handle it correctly. I'm starting to sound like I'm promoting SCons as the best thing since sliced bread, but actually SCons has its own share of problems. But I'm just using it as an example of a design that got *some* things right. A good number of things, in fact, in spite of the warts that still exist. It's a lot saner than, say, make, and that's my point. Such a design is possible, and has been done (the multi-stage website build I described in my previous post, btw, is an SCons-based system -- it's not perfect, but already miles ahead of ancient junk like makefiles).
 There's no 'simple' solution to this problem that I'm aware of. You
 start to address this with higher-level structure, and that is not a
 'simple DAG' anymore.
It's still a DAG. You just have some fancy automatic scanning / generation at the higher level structure, but it all turns into a DAG in the end. And here is my point: the build system should ALLOW the user to enter custom DAG nodes/edges as needed, rather than force the user to only use the available prebaked rules -- because there will always be a situation where you need to do something the build tool authors haven't thought of. You should always have the option of going under the hood when you need to. You should never be limited only to what the authors had in mind. I have nothing against prebaked automatic scanners -- but that should not preclude writing your *own* custom scanners if you wanted to. And it should not prevent you from adding rules to the DAG directly. The correct design is always the one that empowers the user, not the one that spoonfeeds the user yet comes in a straitjacket.
 Now... whatever solution you concluded; express that in make, ninja,
 MSBuild, .xcodeproj...
The fact that doing all of this in make (or whatever else) is such a challenge is exactly proof of what I'm saying: these build systems are fundamentally b0rken, and for no good reason. All the technology necessary to make sane builds possible already exists. It's just that too many build systems are still living in the 80's and refusing to move on. And in the meantime, even better build systems are already being implemented, like Tup, where the build time is proportional to the size of change rather than the size of the workspace (an SCons wart). Yet people still use make like it's still 1985, and people still invent build systems with antiquated designs like it's still 1985. T -- Give a man a fish, and he eats once. Teach a man to fish, and he will sit forever.
Feb 25
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 2/25/19 2:28 PM, H. S. Teoh wrote:
 On Mon, Feb 25, 2019 at 11:04:56AM -0800, Manu via Digitalmars-d wrote:
 On Mon, Feb 25, 2019 at 10:10 AM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
[...]
 Indeed so. There's also the network effect of tooling. Integrating
 within the compiler would be like the proverbial "giving someone a
 fish", whereas framing it as a tool that can be the first inspiring
 many others is akin to "teaching fishing".
That sounds nice, but it's bollocks though; give me dtoh, i'm about 95% less likely to use it. It's easy to add a flag to the command line of our hyper-complex build, but reworking custom tooling into it, not so much. I'm not a build engineer, and I have no idea how I'd wire a second pass to each source compile if I wanted to. Tell me how to wire that into VS? How do I wite that into XCode? How do I express that in the scripts that emit those project formats, and also makefiles and ninja? How do I express that the outputs (which are .h files) are correctly expressed as inputs of dependent .cpp compile steps?
[...] <off-topic rant> This is a perfect example of what has gone completely wrong in the world of build systems. Too many assumptions and poor designs over an extremely simple and straightforward dependency graph walk algorithm, that turn something that ought to be trivial to implement into a gargantuan task that requires a dedicated job title like "build engineer". It's completely insane, yet people accept it as a fact of life. It boggles the mind. </off-topic rant>
Hear, hear. When adding another step to a build process ISN'T a simple "add a line to the script", then something has gone very, VERY wrong. (Incidentally, this is part of why I've long since lost all patience for trying to use IDEs like VS, Eclipse, XCode, and whatnot. Life's too short to tolerate all that mess of complexity they turn a basic build into. I still *HATE* with a passion, the fact I have to put up will all that black-box-build bullshit when I use Unity3D - and don't even get me started on the complete and utter garbage that is MSBuild (used by Unity, naturally)). HOWEVER: All that said, when a single build needs to make multiple passes of *the same sources* through the compiler, that's clearly an architectural failure on the part of the tooling. We can argue all we want about how separate tools is technically superior, but if means adding duplicate passes *and* extra complications to the user's buildsystem, then it clearly ISN'T "technically superior", it's just a different set of tradeoffs and yet another example of D letting perfect be the enemy of good.
Feb 27
parent Daniel N <no public.email> writes:
On Wednesday, 27 February 2019 at 18:48:06 UTC, Nick Sabalausky 
(Abscissa) wrote:
 On 2/25/19 2:28 PM, H. S. Teoh wrote:
 On Mon, Feb 25, 2019 at 11:04:56AM -0800, Manu via 
 Digitalmars-d wrote:
 <off-topic rant>
 This is a perfect example of what has gone completely wrong in 
 the world
 of build systems. Too many assumptions and poor designs over an
 extremely simple and straightforward dependency graph walk 
 algorithm,
 that turn something that ought to be trivial to implement into 
 a
 gargantuan task that requires a dedicated job title like "build
 engineer".  It's completely insane, yet people accept it as a 
 fact of
 life. It boggles the mind.
 </off-topic rant>
 
Hear, hear. When adding another step to a build process ISN'T a simple "add a line to the script", then something has gone very, VERY wrong.
I strongly agree, please consider adding it to the compiler, I had enough of insane build systems for one lifetime, the only buildsystem I need is "dmd -i" If you feel there is a need for a more modular aproach, then I'd rather see the posibility of adding "end user developed" dynamic library plugins to the compiler instead, that aproach can also spur creativity from outside developers just as well as separate tools can.
Feb 27
prev sibling parent Manu <turkeyman gmail.com> writes:
On Mon, Feb 25, 2019 at 11:29 AM H. S. Teoh via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Mon, Feb 25, 2019 at 11:04:56AM -0800, Manu via Digitalmars-d wrote:
 On Mon, Feb 25, 2019 at 10:10 AM Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
[...]
 Indeed so. There's also the network effect of tooling. Integrating
 within the compiler would be like the proverbial "giving someone a
 fish", whereas framing it as a tool that can be the first inspiring
 many others is akin to "teaching fishing".
That sounds nice, but it's bollocks though; give me dtoh, i'm about 95% less likely to use it. It's easy to add a flag to the command line of our hyper-complex build, but reworking custom tooling into it, not so much. I'm not a build engineer, and I have no idea how I'd wire a second pass to each source compile if I wanted to. Tell me how to wire that into VS? How do I wite that into XCode? How do I express that in the scripts that emit those project formats, and also makefiles and ninja? How do I express that the outputs (which are .h files) are correctly expressed as inputs of dependent .cpp compile steps?
[...] <off-topic rant> This is a perfect example of what has gone completely wrong in the world of build systems. Too many assumptions and poor designs over an extremely simple and straightforward dependency graph walk algorithm, that turn something that ought to be trivial to implement into a gargantuan task that requires a dedicated job title like "build engineer". It's completely insane, yet people accept it as a fact of life. It boggles the mind. </off-topic rant>
I couldn't agree more (that existing solutions make it harder than it needs to be... not that it's actually easy in the first place), but this is how it is. I can't change that, and I have work to do. Is D an ecosystem that I use to get my work done, or is it one that I use to do some intellectual masturbation on the weekend? I've been tirelessly trying to make the former my reality for a long time now... I've failed so far... I don't know what to do to correct the trajectory. I figure, if you just try and clear the hurdles; one by one, they will eventually be cleared. But I've also become tired in the meantime.
Feb 25
prev sibling parent Manu <turkeyman gmail.com> writes:
On Mon, Feb 25, 2019 at 2:25 AM Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 2019-02-25 02:04, Manu wrote:

 Why wouldn't you do it in the same pass as the .di output?
* Separation of concerns * Simplifying the compiler ("simplifying" is not the correct description, rather avoid making the compiler more complex) I think the .di generation should be a separate tool as well.
Compile times already suck pretty hard, I feel like it's a very valuable feature that DMD can emit .di from the same compile pass. It's already done all the work... why repeat that build cost a second time for every source file?
Feb 25
prev sibling next sibling parent reply Nicolas D <nicolas serveur.io> writes:
On Sunday, 24 February 2019 at 23:24:53 UTC, Manu wrote:
 I've been talking about this for years, and logged this a while 
 back: https://issues.dlang.org/show_bug.cgi?id=19579

 Is there anyone interested in or knows how to do this?
 It would really be super valuable; this idea would motivate
 inter-language projects that typically go 
 C++-fist-with-a-D-binding to
 work the other way.

 Creating a pressure to write D-code first because the binding 
 part is maintained automatically when you compile is 
 potentially significant; gets programmers thinking and writing 
 D code as first-class.

 Idea would be same as emitting .di files, but in this case only
 extern(C)/extern(C++) declarations would be emit to a .h file.

 I could use this to create some nice demos in my office which I 
 think would get a few people talking and excited.
That sounds like a cool project and it's something I'm a bit used to working with, I just need to get familiar with DMD's source and I can start working on that.
Feb 24
parent reply Seb <seb wilzba.ch> writes:
On Monday, 25 February 2019 at 00:09:48 UTC, Nicolas D wrote:
 That sounds like a cool project and it's something I'm a bit 
 used to working with, I just need to get familiar with DMD's 
 source and I can start working on that.
Have a look at this PR for a head-start on C++ header generation with DMD: https://github.com/dlang/dmd/pull/8591 There's also Mihails's dtoh (for C bindings): https://gitlab.com/mihails.strasuns/dtoh
Feb 24
parent Jacob Carlborg <doob me.com> writes:
On 2019-02-25 02:00, Seb wrote:

 Have a look at this PR for a head-start on C++ header generation with DMD:
 
 https://github.com/dlang/dmd/pull/8591
 
 There's also Mihails's dtoh (for C bindings):
 
 https://gitlab.com/mihails.strasuns/dtoh
And: https://github.com/thewilsonator/dtoh -- /Jacob Carlborg
Feb 25
prev sibling parent Andrea Fontana <nospam example.org> writes:
On Sunday, 24 February 2019 at 23:24:53 UTC, Manu wrote:
 I've been talking about this for years, and logged this a while 
 back: https://issues.dlang.org/show_bug.cgi?id=19579

 Is there anyone interested in or knows how to do this?
 It would really be super valuable; this idea would motivate
 inter-language projects that typically go 
 C++-fist-with-a-D-binding to
 work the other way.
+1
Feb 25