www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Idea: std.build instead of dub and make-like tools

reply Piotrek <starpit tlen.pl> writes:
Hi,

What do you think about concentrating D build system around a 
hypothetical "std.build" module instead of investing in dub or 
other custom tools?

Also instead of custom build file format like JSON/SDL/XML/YAML 
we could simply use a d source file, e.g "build.d".

All specification would be included in "std.build".

Did I miss any blocking point?

Piotrek
Mar 16 2016
next sibling parent ZombineDev <petar.p.kirov gmail.com> writes:
On Wednesday, 16 March 2016 at 15:51:38 UTC, Piotrek wrote:
 Hi,

 What do you think about concentrating D build system around a 
 hypothetical "std.build" module instead of investing in dub or 
 other custom tools?

 Also instead of custom build file format like JSON/SDL/XML/YAML 
 we could simply use a d source file, e.g "build.d".

 All specification would be included in "std.build".

 Did I miss any blocking point?

 Piotrek
Related: Reggae - D meta-build system: http://forum.dlang.org/thread/cycyjqdcsdbnkrbziqgl forum.dlang.org http://forum.dlang.org/post/ranqlmrjornlvopsuris forum.dlang.org http://forum.dlang.org/thread/vwkzgbnxvnteefldlvfi forum.dlang.org http://forum.dlang.org/post/znaffgxodfpnjwggpwok forum.dlang.org http://forum.dlang.org/thread/zrguosahjpftxebaqtxp forum.dlang.org
Mar 16 2016
prev sibling next sibling parent reply Dicebot <public dicebot.lv> writes:
On 03/16/2016 05:51 PM, Piotrek wrote:
 Hi,
 
 What do you think about concentrating D build system around a
 hypothetical "std.build" module instead of investing in dub or other
 custom tools?
 
 Also instead of custom build file format like JSON/SDL/XML/YAML we could
 simply use a d source file, e.g "build.d".
 
 All specification would be included in "std.build".
 
 Did I miss any blocking point?
 
 Piotrek
NB: this is orthogonal to development of dub. Most important functionality of dub is dependency management, acting as a build tool is secondary to that (and can be adjusted to support other build systems instead). Idea itself is good and this is something that reggae could do but it isn't as easy project as it may sound.
Mar 16 2016
parent reply Piotrek <starpit tlen.pl> writes:
On Wednesday, 16 March 2016 at 16:36:47 UTC, Dicebot wrote:
 NB: this is orthogonal to development of dub. Most important 
 functionality of dub is dependency management, acting as a 
 build tool is secondary to that (and can be adjusted to support 
 other build systems instead).

 Idea itself is good and this is something that reggae could do 
 but it isn't as easy project as it may sound.
I didn't know about reggae, looks interesting. However after brief review is seems to be overcomplicated and inconvenient in some cases. As for dub I don't think it is unrelated. Why std.build couldn't be dependency manager? What about: std.build = part of reggae + part of dub Piotrek
Mar 16 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
On 03/17/2016 07:15 AM, Piotrek wrote:
 As for dub I don't think it is unrelated. Why std.build couldn't be
 dependency manager?
For same reason you don't want to distribute any other non-trivial tools as sources :) Compilation takes time and has non-trivial dependencies (i.e. networking libraries, git providers etc.), you simply can't put that stuff as a stdlib module/package and expect developers to compile it each time. Even mature build tool would be too complicated to fit into std.build concept - however there is a real benefit in supporting more simple build system that doesn't try to be performant and robust, as that can be used to bootstrap building compiler distribution itself.
 What about:
 std.build = part of reggae + part of dub
Tight coupling of dependency management and build tool in one entity is just too inflexible. This is single biggest issue I have with dub in its current form.
Mar 17 2016
parent reply Piotrek <starpit tlen.pl> writes:
On Thursday, 17 March 2016 at 15:49:07 UTC, Dicebot wrote:
 On 03/17/2016 07:15 AM, Piotrek wrote:
 As for dub I don't think it is unrelated. Why std.build 
 couldn't be dependency manager?
For same reason you don't want to distribute any other non-trivial tools as sources :) Compilation takes time and has non-trivial dependencies (i.e. networking libraries, git providers etc.), you simply can't put that stuff as a stdlib module/package and expect developers to compile it each time.
Hmm, the build module could be compiled once. It sources are supposed to stay unchanged, right?
 Tight coupling of dependency management and build tool in one 
 entity is just too inflexible. This is single biggest issue I 
 have with dub in its current form.
Can you explain it by example (I don't mean dub problems, which I agree exist, but the inflexibility in general)? I can't see a conflict between the two functionalities. Piotrek
Mar 17 2016
parent reply Dicebot <public dicebot.lv> writes:
On 03/17/2016 06:42 PM, Piotrek wrote:
 On Thursday, 17 March 2016 at 15:49:07 UTC, Dicebot wrote:
 On 03/17/2016 07:15 AM, Piotrek wrote:
 As for dub I don't think it is unrelated. Why std.build couldn't be
 dependency manager?
For same reason you don't want to distribute any other non-trivial tools as sources :) Compilation takes time and has non-trivial dependencies (i.e. networking libraries, git providers etc.), you simply can't put that stuff as a stdlib module/package and expect developers to compile it each time.
Hmm, the build module could be compiled once. It sources are supposed to stay unchanged, right?
Even "once" will be too much for majority of D users (those who are not also Gentoo users at least :D). Remember - we are not speaking about build as simple as `rdmd std/build.d` at this point, it will have numerous additional dependencies, including linker dependencies. And has to be perfectly cross-platform.
 Tight coupling of dependency management and build tool in one entity
 is just too inflexible. This is single biggest issue I have with dub
 in its current form.
Can you explain it by example (I don't mean dub problems, which I agree exist, but the inflexibility in general)? I can't see a conflict between the two functionalities.
Typical example - we use custom Makefile based framework for internal Sociomantic D projects, one which is much more capable than existing dub build support and works decently for our needs overall. There is no way it can be replaced with dub at its current state. However using the very same dub to fetch sources of 3d-party libraries and ensure version compatibility is something quite desired. When build tool and package management system are packed into one tool, you pretty much need for both to be perfect to match everyones desires. And that is simply impossible.
Mar 18 2016
parent reply Piotrek <starpit tlen.pl> writes:
On Friday, 18 March 2016 at 15:31:26 UTC, Dicebot wrote:
 Hmm, the build module could be compiled once. It sources are 
 supposed to stay unchanged, right?
Even "once" will be too much for majority of D users (those who are not also Gentoo users at least :D). Remember - we are not speaking about build as simple as `rdmd std/build.d` at this point, it will have numerous additional dependencies, including linker dependencies. And has to be perfectly cross-platform.
I think that std.build could be up to 10KLOC so the compilation time would be not relevant.
 Tight coupling of dependency management and build tool in one 
 entity is just too inflexible. This is single biggest issue I 
 have with dub in its current form.
Can you explain it by example (I don't mean dub problems, which I agree exist, but the inflexibility in general)? I can't see a conflict between the two functionalities.
Typical example - we use custom Makefile based framework for internal Sociomantic D projects, one which is much more capable than existing dub build support and works decently for our needs overall. There is no way it can be replaced with dub at its current state. However using the very same dub to fetch sources of 3d-party libraries and ensure version compatibility is something quite desired. When build tool and package management system are packed into one tool, you pretty much need for both to be perfect to match everyones desires. And that is simply impossible.
Thanks for explanation. Then let me quote Teoh's great description of a practical build tool: "I think a good balance can be drawn between providing enough primitives that cover almost all conceivable use cases in a build tool, and at the same time provide an "escape hatch" into a full-fledged programming language for those rare but inevitable cases where you need to go outside the box and do what the designers didn't anticipate." IMO this applies to all of compiling, configuring and fetching sources or binaries and their dependencies. I'm aware it's not possible to cover all use cases. But 90-95% is achievable. The rest is relatively easily to be covered by custom d code. I started that Piotrek
Mar 19 2016
parent reply Dicebot <public dicebot.lv> writes:
On 03/19/2016 11:36 AM, Piotrek wrote:
 On Friday, 18 March 2016 at 15:31:26 UTC, Dicebot wrote:
 Hmm, the build module could be compiled once. It sources are supposed
 to stay unchanged, right?
Even "once" will be too much for majority of D users (those who are not also Gentoo users at least :D). Remember - we are not speaking about build as simple as `rdmd std/build.d` at this point, it will have numerous additional dependencies, including linker dependencies. And has to be perfectly cross-platform.
I think that std.build could be up to 10KLOC so the compilation time would be not relevant.
dub existing sources are 24KLOC and it doesn't even have a real build system. Compilation time is not a problem though. What is more important, it needs external dependencies like curl which make impossible to build on target platform transpatently via `rdmd std/build.d`. You'll need a build system to build your build system.
 When build tool and package management system are packed into one
 tool, you pretty much need for both to be perfect to match everyones
 desires. And that is simply impossible.
Thanks for explanation. Then let me quote Teoh's great description of a practical build tool: "I think a good balance can be drawn between providing enough primitives that cover almost all conceivable use cases in a build tool, and at the same time provide an "escape hatch" into a full-fledged programming language for those rare but inevitable cases where you need to go outside the box and do what the designers didn't anticipate." IMO this applies to all of compiling, configuring and fetching sources or binaries and their dependencies. I'm aware it's not possible to cover all use cases. But 90-95% is achievable. The rest is relatively easily to be covered by custom d code.
Even 90% is not enough because it leads to forking functionality for those 10%, greatly diminishing standartization. And build systems are highly opinionated. Some people praise imperative systems like SCons - I find it very hard to reason about compared to declarative ones (even compare to Makefiles). Some like magic deduction of dependencies, some stand hard by making stuff explicit. And list goes on. Most importantly, there is no need to make it one-thing-fits-them-all - building should be private affair of any given project and its developers. All you need is to define standard project description format that already built projects can be exposed with to other projects - and that is something dub.sdl / dub.json are already pretty good at.
 I started that
 
 Piotrek
Good luck with that anyway :)
Mar 19 2016
parent Piotrek <noaddr noaddr.ab> writes:
On Saturday, 19 March 2016 at 17:57:24 UTC, Dicebot wrote:
 Even 90% is not enough because it leads to forking 
 functionality for those 10%, greatly diminishing 
 standartization. And build systems are highly opinionated. Some 
 people praise imperative systems like SCons - I find it very 
 hard to reason about compared to declarative ones (even compare 
 to Makefiles). Some like magic deduction of dependencies, some 
 stand hard by making stuff explicit. And list goes on.
I agree. That's why I wa wondering if consensus is possible. For me declarative style is also the nicest when aplicable.
 I started that
 
 Piotrek
Good luck with that anyway :)
Sorry it was unfinished sentence :) I'm rather on recon stage. Piotrek
Mar 20 2016
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 17 March 2016 at 05:15:25 UTC, Piotrek wrote:
 On Wednesday, 16 March 2016 at 16:36:47 UTC, Dicebot wrote:
 NB: this is orthogonal to development of dub. Most important 
 functionality of dub is dependency management, acting as a 
 build tool is secondary to that (and can be adjusted to 
 support other build systems instead).

 Idea itself is good and this is something that reggae could do 
 but it isn't as easy project as it may sound.
I didn't know about reggae, looks interesting. However after brief review is seems to be overcomplicated and inconvenient in some cases.
Could you explain what is overcomplicated and inconvenient? I'd love some feedback and to be able to fix it.
 As for dub I don't think it is unrelated. Why std.build 
 couldn't be dependency manager?

 What about:
 std.build = part of reggae + part of dub
I don't think mixing up dependency management and build systems is a good idea. I made an explicit decision to defer all dependency management to dub. Atila
Mar 18 2016
parent reply Piotrek <starpit tlen.pl> writes:
On Friday, 18 March 2016 at 09:51:07 UTC, Atila Neves wrote:
 Could you explain what is overcomplicated and inconvenient? I'd 
 love some feedback and to be able to fix it.
This is rather broad topic and most of the points are related to different view on design goal for build tool. Let me try to list general comments (remember that they are not bugs but mostly consequences of chosen requirements) 1. Unnecessary reliance on external backends. 2. Not "slim" syntax I have similar view on the syntax as Dicebot: http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.org
 I don't think mixing up dependency management and build systems 
 is a good idea. I made an explicit decision to defer all 
 dependency management to dub.
Yeah, that's additional complexity IMO. I still don't see a good reason this has to be separated. Piotrek
Mar 19 2016
next sibling parent reply Piotrek <starpit tlen.pl> writes:
On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:
 2. Not "slim" syntax
 I have similar view on the syntax as Dicebot: 
 http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.org
But have to add that I want event simpler (no templates etc.) declarations and primitives like e.g. in SCons. Piotrek
Mar 19 2016
parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 19 March 2016 at 09:54:53 UTC, Piotrek wrote:
 On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:
 2. Not "slim" syntax
 I have similar view on the syntax as Dicebot: 
 http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.org
But have to add that I want event simpler (no templates etc.)
The reason why there are templates is so that the declarations can be in global scope, like a scripting language.
 declarations and primitives like e.g. in SCons.
You mean like `Program` and `Object`? The equivalent are `scriptlike` and `objectFiles`. There are several high-level rules as well. Atila
Mar 19 2016
parent reply Piotrek <noaddr noaddr.ab> writes:
On Saturday, 19 March 2016 at 14:20:23 UTC, Atila Neves wrote:
 On Saturday, 19 March 2016 at 09:54:53 UTC, Piotrek wrote:
 On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:
 2. Not "slim" syntax
 I have similar view on the syntax as Dicebot: 
 http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.org
But have to add that I want event simpler (no templates etc.)
The reason why there are templates is so that the declarations can be in global scope, like a scripting language.
 declarations and primitives like e.g. in SCons.
You mean like `Program` and `Object`? The equivalent are `scriptlike` and `objectFiles`. There are several high-level rules as well. Atila
I was thinking about simple declarative syntax plus fallback to imperative style for custom needs. I will try to give a feedback on reggae after I go further with experimenting. Piotrek
Mar 20 2016
parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 20 March 2016 at 07:35:07 UTC, Piotrek wrote:
 On Saturday, 19 March 2016 at 14:20:23 UTC, Atila Neves wrote:
 On Saturday, 19 March 2016 at 09:54:53 UTC, Piotrek wrote:
 On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:
 [...]
But have to add that I want event simpler (no templates etc.)
The reason why there are templates is so that the declarations can be in global scope, like a scripting language.
 declarations and primitives like e.g. in SCons.
You mean like `Program` and `Object`? The equivalent are `scriptlike` and `objectFiles`. There are several high-level rules as well. Atila
I was thinking about simple declarative syntax plus fallback to imperative style for custom needs.
That's exactly what I thought I'd accomplished ;)
 I will try to give a feedback on reggae after I go further with 
 experimenting.
Please do! Atila
Mar 20 2016
parent reply cym13 <cpicard openmailbox.org> writes:
On Sunday, 20 March 2016 at 11:27:11 UTC, Atila Neves wrote:
 On Sunday, 20 March 2016 at 07:35:07 UTC, Piotrek wrote:
 I was thinking about simple declarative syntax plus fallback to
 imperative style for custom needs.
That's exactly what I thought I'd accomplished ;)
 I will try to give a feedback on reggae after I go further 
 with experimenting.
Please do! Atila
Btw I like what you've done with reggae, it's not saluted enough. Thanks!
Mar 20 2016
parent Atila Neves <atila.neves gmail.com> writes:
On Sunday, 20 March 2016 at 15:33:16 UTC, cym13 wrote:
 On Sunday, 20 March 2016 at 11:27:11 UTC, Atila Neves wrote:
 On Sunday, 20 March 2016 at 07:35:07 UTC, Piotrek wrote:
 I was thinking about simple declarative syntax plus fallback 
 to
 imperative style for custom needs.
That's exactly what I thought I'd accomplished ;)
 I will try to give a feedback on reggae after I go further 
 with experimenting.
Please do! Atila
Btw I like what you've done with reggae, it's not saluted enough. Thanks!
Thanks! Atila
Mar 21 2016
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:
 On Friday, 18 March 2016 at 09:51:07 UTC, Atila Neves wrote:
 Could you explain what is overcomplicated and inconvenient? 
 I'd love some feedback and to be able to fix it.
This is rather broad topic and most of the points are related to different view on design goal for build tool. Let me try to list general comments (remember that they are not bugs but mostly consequences of chosen requirements) 1. Unnecessary reliance on external backends.
Not for a long time, there's a binary backend now. The only dependency is a D compiler.
 2. Not "slim" syntax
I don't know how it could be slimmer given what it does.
 I have similar view on the syntax as Dicebot: 
 http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.org
You can pass a function pointer as the command instead of just a string now. Atila
Mar 19 2016
prev sibling parent reply Mark Isaacson <turck11 hotmail.com> writes:
 From experience, it turns out that having a restricted language 
to specify your builds/dependencies is a very good thing. You 
really don't really want a turning complete language for this; it 
just makes it harder to reason about.
Mar 16 2016
next sibling parent reply Piotrek <starpit tlen.pl> writes:
On Wednesday, 16 March 2016 at 18:36:48 UTC, Mark Isaacson wrote:
 From experience, it turns out that having a restricted language 
 to specify your builds/dependencies is a very good thing. You 
 really don't really want a turning complete language for this; 
 it just makes it harder to reason about.
I agree that everybody has different experience. My, for example, is opposite. 1. Make and derivatives are unintuitive to use and lead to unmaintained mess. 2. Simple and descriptive formats like JSON means neverending demand for extension and alternatives (dub situation). Piotrek
Mar 16 2016
next sibling parent Timothee Cour via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, Mar 16, 2016 at 10:25 PM, Piotrek via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On Wednesday, 16 March 2016 at 18:36:48 UTC, Mark Isaacson wrote:

 From experience, it turns out that having a restricted language to
 specify your builds/dependencies is a very good thing. You really don't
 really want a turning complete language for this; it just makes it harder
 to reason about.
I agree that everybody has different experience. My, for example, is opposite. 1. Make and derivatives are unintuitive to use and lead to unmaintained mess. 2. Simple and descriptive formats like JSON means neverending demand for extension and alternatives (dub situation).
eg, comments (not supported in dub JSON format since it's not standard JSON)
 Piotrek
Mar 16 2016
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Mar 17, 2016 at 05:25:32AM +0000, Piotrek via Digitalmars-d wrote:
 On Wednesday, 16 March 2016 at 18:36:48 UTC, Mark Isaacson wrote:
From experience, it turns out that having a restricted language to
specify your builds/dependencies is a very good thing. You really
don't really want a turning complete language for this; it just makes
it harder to reason about.
I agree that everybody has different experience. My, for example, is opposite. 1. Make and derivatives are unintuitive to use and lead to unmaintained mess. 2. Simple and descriptive formats like JSON means neverending demand for extension and alternatives (dub situation).
[...] Personally, I also find makefiles have a tendency to become unmaintable messes. I have yet to find one non-trivial project whose makefiles *aren't* unmaintainable messes. The dmd toolchain tries to, but fails (esp. with the nasty cross-repo references that will basically fail in ugly ways in every case except when you setup your environment exactly like the makefiles implicitly assume -- the hallmark of design smell). My current go-to build system is SCons, a Python-based build system where build scripts are Python scripts. However, far from becoming a problem due to Turing-completeness, generally I hardly ever use the full expressive power of Python; the primitives provided take care of 99% of the most typical build scenarios. However, for those rare but unavoidable cases where you need something more, having a full programming language at your disposal is a life-saver. When the primitives provided by SCons can't quite express what you want, you can code the desired result by hand, and it integrates smoothly into the rest of the system. No need to plead for extensions and Yet More Complicated Conditional Clauses to handle your particular corner case, and potentially being forced to wait a whole release cycle to get what you need. I think a good balance can be drawn between providing enough primitives that cover almost all conceivable use cases in a build tool, and at the same time provide an "escape hatch" into a full-fledged programming language for those rare but inevitable cases where you need to go outside the box and do what the designers didn't anticipate. (Note that while SCons is pretty good in my book, it isn't perfect, and its dependency management algorithms are somewhat dated. Newer than makefiles, for sure, but not quite up to the standard of new, scalable build tools like tup, whose running time depends on changeset size, as opposed to repo size -- very important for minimizing the code-test-debug cycle times.) T -- If blunt statements had a point, they wouldn't be blunt...
Mar 16 2016
next sibling parent Jacob Carlborg <doob me.com> writes:
On 17/03/16 07:13, H. S. Teoh via Digitalmars-d wrote:

 Personally, I also find makefiles have a tendency to become unmaintable
 messes. I have yet to find one non-trivial project whose makefiles
 *aren't* unmaintainable messes.  The dmd toolchain tries to, but fails
 (esp. with the nasty cross-repo references that will basically fail in
 ugly ways in every case except when you setup your environment exactly
 like the makefiles implicitly assume -- the hallmark of design smell).

 My current go-to build system is SCons, a Python-based build system
 where build scripts are Python scripts. However, far from becoming a
 problem due to Turing-completeness, generally I hardly ever use the full
 expressive power of Python; the primitives provided take care of 99% of
 the most typical build scenarios. However, for those rare but
 unavoidable cases where you need something more, having a full
 programming language at your disposal is a life-saver.
I completely agree. It's the same with Ruby and especially if you include Rails. It has a lot of small DSL's that are all in Ruby. Many which might have been XML, JSON or similar in other language. Take for example a Gemfile (which lists dependencies of a project). The Gemfile is written in a DSL in Ruby. I can't remember the last time I saw a Gemfile with something other than the standard dependency statements: gem 'rails', '4.2.0' gem 'pry', '0.9.1' -- /Jacob Carlborg
Mar 17 2016
prev sibling parent Piotrek <starpit tlen.pl> writes:
On Thursday, 17 March 2016 at 06:13:48 UTC, H. S. Teoh wrote:
 I think a good balance can be drawn between providing enough 
 primitives that cover almost all conceivable use cases in a 
 build tool, and at the same time provide an "escape hatch" into 
 a full-fledged programming language for those rare but 
 inevitable cases where you need to go outside the box and do 
 what the designers didn't anticipate.
Fully agree. Do you think is it worth to have such a tool as std.build? Piotrek
Mar 17 2016
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 16 March 2016 at 18:36:48 UTC, Mark Isaacson wrote:
 From experience, it turns out that having a restricted language 
 to specify your builds/dependencies is a very good thing.
Yes, and it's called a DSL.
 You really don't really want a turning complete language for 
 this; it just makes it harder to reason about.
Not in my experience. Builds that are complicated enough need a proper language, and when you do, you'd better hope that whatever you're using isn't XML-based (looking at you Ant) or something like that. I've written quite a lot of non-trivial CMake code. The whole time I felt like I was bashing my head against the wall and everything would've been easier in Python, D, Ruby, whatever. A good build system description should be declarative as much as possible, but turing-complete when needed. Atila
Mar 18 2016