digitalmars.D - Idea: std.build instead of dub and make-like tools
- Piotrek (9/9) Mar 16 2016 Hi,
- ZombineDev (8/17) Mar 16 2016 Related:
- Dicebot (7/21) Mar 16 2016 NB: this is orthogonal to development of dub. Most important
- Piotrek (9/15) Mar 16 2016 I didn't know about reggae, looks interesting. However after
- Dicebot (13/17) Mar 17 2016 For same reason you don't want to distribute any other non-trivial tools
- Piotrek (7/18) Mar 17 2016 Hmm, the build module could be compiled once. It sources are
- Dicebot (15/35) Mar 18 2016 Even "once" will be too much for majority of D users (those who are not
- Atila Neves (7/22) Mar 18 2016 Could you explain what is overcomplicated and inconvenient? I'd
- Piotrek (12/17) Mar 19 2016 This is rather broad topic and most of the points are related to
- Piotrek (4/7) Mar 19 2016 But have to add that I want event simpler (no templates etc.)
- Atila Neves (7/13) Mar 19 2016 The reason why there are templates is so that the declarations
- Piotrek (6/20) Mar 20 2016 I was thinking about simple declarative syntax plus fallback to
- Atila Neves (4/25) Mar 20 2016 Please do!
- cym13 (3/11) Mar 20 2016 Btw I like what you've done with reggae, it's not saluted enough.
- Atila Neves (3/19) Mar 21 2016 Thanks!
- Atila Neves (7/18) Mar 19 2016 Not for a long time, there's a binary backend now. The only
- Mark Isaacson (4/4) Mar 16 2016 From experience, it turns out that having a restricted language
- Piotrek (8/12) Mar 16 2016 I agree that everybody has different experience. My, for example,
- Timothee Cour via Digitalmars-d (3/16) Mar 16 2016 eg, comments (not supported in dub JSON format since it's not standard J...
- H. S. Teoh via Digitalmars-d (35/48) Mar 16 2016 [...]
- Jacob Carlborg (11/24) Mar 17 2016 I completely agree. It's the same with Ruby and especially if you
- Piotrek (4/10) Mar 17 2016 Fully agree. Do you think is it worth to have such a tool as
- Atila Neves (12/16) Mar 18 2016 Not in my experience. Builds that are complicated enough need a
Hi, What do you think about concentrating D build system around a hypothetical "std.build" module instead of investing in dub or other custom tools? Also instead of custom build file format like JSON/SDL/XML/YAML we could simply use a d source file, e.g "build.d". All specification would be included in "std.build". Did I miss any blocking point? Piotrek
Mar 16 2016
On Wednesday, 16 March 2016 at 15:51:38 UTC, Piotrek wrote:Hi, What do you think about concentrating D build system around a hypothetical "std.build" module instead of investing in dub or other custom tools? Also instead of custom build file format like JSON/SDL/XML/YAML we could simply use a d source file, e.g "build.d". All specification would be included in "std.build". Did I miss any blocking point? PiotrekRelated: Reggae - D meta-build system: http://forum.dlang.org/thread/cycyjqdcsdbnkrbziqgl forum.dlang.org http://forum.dlang.org/post/ranqlmrjornlvopsuris forum.dlang.org http://forum.dlang.org/thread/vwkzgbnxvnteefldlvfi forum.dlang.org http://forum.dlang.org/post/znaffgxodfpnjwggpwok forum.dlang.org http://forum.dlang.org/thread/zrguosahjpftxebaqtxp forum.dlang.org
Mar 16 2016
On 03/16/2016 05:51 PM, Piotrek wrote:Hi, What do you think about concentrating D build system around a hypothetical "std.build" module instead of investing in dub or other custom tools? Also instead of custom build file format like JSON/SDL/XML/YAML we could simply use a d source file, e.g "build.d". All specification would be included in "std.build". Did I miss any blocking point? PiotrekNB: this is orthogonal to development of dub. Most important functionality of dub is dependency management, acting as a build tool is secondary to that (and can be adjusted to support other build systems instead). Idea itself is good and this is something that reggae could do but it isn't as easy project as it may sound.
Mar 16 2016
On Wednesday, 16 March 2016 at 16:36:47 UTC, Dicebot wrote:NB: this is orthogonal to development of dub. Most important functionality of dub is dependency management, acting as a build tool is secondary to that (and can be adjusted to support other build systems instead). Idea itself is good and this is something that reggae could do but it isn't as easy project as it may sound.I didn't know about reggae, looks interesting. However after brief review is seems to be overcomplicated and inconvenient in some cases. As for dub I don't think it is unrelated. Why std.build couldn't be dependency manager? What about: std.build = part of reggae + part of dub Piotrek
Mar 16 2016
On 03/17/2016 07:15 AM, Piotrek wrote:As for dub I don't think it is unrelated. Why std.build couldn't be dependency manager?For same reason you don't want to distribute any other non-trivial tools as sources :) Compilation takes time and has non-trivial dependencies (i.e. networking libraries, git providers etc.), you simply can't put that stuff as a stdlib module/package and expect developers to compile it each time. Even mature build tool would be too complicated to fit into std.build concept - however there is a real benefit in supporting more simple build system that doesn't try to be performant and robust, as that can be used to bootstrap building compiler distribution itself.What about: std.build = part of reggae + part of dubTight coupling of dependency management and build tool in one entity is just too inflexible. This is single biggest issue I have with dub in its current form.
Mar 17 2016
On Thursday, 17 March 2016 at 15:49:07 UTC, Dicebot wrote:On 03/17/2016 07:15 AM, Piotrek wrote:Hmm, the build module could be compiled once. It sources are supposed to stay unchanged, right?As for dub I don't think it is unrelated. Why std.build couldn't be dependency manager?For same reason you don't want to distribute any other non-trivial tools as sources :) Compilation takes time and has non-trivial dependencies (i.e. networking libraries, git providers etc.), you simply can't put that stuff as a stdlib module/package and expect developers to compile it each time.Tight coupling of dependency management and build tool in one entity is just too inflexible. This is single biggest issue I have with dub in its current form.Can you explain it by example (I don't mean dub problems, which I agree exist, but the inflexibility in general)? I can't see a conflict between the two functionalities. Piotrek
Mar 17 2016
On 03/17/2016 06:42 PM, Piotrek wrote:On Thursday, 17 March 2016 at 15:49:07 UTC, Dicebot wrote:Even "once" will be too much for majority of D users (those who are not also Gentoo users at least :D). Remember - we are not speaking about build as simple as `rdmd std/build.d` at this point, it will have numerous additional dependencies, including linker dependencies. And has to be perfectly cross-platform.On 03/17/2016 07:15 AM, Piotrek wrote:Hmm, the build module could be compiled once. It sources are supposed to stay unchanged, right?As for dub I don't think it is unrelated. Why std.build couldn't be dependency manager?For same reason you don't want to distribute any other non-trivial tools as sources :) Compilation takes time and has non-trivial dependencies (i.e. networking libraries, git providers etc.), you simply can't put that stuff as a stdlib module/package and expect developers to compile it each time.Typical example - we use custom Makefile based framework for internal Sociomantic D projects, one which is much more capable than existing dub build support and works decently for our needs overall. There is no way it can be replaced with dub at its current state. However using the very same dub to fetch sources of 3d-party libraries and ensure version compatibility is something quite desired. When build tool and package management system are packed into one tool, you pretty much need for both to be perfect to match everyones desires. And that is simply impossible.Tight coupling of dependency management and build tool in one entity is just too inflexible. This is single biggest issue I have with dub in its current form.Can you explain it by example (I don't mean dub problems, which I agree exist, but the inflexibility in general)? I can't see a conflict between the two functionalities.
Mar 18 2016
On Friday, 18 March 2016 at 15:31:26 UTC, Dicebot wrote:I think that std.build could be up to 10KLOC so the compilation time would be not relevant.Hmm, the build module could be compiled once. It sources are supposed to stay unchanged, right?Even "once" will be too much for majority of D users (those who are not also Gentoo users at least :D). Remember - we are not speaking about build as simple as `rdmd std/build.d` at this point, it will have numerous additional dependencies, including linker dependencies. And has to be perfectly cross-platform.Thanks for explanation. Then let me quote Teoh's great description of a practical build tool: "I think a good balance can be drawn between providing enough primitives that cover almost all conceivable use cases in a build tool, and at the same time provide an "escape hatch" into a full-fledged programming language for those rare but inevitable cases where you need to go outside the box and do what the designers didn't anticipate." IMO this applies to all of compiling, configuring and fetching sources or binaries and their dependencies. I'm aware it's not possible to cover all use cases. But 90-95% is achievable. The rest is relatively easily to be covered by custom d code. I started that PiotrekTypical example - we use custom Makefile based framework for internal Sociomantic D projects, one which is much more capable than existing dub build support and works decently for our needs overall. There is no way it can be replaced with dub at its current state. However using the very same dub to fetch sources of 3d-party libraries and ensure version compatibility is something quite desired. When build tool and package management system are packed into one tool, you pretty much need for both to be perfect to match everyones desires. And that is simply impossible.Tight coupling of dependency management and build tool in one entity is just too inflexible. This is single biggest issue I have with dub in its current form.Can you explain it by example (I don't mean dub problems, which I agree exist, but the inflexibility in general)? I can't see a conflict between the two functionalities.
Mar 19 2016
On 03/19/2016 11:36 AM, Piotrek wrote:On Friday, 18 March 2016 at 15:31:26 UTC, Dicebot wrote:dub existing sources are 24KLOC and it doesn't even have a real build system. Compilation time is not a problem though. What is more important, it needs external dependencies like curl which make impossible to build on target platform transpatently via `rdmd std/build.d`. You'll need a build system to build your build system.I think that std.build could be up to 10KLOC so the compilation time would be not relevant.Hmm, the build module could be compiled once. It sources are supposed to stay unchanged, right?Even "once" will be too much for majority of D users (those who are not also Gentoo users at least :D). Remember - we are not speaking about build as simple as `rdmd std/build.d` at this point, it will have numerous additional dependencies, including linker dependencies. And has to be perfectly cross-platform.Even 90% is not enough because it leads to forking functionality for those 10%, greatly diminishing standartization. And build systems are highly opinionated. Some people praise imperative systems like SCons - I find it very hard to reason about compared to declarative ones (even compare to Makefiles). Some like magic deduction of dependencies, some stand hard by making stuff explicit. And list goes on. Most importantly, there is no need to make it one-thing-fits-them-all - building should be private affair of any given project and its developers. All you need is to define standard project description format that already built projects can be exposed with to other projects - and that is something dub.sdl / dub.json are already pretty good at.When build tool and package management system are packed into one tool, you pretty much need for both to be perfect to match everyones desires. And that is simply impossible.Thanks for explanation. Then let me quote Teoh's great description of a practical build tool: "I think a good balance can be drawn between providing enough primitives that cover almost all conceivable use cases in a build tool, and at the same time provide an "escape hatch" into a full-fledged programming language for those rare but inevitable cases where you need to go outside the box and do what the designers didn't anticipate." IMO this applies to all of compiling, configuring and fetching sources or binaries and their dependencies. I'm aware it's not possible to cover all use cases. But 90-95% is achievable. The rest is relatively easily to be covered by custom d code.I started that PiotrekGood luck with that anyway :)
Mar 19 2016
On Saturday, 19 March 2016 at 17:57:24 UTC, Dicebot wrote:Even 90% is not enough because it leads to forking functionality for those 10%, greatly diminishing standartization. And build systems are highly opinionated. Some people praise imperative systems like SCons - I find it very hard to reason about compared to declarative ones (even compare to Makefiles). Some like magic deduction of dependencies, some stand hard by making stuff explicit. And list goes on.I agree. That's why I wa wondering if consensus is possible. For me declarative style is also the nicest when aplicable.Sorry it was unfinished sentence :) I'm rather on recon stage. PiotrekI started that PiotrekGood luck with that anyway :)
Mar 20 2016
On Thursday, 17 March 2016 at 05:15:25 UTC, Piotrek wrote:On Wednesday, 16 March 2016 at 16:36:47 UTC, Dicebot wrote:Could you explain what is overcomplicated and inconvenient? I'd love some feedback and to be able to fix it.NB: this is orthogonal to development of dub. Most important functionality of dub is dependency management, acting as a build tool is secondary to that (and can be adjusted to support other build systems instead). Idea itself is good and this is something that reggae could do but it isn't as easy project as it may sound.I didn't know about reggae, looks interesting. However after brief review is seems to be overcomplicated and inconvenient in some cases.As for dub I don't think it is unrelated. Why std.build couldn't be dependency manager? What about: std.build = part of reggae + part of dubI don't think mixing up dependency management and build systems is a good idea. I made an explicit decision to defer all dependency management to dub. Atila
Mar 18 2016
On Friday, 18 March 2016 at 09:51:07 UTC, Atila Neves wrote:Could you explain what is overcomplicated and inconvenient? I'd love some feedback and to be able to fix it.This is rather broad topic and most of the points are related to different view on design goal for build tool. Let me try to list general comments (remember that they are not bugs but mostly consequences of chosen requirements) 1. Unnecessary reliance on external backends. 2. Not "slim" syntax I have similar view on the syntax as Dicebot: http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.orgI don't think mixing up dependency management and build systems is a good idea. I made an explicit decision to defer all dependency management to dub.Yeah, that's additional complexity IMO. I still don't see a good reason this has to be separated. Piotrek
Mar 19 2016
On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:2. Not "slim" syntax I have similar view on the syntax as Dicebot: http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.orgBut have to add that I want event simpler (no templates etc.) declarations and primitives like e.g. in SCons. Piotrek
Mar 19 2016
On Saturday, 19 March 2016 at 09:54:53 UTC, Piotrek wrote:On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:The reason why there are templates is so that the declarations can be in global scope, like a scripting language.2. Not "slim" syntax I have similar view on the syntax as Dicebot: http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.orgBut have to add that I want event simpler (no templates etc.)declarations and primitives like e.g. in SCons.You mean like `Program` and `Object`? The equivalent are `scriptlike` and `objectFiles`. There are several high-level rules as well. Atila
Mar 19 2016
On Saturday, 19 March 2016 at 14:20:23 UTC, Atila Neves wrote:On Saturday, 19 March 2016 at 09:54:53 UTC, Piotrek wrote:I was thinking about simple declarative syntax plus fallback to imperative style for custom needs. I will try to give a feedback on reggae after I go further with experimenting. PiotrekOn Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:The reason why there are templates is so that the declarations can be in global scope, like a scripting language.2. Not "slim" syntax I have similar view on the syntax as Dicebot: http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.orgBut have to add that I want event simpler (no templates etc.)declarations and primitives like e.g. in SCons.You mean like `Program` and `Object`? The equivalent are `scriptlike` and `objectFiles`. There are several high-level rules as well. Atila
Mar 20 2016
On Sunday, 20 March 2016 at 07:35:07 UTC, Piotrek wrote:On Saturday, 19 March 2016 at 14:20:23 UTC, Atila Neves wrote:That's exactly what I thought I'd accomplished ;)On Saturday, 19 March 2016 at 09:54:53 UTC, Piotrek wrote:I was thinking about simple declarative syntax plus fallback to imperative style for custom needs.On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:The reason why there are templates is so that the declarations can be in global scope, like a scripting language.[...]But have to add that I want event simpler (no templates etc.)declarations and primitives like e.g. in SCons.You mean like `Program` and `Object`? The equivalent are `scriptlike` and `objectFiles`. There are several high-level rules as well. AtilaI will try to give a feedback on reggae after I go further with experimenting.Please do! Atila
Mar 20 2016
On Sunday, 20 March 2016 at 11:27:11 UTC, Atila Neves wrote:On Sunday, 20 March 2016 at 07:35:07 UTC, Piotrek wrote:Btw I like what you've done with reggae, it's not saluted enough. Thanks!I was thinking about simple declarative syntax plus fallback to imperative style for custom needs.That's exactly what I thought I'd accomplished ;)I will try to give a feedback on reggae after I go further with experimenting.Please do! Atila
Mar 20 2016
On Sunday, 20 March 2016 at 15:33:16 UTC, cym13 wrote:On Sunday, 20 March 2016 at 11:27:11 UTC, Atila Neves wrote:Thanks! AtilaOn Sunday, 20 March 2016 at 07:35:07 UTC, Piotrek wrote:Btw I like what you've done with reggae, it's not saluted enough. Thanks!I was thinking about simple declarative syntax plus fallback to imperative style for custom needs.That's exactly what I thought I'd accomplished ;)I will try to give a feedback on reggae after I go further with experimenting.Please do! Atila
Mar 21 2016
On Saturday, 19 March 2016 at 09:51:03 UTC, Piotrek wrote:On Friday, 18 March 2016 at 09:51:07 UTC, Atila Neves wrote:Not for a long time, there's a binary backend now. The only dependency is a D compiler.Could you explain what is overcomplicated and inconvenient? I'd love some feedback and to be able to fix it.This is rather broad topic and most of the points are related to different view on design goal for build tool. Let me try to list general comments (remember that they are not bugs but mostly consequences of chosen requirements) 1. Unnecessary reliance on external backends.2. Not "slim" syntaxI don't know how it could be slimmer given what it does.I have similar view on the syntax as Dicebot: http://forum.dlang.org/post/vqdhbplqezgdmgumfxte forum.dlang.orgYou can pass a function pointer as the command instead of just a string now. Atila
Mar 19 2016
From experience, it turns out that having a restricted language to specify your builds/dependencies is a very good thing. You really don't really want a turning complete language for this; it just makes it harder to reason about.
Mar 16 2016
On Wednesday, 16 March 2016 at 18:36:48 UTC, Mark Isaacson wrote:From experience, it turns out that having a restricted language to specify your builds/dependencies is a very good thing. You really don't really want a turning complete language for this; it just makes it harder to reason about.I agree that everybody has different experience. My, for example, is opposite. 1. Make and derivatives are unintuitive to use and lead to unmaintained mess. 2. Simple and descriptive formats like JSON means neverending demand for extension and alternatives (dub situation). Piotrek
Mar 16 2016
On Wed, Mar 16, 2016 at 10:25 PM, Piotrek via Digitalmars-d < digitalmars-d puremagic.com> wrote:On Wednesday, 16 March 2016 at 18:36:48 UTC, Mark Isaacson wrote:eg, comments (not supported in dub JSON format since it's not standard JSON)From experience, it turns out that having a restricted language to specify your builds/dependencies is a very good thing. You really don't really want a turning complete language for this; it just makes it harder to reason about.I agree that everybody has different experience. My, for example, is opposite. 1. Make and derivatives are unintuitive to use and lead to unmaintained mess. 2. Simple and descriptive formats like JSON means neverending demand for extension and alternatives (dub situation).Piotrek
Mar 16 2016
On Thu, Mar 17, 2016 at 05:25:32AM +0000, Piotrek via Digitalmars-d wrote:On Wednesday, 16 March 2016 at 18:36:48 UTC, Mark Isaacson wrote:[...] Personally, I also find makefiles have a tendency to become unmaintable messes. I have yet to find one non-trivial project whose makefiles *aren't* unmaintainable messes. The dmd toolchain tries to, but fails (esp. with the nasty cross-repo references that will basically fail in ugly ways in every case except when you setup your environment exactly like the makefiles implicitly assume -- the hallmark of design smell). My current go-to build system is SCons, a Python-based build system where build scripts are Python scripts. However, far from becoming a problem due to Turing-completeness, generally I hardly ever use the full expressive power of Python; the primitives provided take care of 99% of the most typical build scenarios. However, for those rare but unavoidable cases where you need something more, having a full programming language at your disposal is a life-saver. When the primitives provided by SCons can't quite express what you want, you can code the desired result by hand, and it integrates smoothly into the rest of the system. No need to plead for extensions and Yet More Complicated Conditional Clauses to handle your particular corner case, and potentially being forced to wait a whole release cycle to get what you need. I think a good balance can be drawn between providing enough primitives that cover almost all conceivable use cases in a build tool, and at the same time provide an "escape hatch" into a full-fledged programming language for those rare but inevitable cases where you need to go outside the box and do what the designers didn't anticipate. (Note that while SCons is pretty good in my book, it isn't perfect, and its dependency management algorithms are somewhat dated. Newer than makefiles, for sure, but not quite up to the standard of new, scalable build tools like tup, whose running time depends on changeset size, as opposed to repo size -- very important for minimizing the code-test-debug cycle times.) T -- If blunt statements had a point, they wouldn't be blunt...From experience, it turns out that having a restricted language to specify your builds/dependencies is a very good thing. You really don't really want a turning complete language for this; it just makes it harder to reason about.I agree that everybody has different experience. My, for example, is opposite. 1. Make and derivatives are unintuitive to use and lead to unmaintained mess. 2. Simple and descriptive formats like JSON means neverending demand for extension and alternatives (dub situation).
Mar 16 2016
On 17/03/16 07:13, H. S. Teoh via Digitalmars-d wrote:Personally, I also find makefiles have a tendency to become unmaintable messes. I have yet to find one non-trivial project whose makefiles *aren't* unmaintainable messes. The dmd toolchain tries to, but fails (esp. with the nasty cross-repo references that will basically fail in ugly ways in every case except when you setup your environment exactly like the makefiles implicitly assume -- the hallmark of design smell). My current go-to build system is SCons, a Python-based build system where build scripts are Python scripts. However, far from becoming a problem due to Turing-completeness, generally I hardly ever use the full expressive power of Python; the primitives provided take care of 99% of the most typical build scenarios. However, for those rare but unavoidable cases where you need something more, having a full programming language at your disposal is a life-saver.I completely agree. It's the same with Ruby and especially if you include Rails. It has a lot of small DSL's that are all in Ruby. Many which might have been XML, JSON or similar in other language. Take for example a Gemfile (which lists dependencies of a project). The Gemfile is written in a DSL in Ruby. I can't remember the last time I saw a Gemfile with something other than the standard dependency statements: gem 'rails', '4.2.0' gem 'pry', '0.9.1' -- /Jacob Carlborg
Mar 17 2016
On Thursday, 17 March 2016 at 06:13:48 UTC, H. S. Teoh wrote:I think a good balance can be drawn between providing enough primitives that cover almost all conceivable use cases in a build tool, and at the same time provide an "escape hatch" into a full-fledged programming language for those rare but inevitable cases where you need to go outside the box and do what the designers didn't anticipate.Fully agree. Do you think is it worth to have such a tool as std.build? Piotrek
Mar 17 2016
On Wednesday, 16 March 2016 at 18:36:48 UTC, Mark Isaacson wrote:From experience, it turns out that having a restricted language to specify your builds/dependencies is a very good thing.Yes, and it's called a DSL.You really don't really want a turning complete language for this; it just makes it harder to reason about.Not in my experience. Builds that are complicated enough need a proper language, and when you do, you'd better hope that whatever you're using isn't XML-based (looking at you Ant) or something like that. I've written quite a lot of non-trivial CMake code. The whole time I felt like I was bashing my head against the wall and everything would've been easier in Python, D, Ruby, whatever. A good build system description should be declarative as much as possible, but turing-complete when needed. Atila
Mar 18 2016