digitalmars.D.announce - Redub: A faster build system promising 90% compatibility with dub
Hello, people. Here I come announce a package I've just released: https://code.dlang.org/packages/redub) : A faster build system based on dub It is meant to be an almost drop-in replacement on dub. Right now, I've been implementing only features that dub already has. But in the future, I can't guarantee I'll keep following the same direction from dub since there is a bunch I disagree with. - This is not a *reggae* clone (it is a lot easier to use by the way). This is a dub that simply can build with parallelization. - It completely ignores optional dependencies - Version range won't be implemented, at least not by me It has a very raw package management (by using dub CLI) and semver matching. 1. People with a project with many dependencies: Those people will get a big speedup on recompilations since each library can be built independently (if they don't mess up with pre/post commands) 2. People which don't rely on version ranges: Personally, I have 0 interest on implementing that, if you wish support for it, issue a pull request. 3. People which wants rule-based software instead of exception based: The features are designed around rules instead of workarounds. If you need an exception in using it, I won't listen to this request or pull request. Only generalizations may be implemented. 4. People which wants better caching: Dub currently is completely buggy on caches with string import paths, (and actually, for some unknown reason, can't even detect up to date redub when running it. ) - dub recipe files are parsed lazily, which means, you can use inexistent dependencies inside other configurations, and also faster parsing even though I'm using std.json - The code is following a 95% functional style. Almost no state is changed in between compilation steps - This project is focused on readability and easy of maintenance: Don't try to be too smart, unless real difference can be made on its performance - Libraries are always built recursively. (Currently was only achieved with --deep flag). Which for me was simply a big "why" - The default output now is inside a separate folder: "bin". This makes it easier for making the caching system to behave better since they will be check on other root. - I won't focus on any moment into implementing support for a single file build, this is too much work for no real gain, if you want to do it, do it on a completely isolated file. - This software was built in 2.5 weeks, which first was a 1 week project, but, it was so successful in what it done that I decided to expand its scope (I would initially only support what Hipreme Engine needs). Some people said that no one done better than dub until now, so, I've done this project because it was simply unbelievable that dub wasn't able to do any level of parallelization. - I have tried contributing to dub's project on parallelization, I waste 1 week trying that and could not get it working, so, I decided it would be a better use of my time into rewriting a completely new dub which would focus on being easy to read -- An example of that is how easily someone was able to integrate an experimental support to building C with it. This project has been tested with - [Hipreme Engine](https://code.dlang.org/packages/hipreme_engine) - [Fluid](https://code.dlang.org/packages/fluid) - [arsd](https://code.dlang.org/packages/arsd-official) - [dplug](https://code.dlang.org/packages/dplug) And possibly more to come. I didn't done a complete vibe-d support because there was too many features which I am not going to use (hello arch-based filters on dub.json). - A full rebuild on Hipreme Engine reduced its compilation time from 14 seconds to 3. One could say that full rebuilds aren't common, but this is untrue. Specially when I was developing my own standard library (util), this would get into my nerves. You may test it with `dub run redub`. But since `dub` is buggy, it will always be identified as not up to date :) By the way: People which hates `dub` and use `dmd -i` instead: your project ought to be built faster by modularizing them :)
Jan 20
On Saturday, 20 January 2024 at 14:14:10 UTC, Hipreme wrote:- I have tried contributing to dub's project on parallelization, I waste 1 week trying that and could not get it working, so, I decided it would be a better use of my time into rewriting a completely new dub which would focus on being easy to read -- An example of that is how easily someone was able to integrate an experimental support to building C with it.Could you expand on that? Perhaps someone else could continue your work. While I understand you preferred to write your own system (we all probably prefer greenfield development, after all that means it's fully yours, and you can fully understand it, own it, and change it as you see fit which is a very attractive proposition), I'm afraid your project will only be alive while you have interest in maintaining it (I've seen similar projects in other languages before, often they have one or two releases before the author moves on to the next greenfield project) - unless you manage to create an active community around it, which is exceedingly difficult and might be more than you're signing up for once a lot of people join in and start demanding features - while dub, even if it's so "buggy" as you claim, is the official tool everyone is likely to continue to use, and consequently improvements to it would be highly welcome.
Jan 20
Hipreme <msnmancini hotmail.com> On Saturday, 20 January 2024 at 15:25:35 UTC, Renato wrote:On Saturday, 20 January 2024 at 14:14:10 UTC, Hipreme wrote:I don't care if people actually use or not. Dub is not easy to understand, not easy to maintain, it is less performant than redub and the main reason for redub even exist was because my project (Hipreme Engine) needed a better solution. Reggae is also an official tool which no one uses. RDMD is an official solution that should be deprecated since the existence of RUND. Also, contributing to an official project would only slow down my progress, which for me, is not negotiable. There are a bunch of things which I've done because D doesn't move at the same speed as me, such as: - My WASM runtime - Make the runtime generic enough to support an unrelated platform (PSVita) - My solution to cross compilation which is able to automatically identify what is missing on an user setup for being able to release to non standard D platforms. - redub (the slow release of the flags `--deep` and `--recipe` made me stall for quite a lot more of time than I expected), the fully stateful and confuse code only made a lot of behaviors in dub that does not scale well. - My own standard library which does not depend on C runtime, which makes it a lot more portable since it becomes D-only code. - The existence of `objc_meta` which gave an easy and performant solution for calling Objective-C functions. Dub needs a rewrite, but it can't change, because they are caring about breaking dub as a library, even though versioning exists only for that. I have a lot to work right now and can't give more time to things that aren't an issue for me. Nonetheless, if this project suits you well, there's no reason not to use it.- I have tried contributing to dub's project on parallelization, I waste 1 week trying that and could not get it working, so, I decided it would be a better use of my time into rewriting a completely new dub which would focus on being easy to read -- An example of that is how easily someone was able to integrate an experimental support to building C with it.Could you expand on that? Perhaps someone else could continue your work. While I understand you preferred to write your own system (we all probably prefer greenfield development, after all that means it's fully yours, and you can fully understand it, own it, and change it as you see fit which is a very attractive proposition), I'm afraid your project will only be alive while you have interest in maintaining it (I've seen similar projects in other languages before, often they have one or two releases before the author moves on to the next greenfield project) - unless you manage to create an active community around it, which is exceedingly difficult and might be more than you're signing up for once a lot of people join in and start demanding features - while dub, even if it's so "buggy" as you claim, is the official tool everyone is likely to continue to use, and consequently improvements to it would be highly welcome.
Jan 20
On Saturday, 20 January 2024 at 15:25:35 UTC, Renato wrote:On Saturday, 20 January 2024 at 14:14:10 UTC, Hipreme wrote:This quote says it all:- I have tried contributing to dub's project on parallelization, I waste 1 week trying that and could not get it working, so, I decided it would be a better use of my time into rewriting a completely new dub which would focus on being easy to read -- An example of that is how easily someone was able to integrate an experimental support to building C with it.Could you expand on that? Perhaps someone else could continue your work. While I understand you preferred to write your own system (we all probably prefer greenfield development, after all that means it's fully yours, and you can fully understand it, own it, and change it as you see fit which is a very attractive proposition), I'm afraid your project will only be alive while you have interest in maintaining it (I've seen similar projects in other languages before, often they have one or two releases before the author moves on to the next greenfield project) - unless you manage to create an active community around it, which is exceedingly difficult and might be more than you're signing up for once a lot of people join in and start demanding features - while dub, even if it's so "buggy" as you claim, is the official tool everyone is likely to continue to use, and consequently improvements to it would be highly welcome.A full rebuild on Hipreme Engine reduced its compilation time from 14 seconds to 3dub is over engineered We should encourage more of these projects, they defy the stalled status quo in D
Jan 20
On Saturday, 20 January 2024 at 16:03:52 UTC, ryuukk_ wrote:On Saturday, 20 January 2024 at 15:25:35 UTC, Renato wrote:Just came to announce that I've finished implementing redub as a library, present on version 1.2.1: ```d import redub.api; import redub.logging; void main() { import std.file; setLogLevel(LogLevel.verbose); ProjectDetails d = resolveDependencies( invalidateCache: false, std.system.os, CompilationDetails("dmd", "arch not yet implemented", "dmd v[2.105.0]"), ProjectToParse("configuration", getcwd(), "subPackage", "path/to/dub/recipe.json (optional)") ); buildProject(d); } ``` Beyond that, I've also done 2 new optimizations: 1. Fixed a problem where memory was being allocated in a non essential task, causing some GC pauses while resolving dependencies. This reduced dependency resolution time to 1/4. Hipreme Engine went from 200ms to 50m, now the bottleneck being mostly `std.json`, but I don't think it is worth changing it. 2. Added support for compiler version assumption. This makes it even faster since it won't need to keep rerunning the `compiler --version` each time. This is a minor optimization that got 30 ms faster (on Windows).On Saturday, 20 January 2024 at 14:14:10 UTC, Hipreme wrote:This quote says it all:- I have tried contributing to dub's project on parallelization, I waste 1 week trying that and could not get it working, so, I decided it would be a better use of my time into rewriting a completely new dub which would focus on being easy to read -- An example of that is how easily someone was able to integrate an experimental support to building C with it.Could you expand on that? Perhaps someone else could continue your work. While I understand you preferred to write your own system (we all probably prefer greenfield development, after all that means it's fully yours, and you can fully understand it, own it, and change it as you see fit which is a very attractive proposition), I'm afraid your project will only be alive while you have interest in maintaining it (I've seen similar projects in other languages before, often they have one or two releases before the author moves on to the next greenfield project) - unless you manage to create an active community around it, which is exceedingly difficult and might be more than you're signing up for once a lot of people join in and start demanding features - while dub, even if it's so "buggy" as you claim, is the official tool everyone is likely to continue to use, and consequently improvements to it would be highly welcome.A full rebuild on Hipreme Engine reduced its compilation time from 14 seconds to 3dub is over engineered We should encourage more of these projects, they defy the stalled status quo in D
Jan 22