## digitalmars.D - DUB - call to arms

• Seb (77/77) Apr 14 2019 Hi all,
• Guillaume Piolat (4/7) Apr 14 2019 If think DUB is completely _unsound_ and need a rewrite from
• Andre Pany (9/16) Apr 14 2019 Dub has a working eco system and a lot of developers put effort
• Guillaume Piolat (17/23) Apr 14 2019 Don't mistake me, DUB is a good software that is very useful and
• Guillaume Piolat (4/6) Apr 14 2019 With that said, I think I have been unreasonable, would
• Atila Neves (8/35) Apr 15 2019 +1 to all your points, including your previous post on this
• JN (19/25) Apr 14 2019 I think on the ecosystem side, dub is the #1 thing that ever came
• Zekereth (4/17) Apr 15 2019 I just want to mirror this exactly. I didn't really play with D
• Guillaume Piolat (5/7) Apr 15 2019 +1
• Jacob Shtokolov (9/12) Apr 14 2019 Hey guys, maybe this is an offtopic (which could sound heretical
• Andre Pany (10/24) Apr 14 2019 In addition to the Dub executable, you can also use Dub as
• Tourist (3/17) Apr 14 2019 We are too small a community so can you please do this for us?
• bachmeier (9/23) Apr 15 2019 I wonder what would possibly be gained from using Python or
• Andre Pany (15/19) Apr 14 2019 I fully agree.
• Adam D. Ruppe (21/24) Apr 14 2019 No and no. Dub delivers negative value for my use-case; it is a
• Russel Winder (32/56) Apr 15 2019 [=E2=80=A6]
• Adam D. Ruppe (13/17) Apr 15 2019 You can still have that (and more of it, as we can get more
• Russel Winder (20/43) Apr 16 2019 To be honest this is all implementation detail about which most users
• Russel Winder (32/36) Apr 15 2019 [=E2=80=A6]
• Anton Fediushin (46/46) Apr 15 2019 I don't have much experience contributing to dub but I have
• Andre Pany (7/13) Apr 15 2019 What issues do you have specific to Dub?
• Anton Fediushin (19/33) Apr 15 2019 It is slowing down the d community because it's not what a
• Andre Pany (12/49) Apr 15 2019 Integration of dub with LDC is working fine. I created a tutorial
• Atila Neves (5/15) Apr 15 2019 My experience matches Anton's - as soon as you try to do anything
• Andre Pany (10/30) Apr 15 2019 Actually I have a bat script for the windows release pipeline and
• Nick Sabalausky (Abscissa) (3/7) Apr 15 2019 We can't really say that replacing dub would be a pain without looking
• H. S. Teoh (8/15) Apr 16 2019 As long as dub's current functionality is supported and remains the
• Atila Neves (7/38) Apr 16 2019 Of course! For trivial applications that want to use dependencies
• aberba (4/10) Apr 25 2019 Dub works great for me too. I could use some build speed though.
• H. S. Teoh (20/22) Apr 25 2019 Because I tried using it and discovered that it wouldn't let me do what
• Atila Neves (7/19) Apr 26 2019 As I've said on more than one occasion, for simple usage dub
• Andre Pany (13/34) Apr 26 2019 I fully agree with you on the current state while I have another
• Adam D. Ruppe (4/8) Apr 15 2019 Yeah, I think the code.dlang.org site is salvageable, but I am
• Nick Sabalausky (Abscissa) (15/15) Apr 15 2019 I've personally put a lot of effort into trying to fix DUB's problems in...
• Guillaume Piolat (14/32) Apr 15 2019 I've tried paying MrSmith33 to write a DUB replacement for a
• Nick Sabalausky (Abscissa) (8/25) Apr 15 2019 To clarify, I'm mainly referring to the "package manager" aspect of DUB
• H. S. Teoh (30/37) Apr 15 2019 Y'know, the idea just occurred to me: why even bind the package manager
• Nick Sabalausky (Abscissa) (25/44) Apr 15 2019 Yup, from the sound of it, I think we're on exactly the same page here.
• rikki cattermole (4/8) Apr 15 2019 I can understand that argument. Regarding targets same compiler ext.
• H. S. Teoh (59/83) Apr 16 2019 I'd add:
• drug (2/5) Apr 17 2019 How do you resolve conflicts using DAG only?
• H. S. Teoh (5/10) Apr 17 2019 What kind of conflicts?
• drug (2/5) Apr 17 2019 To be more correct I mean version constraints.
• H. S. Teoh (14/19) Apr 17 2019 Ah, I see what you mean. So package xyz may have, say, 10 different
• Paul Backus (5/22) Apr 17 2019 Dependency resolution is actually NP-complete in the general case:
• H. S. Teoh (16/37) Apr 17 2019 Ouch. The term "dependency hell" never seemed more apt, after reading
• Russel Winder (15/19) Apr 18 2019 [=E2=80=A6]
• Nick Sabalausky (Abscissa) (13/20) Apr 18 2019 My old JVM days predate Gradle, but at a glance it looks like Gradle is
• Russel Winder (21/27) Apr 18 2019 On Thu, 2019-04-18 at 12:01 -0400, Nick Sabalausky (Abscissa) via Digita...
• Nick Sabalausky (Abscissa) (22/65) Apr 17 2019 Oh, don't get me wrong, there's a whole BUNCH of things I'd like to see
• Russel Winder (23/30) Apr 17 2019 [=E2=80=A6]
• NaN (4/31) Apr 17 2019 The people who are actually going to do the work should setup a
• rikki cattermole (9/39) Apr 17 2019 Over on Discord we have the Graphics work group which is working out
• bachmeier (20/27) Apr 17 2019 This is a leadership problem. Notably absent from these
• Andre Pany (15/43) Apr 17 2019 Please be more precise. Dub does a perfect job if you have
• drug (5/24) Apr 17 2019 I agree with you totally. I don't understand why W&A should appear in
• bachmeier (16/28) Apr 17 2019 Well clearly others disagree with you. You are saying "it works
• Nick Sabalausky (Abscissa) (13/31) Apr 17 2019 FWIW, the only reason I was being non-canonical about desired features
• H. S. Teoh (30/64) Apr 17 2019 The idea behind this is that in order to avoid baking in support for a
• drug (16/39) Apr 17 2019 Couple of years ago I've made an example -
• Dragos Carp (22/38) Apr 17 2019 Invoking the external tools is probably doable, the problem is
• H. S. Teoh (86/126) Apr 17 2019 It's not really a problem, as far as dub is concerned. All we need is
• Julian (35/40) Apr 17 2019 What you lose with this is dub's current platform independence.
• Nick Sabalausky (Abscissa) (2/11) Apr 17 2019 +1, exactly correct
• JN (23/41) Apr 17 2019 In the ideal world, perhaps. I am just worried opening dub to
• Julian (17/24) Apr 17 2019 Is there no good C or C++ XML library that you could make a
• Julian (5/8) Apr 17 2019 37x, rather. It's still 2s ("I can hit enter and wait for the
• Russel Winder (11/18) Apr 18 2019 [=E2=80=A6]
• Dmitry Olshansky (30/37) Apr 25 2019 Author of std.regex here. It's been awhile since I monitored its
• Nick Sabalausky (Abscissa) (29/56) Apr 17 2019 No, there's no "ideal world" about this. What HSTeoh describes above are...
• H. S. Teoh (74/94) Apr 17 2019 Again, this concern is addressed by my proposal of build system wrapper
• Nick Sabalausky (Abscissa) (3/7) Apr 17 2019 Oh dear god that is not even *REMOTELY* in the ballpark of actual
• Nick Sabalausky (Abscissa) (38/79) Apr 17 2019 I agree, letting build system packages tell the packmangr how to use
• H. S. Teoh (7/15) Apr 17 2019 Awesome, let's see if we can actually get this thing off the ground.
• Atila Neves (5/15) Apr 16 2019 They really shouldn't be. If anything, dub is a good example of
• Dragos Carp (28/30) Apr 16 2019 I have a different proposal.
• aliak (4/17) Apr 16 2019 Super nice! I was just going to ask if anyone's been using bazel
• 9il (4/19) Apr 16 2019 Very nice, I can add support for mir packages. Do you have an
• Dragos Carp (4/7) Apr 16 2019 Not yet. I'll get back to you when this is set. Travis config is
• H. S. Teoh (33/33) Apr 26 2019 Recently, Nick & some others (including myself) created a Github
• H. S. Teoh (8/15) Apr 26 2019 [...]
• Andre Pany (8/23) Apr 26 2019 Also with recent Dub version a major performance optimization was
• Seb (8/34) Apr 26 2019 Unfortunately I'm not actively working on this. I have too many
• Basile B. (9/24) Apr 26 2019 Speaking of bottlenecks... (and probably unrelated):
• H. S. Teoh (10/18) Apr 26 2019 Shouldn't we be able to just run a profiler on dub to figure out where
• Basile B. (6/23) May 01 2019 I've determined that the slowdown is caused by the auto-fetch
• H. S. Teoh (43/59) Apr 26 2019 [...]
• Seb (4/13) Apr 26 2019 10 seconds for a rebuild is still too much.
• Seb (24/29) Apr 26 2019 Thanks a lot for your interest in dub and investigating this!
• Seb (4/11) Apr 26 2019 https://dlang.org/changelog/2.086.0.html#single-api-requests
• Nick Sabalausky (Abscissa) (14/17) Apr 27 2019 To clarify, this project is only intended to replace the package
• H. S. Teoh (32/74) Apr 29 2019 Just to be clear: the 10 seconds rebuild time is only if dub.selections
Seb <seb wilzba.ch> writes:
Hi all,

The problem
-----------

I think this is well-known, but I still highly encourage you all
to have a look at this thread:

https://forum.dlang.org/thread/eftttpylxanvxjhoigqu forum.dlang.org?page=1

or the issue tracker:

https://github.com/dlang/dub/issues

tl;dr: the status quo of dub is really really terrible:

- tons of issues (often unanswered)
- __critical__ piece of infrastructure
- less than a PR per month

I had to use Cargo the other day and compiling 80 (!)
dependencies was a
miracle (parallel fetching and builds).

We want the D ecosystem to grow and flourish and I don't think I
have to
stress how important a good package manager is for this.

What can we do?
---------------

We have about $6k in our OpenCollective budget [1] (there's still the$3k that we owe WebFreak) which means we have about $3k left. I propose to use this as a starting base to fund serious work on Dub (see the forum thread for a few ideas). I believe that with the D community pitching in and a few companies (they complain about dub too), we could get about$10k for this
effort.

Now, of course, the question is what can we do to improve Dub's
status quo?

It would be amazing if we could hire Martin or Sönke (even if
it's only
part-time), but unfortunately as far as I am aware that's not an
option
as both are pretty busy these days.

[1] https://opencollective.com/dlang/

Proposed actionable
-------------------

Hence, here's my proposal:

Publish a blog post analog to this "Calls to arms for Dub". The
post could

- summarize the reasons why funding dub is so crucial
- what the biggest problems of dub are (i.e. the issues that we
would
target first)
- announce that we start a new funding round (with $3k already seeded from OC) - maybe some companies are willing to pitch in too (e.g. Symmetry, Funkwerk, Sociomantic, ...) - put out a call open for anyone to apply for this project The exact details on payment and expectations would need to be specified, but I guess we should be able to fund roughly three months of work. OTOH we can leave these details open to the applicants as students might be willing to work for$10k for
three months, but experienced software engineers might not. Also,
some people probably could only afford to work part-time on these
which imho would be fine too.

community:

1) Would this be sth. you would consider applying in general? If
not what could we do to improve this?
2) Would this be sth. you would consider donating to?

For completeness, my personal favorite items for this would be
sth. like:

- parallel dependency downloads and builds (dub could and should
be a lot faster)
- option to emit only include flags (for good integration in
other tools)
- "dub install"
- dub watch (there's a PR that needs to be revived -
https://github.com/dlang/dub/pull/446)
- colored error messages (there's already a PR that just needs to
be revived - https://github.com/dlang/dub/pull/1490)
- cached dub registry index

Though I think if we crowd-fund this campaign, we should make a
list of the top 20-30 issues and then let everyone vote on them.

Apr 14 2019
Guillaume Piolat <first.last gmail.com> writes:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
1) Would this be sth. you would consider applying in general?
If not what could we do to improve this?

If think DUB is completely _unsound_ and need a rewrite from
someone versed in compilers and correctness.

2) Would this be sth. you would consider donating to?

Only if it's a rewrite.

Apr 14 2019
Andre Pany <andre s-e-a-p.de> writes:
On Sunday, 14 April 2019 at 10:59:29 UTC, Guillaume Piolat wrote:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
1) Would this be sth. you would consider applying in general?
If not what could we do to improve this?

If think DUB is completely _unsound_ and need a rewrite from
someone versed in compilers and correctness.

2) Would this be sth. you would consider donating to?

Only if it's a rewrite.

Dub has a working eco system and a lot of developers put effort
into it. It will take years to have another solution with the
same functionality like Dub and please do not forget we still
have to maintain Dub until the new solution can replace Dub.
Could you please write the details why a correction of Dub isn't
possible and a complete rewrite is the only solution?

Kind regards
Andre

Apr 14 2019
Guillaume Piolat <first.last gmail.com> writes:
On Sunday, 14 April 2019 at 11:21:09 UTC, Andre Pany wrote:
Dub has a working eco system and a lot of developers put effort
into it. It will take years to have another solution with the
same functionality like Dub and please do not forget we still
have to maintain Dub until the new solution can replace Dub.
Could you please write the details why a correction of Dub
isn't possible and a complete rewrite is the only solution?

Don't mistake me, DUB is a good software that is very useful and
that's a feat for a build system - something very easy to hate.

A lot of things are possible incrementally, but I believe in the
DUB design some things will not be possible.

- dub.selections.json is supposed to encode dependency
resolution, but fundamentally it does that for a particular
optional dependencies selections set and configuration.

- I personally think it has way to many features, and in such
situations it's way harder to find those to remove because
agreement is difficult

- work is piling on for DUB because it is designed in a way that
say yes to everything, features have piled on with no regards for
who would maintain them

Withing DUB there is a much simpler subset to be discovered I
believe. For the record I was an early contributor, I did what I
could.

Apr 14 2019
Guillaume Piolat <first.last gmail.com> writes:
On Sunday, 14 April 2019 at 11:38:23 UTC, Guillaume Piolat wrote:
Withing DUB there is a much simpler subset to be discovered I
believe.

With that said, I think I have been unreasonable, would
contribute to an incremental effort too.

Dub probably just lacks a tutorial.

Apr 14 2019
Atila Neves <atila.neves gmail.com> writes:
On Sunday, 14 April 2019 at 11:38:23 UTC, Guillaume Piolat wrote:
On Sunday, 14 April 2019 at 11:21:09 UTC, Andre Pany wrote:
Dub has a working eco system and a lot of developers put
effort into it. It will take years to have another solution
with the same functionality like Dub and please do not forget
we still have to maintain Dub until the new solution can
replace Dub.
Could you please write the details why a correction of Dub
isn't possible and a complete rewrite is the only solution?

Don't mistake me, DUB is a good software that is very useful
and that's a feat for a build system - something very easy to
hate.

A lot of things are possible incrementally, but I believe in
the DUB design some things will not be possible.

- dub.selections.json is supposed to encode dependency
resolution, but fundamentally it does that for a particular
optional dependencies selections set and configuration.

- I personally think it has way to many features, and in such
situations it's way harder to find those to remove because
agreement is difficult

- work is piling on for DUB because it is designed in a way
that say yes to everything, features have piled on with no
regards for who would maintain them

Withing DUB there is a much simpler subset to be discovered I
believe. For the record I was an early contributor, I did what
I could.

+1 to all your points, including your previous post on this
thread.

FWIW, I'm currently writing code to use dub as a library and
bypass most of it.

Rewriting from scratch isn't feasible - it has to be bug
compatible. But reusing the parts that get information from
packages and ignoring everything else...

Apr 15 2019
JN <666total wp.pl> writes:
On Sunday, 14 April 2019 at 11:21:09 UTC, Andre Pany wrote:
Dub has a working eco system and a lot of developers put effort
into it. It will take years to have another solution with the
same functionality like Dub and please do not forget we still
have to maintain Dub until the new solution can replace Dub.
Could you please write the details why a correction of Dub
isn't possible and a complete rewrite is the only solution?

to D. I remember what life was before dub. First of all, there
was no central repository, so it was hard to find a package
you're interested in to begin with. Secondly, every package
required additional steps to download dependencies, usually
through a bash script. Then you had to roll a dice which build
tool it expects, will it be bud, dsss, maybe makefiles, maybe
something else? Oh, and if you were on Windows, you were screwed
99% of time, because the author loves bash and never tested it on
non-POSIX platforms.

Right now the situation is soo much better. I can download a
random project from the internet, type dub build and it just
works. It will download and build necessary dependencies and then
build the project, usually without any manual steps involved.

Personally I haven't really encountered these pain points with D,
so it's hard for me to say if it's a cause I'd donate to, but I
think anything related to improving Dub is crucial for D
ecosystem to flourish.

Apr 14 2019
Zekereth <paul acheronsoft.com> writes:
On Sunday, 14 April 2019 at 20:09:17 UTC, JN wrote:
On Sunday, 14 April 2019 at 11:21:09 UTC, Andre Pany wrote:
[...]

came to D. I remember what life was before dub. First of all,
there was no central repository, so it was hard to find a
package you're interested in to begin with. Secondly, every
package required additional steps to download dependencies,
usually through a bash script. Then you had to roll a dice
which build tool it expects, will it be bud, dsss, maybe
makefiles, maybe something else? Oh, and if you were on
Windows, you were screwed 99% of time, because the author loves
bash and never tested it on non-POSIX platforms.

[...]

I just want to mirror this exactly. I didn't really play with D
much until DUB came along. Now is nearly always the language that
I reach for when starting a new project.

Apr 15 2019
Guillaume Piolat <first.last gmail.com> writes:
On Sunday, 14 April 2019 at 20:09:17 UTC, JN wrote:

came to D. I remember what life was before dub.

+1

Life before DUB was hellish because it was impossible to have
many projects, or rely on anithing external. Improving DUB
(whatever the way) is a fantastic opportunity to improve D.

Apr 15 2019
Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
Hi all,

The problem
-----------

Hey guys, maybe this is an offtopic (which could sound heretical
btw), but didn't you ever think about writing a package manager
using some scripting languages like Python or JavaScript?

The package manager is an infrastructure thing, so it not so
critical as the rest of other tools.

Maybe, a consideration of this idea would help to attract more
people from the outside of the community, because there is a
feeling that we're pretty much locked on the community itself.

Apr 14 2019
Andre Pany <andre s-e-a-p.de> writes:
On Sunday, 14 April 2019 at 11:27:18 UTC, Jacob Shtokolov wrote:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
Hi all,

The problem
-----------

Hey guys, maybe this is an offtopic (which could sound
heretical btw), but didn't you ever think about writing a
package manager using some scripting languages like Python or
JavaScript?

The package manager is an infrastructure thing, so it not so
critical as the rest of other tools.

Maybe, a consideration of this idea would help to attract more
people from the outside of the community, because there is a
feeling that we're pretty much locked on the community itself.

In addition to the Dub executable, you can also use Dub as
Library in your D coding. This will not longer be possible, or
only   with a lot more effort.

Several developers have put a lot of effort to make DMD
independent from the Microsoft build tools / visual studio.
If we now add a new dependency to Python or NPM, that would be
really strange)

Kind regards
Andre

Apr 14 2019
Tourist <gravatar gravatar.com> writes:
On Sunday, 14 April 2019 at 11:27:18 UTC, Jacob Shtokolov wrote:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
Hi all,

The problem
-----------

Hey guys, maybe this is an offtopic (which could sound
heretical btw), but didn't you ever think about writing a
package manager using some scripting languages like Python or
JavaScript?

The package manager is an infrastructure thing, so it not so
critical as the rest of other tools.

Maybe, a consideration of this idea would help to attract more
people from the outside of the community, because there is a
feeling that we're pretty much locked on the community itself.

We are too small a community so can you please do this for us?
Not a good argument.

Apr 14 2019
bachmeier <no spam.net> writes:
On Sunday, 14 April 2019 at 11:27:18 UTC, Jacob Shtokolov wrote:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
Hi all,

The problem
-----------

Hey guys, maybe this is an offtopic (which could sound
heretical btw), but didn't you ever think about writing a
package manager using some scripting languages like Python or
JavaScript?

The package manager is an infrastructure thing, so it not so
critical as the rest of other tools.

Maybe, a consideration of this idea would help to attract more
people from the outside of the community, because there is a
feeling that we're pretty much locked on the community itself.

I wonder what would possibly be gained from using Python or
Javascript. If the goal is to have developers that are not
current D users: (i) Why would they want to work on Dub? (ii) Why
would we turn over the development of critical infrastructure to
someone that doesn't know D and isn't interested in learning?

And then there's the marketing: D community rewrites critical
infrastructure in an alternative language because D doesn't cut
it. A better decision would be to shut down the project entirely.

Apr 15 2019
Andre Pany <andre s-e-a-p.de> writes:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
Hi all,

The problem
-----------

[...]

I fully agree.

Also somehow off topic but we should not only think about how to
solve coding problems but also how to attract new developers
which in the end increases the community and will also help us to
solve issues and build enhancements and in the end will also
attract other developers.

Maybe we should invest some money into marketing. Also another
idea is to setup a campaing, for 2 weeks all community members
concentrate not on development but on advertising D in some form.

Maybe we can also donate some exemplares of Ali's book to
students in universities.

There are so much possibilities to increase the visibility of D.

Kind regards
Andre

Apr 14 2019
Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
1) Would this be sth. you would consider applying in general?
If not what could we do to improve this?
2) Would this be sth. you would consider donating to?

No and no. Dub delivers negative value for my use-case; it is a
cost to me to maintain these definition files for other people
(and it is a support pain too, which people dropping in having
dub problems that don't exist in the underlying compiler), even
though I don't use it.

I don't use it because it offers nothing of value to me; it
solves a problem I don't even have (and does so in a way that is
mostly incompatible with my existing code). It is all negative
with nothing in the positive column to balance it out.

I'd say we split dub up into three concerns:

1) code.dlang.org. I do see some value in this, and think it is
salvageable in its current form.

2) dub, the package manager. Maybe useful, though I don't
personally believe in dependencies, I can see some value, though
I'd want to change it so it has more compatibility with other
workflows (like mine) and be sure to limit its scope to developer
use-cases; end users should never know.

3) dub, the build tool. I'd rather have it either do absolutely
nothing here, or just delegate to something else.

I gotta run, might write more later.

Apr 14 2019
Russel Winder <russel winder.org.uk> writes:
On Sun, 2019-04-14 at 13:40 +0000, Adam D. Ruppe via Digitalmars-d
wrote:
=20

[=E2=80=A6]
No and no. Dub delivers negative value for my use-case; it is a=20
cost to me to maintain these definition files for other people=20
(and it is a support pain too, which people dropping in having=20
dub problems that don't exist in the underlying compiler), even=20
though I don't use it.

So that is fine, you do not use it. But for those who have experience
Cargo with Rust, Go, Conan with C++, etc. D needs something like Dub.

I don't use it because it offers nothing of value to me; it=20
solves a problem I don't even have (and does so in a way that is=20
mostly incompatible with my existing code). It is all negative=20
with nothing in the positive column to balance it out.

See above. :-)

=20
I'd say we split dub up into three concerns:
=20
1) code.dlang.org. I do see some value in this, and think it is=20
salvageable in its current form.
=20
2) dub, the package manager. Maybe useful, though I don't=20
personally believe in dependencies, I can see some value, though=20
I'd want to change it so it has more compatibility with other=20
workflows (like mine) and be sure to limit its scope to developer=20
use-cases; end users should never know.
=20
3) dub, the build tool. I'd rather have it either do absolutely=20
nothing here, or just delegate to something else.

I disagree. The lessons, particularly from Cargo/Rust, and Go is that
"small language, small standard library, large pool of trivially
accessible dependencies" is the way things are going just now. Having a
dependency and build manager is what Maven and (far better) Gradle have
been doing for yonks in the JVM-based milieu, this way of working is
finally penetrating into the native code arena.

Make, CMake, SCons, Meson, etc. remain tied to the "I have all the
source I need locally, everything else is provided via precompiled
static or dynamic libraries that are present locally". If this works
for a given use case fine. But increasingly the JVM world, Cargo/Rust,
Go are working with a distributed source code approach. An approach
that can use a central publishing place, local file locations, or
Git/Mercurial/Breezy repositories.=20

Having worked with Gradle, Cargo, and Go, returning to the approach of
Make, CMake, SCons, and Meson really has no appeal to me at all.

A consequence of this for me is that for D almost all of Phobos should
not be in Phobos but should be in the Dub repository as packages, and
Dub should work with Git/Mercurial/Breezy repositories as easily as it
does with the Dub repostory and local filestores.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 15 2019
Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 15 April 2019 at 07:27:54 UTC, Russel Winder wrote:
I disagree. The lessons, particularly from Cargo/Rust, and Go
is that "small language, small standard library, large pool of
trivially accessible dependencies" is the way things are going
just now.

You can still have that (and more of it, as we can get more
library authors on board) if the systems are more compatible.

You can keep your dub build command if you must, but it should
just forward to some user-defined system, which is able to get
metadata like build target back out of dub.

So, the dub executable does four (actually more!), conceptually
independent tasks:

1) get metadata about package
2) download package for use
3) full dependency resolution
4) build

And I want to decouple those as much as we can.

Apr 15 2019
Russel Winder <russel winder.org.uk> writes:
On Mon, 2019-04-15 at 16:44 +0000, Adam D. Ruppe via Digitalmars-d
wrote:
On Monday, 15 April 2019 at 07:27:54 UTC, Russel Winder wrote:
I disagree. The lessons, particularly from Cargo/Rust, and Go=20
is that "small language, small standard library, large pool of=20
trivially accessible dependencies" is the way things are going=20
just now.

=20
You can still have that (and more of it, as we can get more=20
library authors on board) if the systems are more compatible.
=20
You can keep your dub build command if you must, but it should=20
just forward to some user-defined system, which is able to get=20
metadata like build target back out of dub.
=20
=20
So, the dub executable does four (actually more!), conceptually=20
independent tasks:
=20
1) get metadata about package
2) download package for use
3) full dependency resolution
4) build
=20
And I want to decouple those as much as we can.

To be honest this is all implementation detail about which most users
will not care. Most users really just want a single command with a CLI
or a way of integrating with their IDE or editor.=20

So with Cargo and Rust you have the cargo command. How it is
implemented really doesn't matter.

With Go there is the go command. How it is implemented really doesn't
matter.

If Dub is the system for D then dub has a command line, how it is
implemented under the hood really doesn't matter as long as it works.
And currently it seems a bit of a mess. Especially compared to Cargo
and Go.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 16 2019
Russel Winder <russel winder.org.uk> writes:
On Sun, 2019-04-14 at 10:53 +0000, Seb via Digitalmars-d wrote:
[=E2=80=A6]
=20
I had to use Cargo the other day and compiling 80 (!)=20
dependencies was a
miracle (parallel fetching and builds).

[=E2=80=A6]

And herein lies the problem. Cargo is properly supported with actual
resource that isn't the occasional volunteer effort. Also Cargo has a
significantly better way of working with dependencies and build than
Dub. Perhaps the D community as a whole should stop working on
DMD/LDC/GDC etc. and start working on Dub?

Much as I prefer D over Rust for the GTK+ and GStreamer stuff I am
doing, there is always the Rustward drive because of Cargo and because
of the Rust plugin in CLion (*).

It is clear that Dub was and is a good idea per se, but that lots of
decisions made along the way have meant it not as good for D as Cargo
is for Rust. :-(

(*) I know I had promised to help out on the CLion D plugin and have
failed. The Big Problem=E2=84=A2 is that it is easier to write code now usi=
ng
Rust and CLion with the Rust plugin than it is to get the CLion D
plugin to a state where it is usable in production. I chat with the
JetBrains/CLion folks from time to time, especially at ACCU
conferences. Clearly they cannot take over management of the D plugin
for IntelliJ IDEA and CLion as there is no perceived traction of D
compared to Rust, but neither can they justify providing any resource
to support the D plugin for the exact same reason. Rust has perceived
traction in the market, D has none. :-(

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 15 2019
Anton Fediushin <fediushin.anton yandex.com> writes:
I don't have much experience contributing to dub but I have
contributed to dub-registry in the past and must I say, it wasn't
a pleasant experience for me. Everything I'm writing in this post
is from a year ago, I am not sure what's the state of the
ecosystem now.

Contributing starts with setting up a local development
environment. For dub-registry it was awful to say the least. I
had to fix my local vibe-d source and some setup code while
making sure that I don't commit it to one of my frontend-related
branches.

Compilation of dub-registry on a laptop with just 4Gb RAM takes
ages and hangs up the whole system. A tiny edit in any diet
template caused them all to be parsed over and over again. I'd
like to remind you that I was focusing on frontend and user
integration therefore I had to recompile registry *many times*.
Imagine how much it slowed me down.

Other thing is, of course, technology that registry used. It was
ancient. I couldn't push the idea of adding a css preprocessor
like SASS to sort out the CSS anarchy we had through maintainers
who wanted to avoid any third-party dependencies for the sake of
simpler development. Yes, 1000-line CSS file which had styles for
the whole registry and had 0 useful documentation was kept. (I
wonder if it is still there lol)

After all the struggles I opened a PR. Usually it took a week or
two to get merged, bigger changes were hanging for an entire
month. This was so frustrating. I appreciate all of the
maintainers who provided their feedback and pointed out at my
obvious mistakes but we could have saved so much of each other's
time if dub-registry used somewhat modern technologies, had
documentation and maybe some guidelines.

Later dub-registry has been split into the registry itself and
dub documentation. Yes, now simple PRs to dub had to have a
documentation PR to this new repository. They are slowing each
other down for very little benefit.

In conclusion, I will not support dub's further development. It
is a piece of software I saw so much potential in, but it failed
all of my expectations. It is not up to modern world's standards.
It slows down the whole community.

I totally agree with people who see the only solution to this
problem in a complete rewrite of dub. My only addition is that
registry has to be rewritten as well, now using modern
technologies. There is no shame if it will be written in
JavaScript or whatever. D is not the right tool for that
particular job.

Best regards,
Anton

Apr 15 2019
Andre Pany <andre s-e-a-p.de> writes:
On Monday, 15 April 2019 at 09:34:00 UTC, Anton Fediushin wrote:
I don't have much experience contributing to dub but I have
contributed to dub-registry in the past and must I say, it
wasn't a pleasant experience for me. Everything I'm writing in
this post is from a year ago, I am not sure what's the state of
the ecosystem now.

[...]

What issues do you have specific to Dub?

You wrote: it slows down the whole community. This statement is
not true. It does not slow me down, actually it is working like a
charme for me now (I did a few dub pull requests).

Kind regards
Andre

Apr 15 2019
Anton Fediushin <fediushin.anton yandex.com> writes:
On Monday, 15 April 2019 at 11:06:56 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 09:34:00 UTC, Anton Fediushin wrote:
I don't have much experience contributing to dub but I have
contributed to dub-registry in the past and must I say, it
wasn't a pleasant experience for me. Everything I'm writing in
this post is from a year ago, I am not sure what's the state
of the ecosystem now.

[...]

What issues do you have specific to Dub?

You wrote: it slows down the whole community. This statement is
not true. It does not slow me down, actually it is working like
a charme for me now (I did a few dub pull requests).

Kind regards
Andre

It is slowing down the d community because it's not what a
package manager and a build system of a modern programming
language should look like. For example, LDC is able to compile
code for quite a few architectures, even GPUs, yet you cannot
painlessly integrate that with dub.

As soon as you want to do something slightly unusual, your
dub.json/dub.sdl becomes a spaghetti of
preBuildCommands/postBuildCommands. In my personal opinion,
package file of a high-level build system (which is what dub is
trying to be) should never contain any shell commands.

And yes, dub.json/dub.sdl is another problem of dub. Having two
package formats is nothing but a bad decision. Sure, sdl can
contain comments that are very useful when you are trying to make
sense of pre/postBuildCommands-spaghetti. This is how one bad
design decision depends on another. Thing is, it's impossible to
fix without breaking changes

Best regards,
Anton

Apr 15 2019
Andre Pany <andre s-e-a-p.de> writes:
On Monday, 15 April 2019 at 13:34:21 UTC, Anton Fediushin wrote:
On Monday, 15 April 2019 at 11:06:56 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 09:34:00 UTC, Anton Fediushin
wrote:
I don't have much experience contributing to dub but I have
contributed to dub-registry in the past and must I say, it
wasn't a pleasant experience for me. Everything I'm writing
in this post is from a year ago, I am not sure what's the
state of the ecosystem now.

[...]

What issues do you have specific to Dub?

You wrote: it slows down the whole community. This statement
is not true. It does not slow me down, actually it is working
like a charme for me now (I did a few dub pull requests).

Kind regards
Andre

It is slowing down the d community because it's not what a
package manager and a build system of a modern programming
language should look like. For example, LDC is able to compile
code for quite a few architectures, even GPUs, yet you cannot
painlessly integrate that with dub.

As soon as you want to do something slightly unusual, your
dub.json/dub.sdl becomes a spaghetti of
preBuildCommands/postBuildCommands. In my personal opinion,
package file of a high-level build system (which is what dub is
trying to be) should never contain any shell commands.

And yes, dub.json/dub.sdl is another problem of dub. Having two
package formats is nothing but a bad decision. Sure, sdl can
contain comments that are very useful when you are trying to
make sense of pre/postBuildCommands-spaghetti. This is how one
bad design decision depends on another. Thing is, it's
impossible to fix without breaking changes

Best regards,
Anton

Integration of dub with LDC is working fine. I created a tutorial
(german) here:

http://d-land.sepany.de/tutorials/einplatinenrechner/einstieg-in-die-raspberry-pi-entwicklung-mit-ldc/

But yes, integration could be better.

I use the shell commands to compile a git commit ID into the
executable. It works like a charme. There are cases where shell
commands are highly valuable and does not lead to spaghetti code.

Without specific examples it is hard to discuss wheter something
works or not.

Kind regards
Andre

Apr 15 2019
Atila Neves <atila.neves gmail.com> writes:
On Monday, 15 April 2019 at 16:09:10 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 13:34:21 UTC, Anton Fediushin wrote:
[...]

Integration of dub with LDC is working fine. I created a
tutorial (german) here:

http://d-land.sepany.de/tutorials/einplatinenrechner/einstieg-in-die-raspberry-pi-entwicklung-mit-ldc/

But yes, integration could be better.

I use the shell commands to compile a git commit ID into the
executable. It works like a charme.

Try doing that cross platform between Windows and Linux.

Without specific examples it is hard to discuss wheter
something works or not.

My experience matches Anton's - as soon as you try to do anything
non-trivial in "pure dub" it gets frustrating pretty quickly.
I've given examples before.

Apr 15 2019
Andre Pany <andre s-e-a-p.de> writes:
On Monday, 15 April 2019 at 17:17:49 UTC, Atila Neves wrote:
On Monday, 15 April 2019 at 16:09:10 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 13:34:21 UTC, Anton Fediushin
wrote:
[...]

Integration of dub with LDC is working fine. I created a
tutorial (german) here:

http://d-land.sepany.de/tutorials/einplatinenrechner/einstieg-in-die-raspberry-pi-entwicklung-mit-ldc/

But yes, integration could be better.

I use the shell commands to compile a git commit ID into the
executable. It works like a charme.

Try doing that cross platform between Windows and Linux.

Without specific examples it is hard to discuss wheter
something works or not.

My experience matches Anton's - as soon as you try to do
anything non-trivial in "pure dub" it gets frustrating pretty
quickly. I've given examples before.

Actually I have a bat script for the windows release pipeline and
a bash script for linux/Darwin pipeline.

My assumption is, there are a lot of developers which uses dub
without any issue and for them replacing Dub would be a major
pain.

Also a lot of bugs were already solved in Dub in the recent
month's. Bug fixes are just not listed in the D changelog.

Kind regards
Andre

Apr 15 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/15/19 2:31 PM, Andre Pany wrote:

My assumption is, there are a lot of developers which uses dub without
any issue and for them replacing Dub would be a major pain.

We can't really say that replacing dub would be a pain without looking
at what the replacement would be. Without that, we can only speculate.

Apr 15 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 15, 2019 at 08:59:26PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
On 4/15/19 2:31 PM, Andre Pany wrote:

My assumption is, there are a lot of developers which uses dub
without any issue and for them replacing Dub would be a major pain.

We can't really say that replacing dub would be a pain without looking
at what the replacement would be. Without that, we can only speculate.

As long as dub's current functionality is supported and remains the
default behaviour, this is not an issue.  What's currently lacking isn't
dub's default behaviour, it's the lack of support for changing the
default behaviour.

T

--
Who told you to swim in Crocodile Lake without life insurance??

Apr 16 2019
Atila Neves <atila.neves gmail.com> writes:
On Monday, 15 April 2019 at 18:31:01 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 17:17:49 UTC, Atila Neves wrote:
On Monday, 15 April 2019 at 16:09:10 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 13:34:21 UTC, Anton Fediushin
wrote:
[...]

Integration of dub with LDC is working fine. I created a
tutorial (german) here:

http://d-land.sepany.de/tutorials/einplatinenrechner/einstieg-in-die-raspberry-pi-entwicklung-mit-ldc/

But yes, integration could be better.

I use the shell commands to compile a git commit ID into the
executable. It works like a charme.

Try doing that cross platform between Windows and Linux.

Without specific examples it is hard to discuss wheter
something works or not.

My experience matches Anton's - as soon as you try to do
anything non-trivial in "pure dub" it gets frustrating pretty
quickly. I've given examples before.

Actually I have a bat script for the windows release pipeline
and a bash script for linux/Darwin pipeline.

Which is less than ideal.

My assumption is, there are a lot of developers which uses dub
without any issue and for them replacing Dub would be a major
pain.

Of course! For trivial applications that want to use dependencies
without hassle, it's great. But it doesn't scale.

Also a lot of bugs were already solved in Dub in the recent
month's. Bug fixes are just not listed in the D changelog.

The ones I filed are all open. And some aren't even filed yet
AFAIK, such as the fact that dub.selections.json, as it exists
now, is broken.

Apr 16 2019
aberba <karabutaworld gmail.com> writes:
On Monday, 15 April 2019 at 18:31:01 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 17:17:49 UTC, Atila Neves wrote:
Integration of dub with LDC is working fine. I created a
tutorial (german) here:

My assumption is, there are a lot of developers which uses dub
without any issue and for them replacing Dub would be a major
pain.

Dub works great for me too. I could  use some build speed though.

By the way, there are some people who consistently bash Dub at
any opportunity they get. They just don't like Dub.

Apr 25 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Apr 26, 2019 at 12:18:54AM +0000, aberba via Digitalmars-d wrote:
[...]
By the way, there are some people who consistently bash Dub at any
opportunity they get. They just don't like Dub.

Because I tried using it and discovered that it wouldn't let me do what
ought to be easy to do.  If it works fine for you, then good for you. It
doesn't do what I need it to do, so I'd rather not use it, but it's
forced upon me because without dub it's impractical to use code from
code.dlang.org.  That's why I feel a lot of frustration with it -- it's
something I can't avoid if I want to use code.dlang.org, but it also
works very poorly for what I do.  I often feel like I'm up the
proverbial creek without a paddle when I'm using dub.  It seems to work
fine up your creek, but I need to be on *this* creek and dub refuses to
give me a paddle to work with.

T

--
A linguistics professor was lecturing to his class one day. "In
English," he said, "A double negative forms a positive. In some
languages, though, such as Russian, a double negative is still a
negative. However, there is no language wherein a double positive can
form a negative." A voice from the back of the room piped up, "Yeah,
yeah."

Apr 25 2019
Atila Neves <atila.neves gmail.com> writes:
On Friday, 26 April 2019 at 00:18:54 UTC, aberba wrote:
On Monday, 15 April 2019 at 18:31:01 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 17:17:49 UTC, Atila Neves wrote:
Integration of dub with LDC is working fine. I created a
tutorial (german) here:

My assumption is, there are a lot of developers which uses dub
without any issue and for them replacing Dub would be a major
pain.

Dub works great for me too. I could  use some build speed
though.

By the way, there are some people who consistently bash Dub at
any opportunity they get. They just don't like Dub.

As I've said on more than one occasion, for simple usage dub
works fine. Slowly, but fine.

It just doesn't scale. Anything non-trivial is hard to do and has
to be hacked together. I liked dub too when I used it for
personal projects but then I had to use it to do "real work" and
it quickly broke down.

Apr 26 2019
Andre Pany <andre s-e-a-p.de> writes:
On Friday, 26 April 2019 at 10:01:26 UTC, Atila Neves wrote:
On Friday, 26 April 2019 at 00:18:54 UTC, aberba wrote:
On Monday, 15 April 2019 at 18:31:01 UTC, Andre Pany wrote:
On Monday, 15 April 2019 at 17:17:49 UTC, Atila Neves wrote:
Integration of dub with LDC is working fine. I created a
tutorial (german) here:

My assumption is, there are a lot of developers which uses
dub without any issue and for them replacing Dub would be a
major pain.

Dub works great for me too. I could  use some build speed
though.

By the way, there are some people who consistently bash Dub at
any opportunity they get. They just don't like Dub.

As I've said on more than one occasion, for simple usage dub
works fine. Slowly, but fine.

It just doesn't scale. Anything non-trivial is hard to do and
has to be hacked together. I liked dub too when I used it for
personal projects but then I had to use it to do "real work"
and it quickly broke down.

I fully agree with you on the current state while I have another
idea on consequences.
Dub is no fully sophisticated build tool but that is OK. It is
intended for the simple cases and it does a good job on that.

Use Dub for simple cases. Integrate Dub into existing Build tools
for complex cases.

We may have to work on the integration cases to make them more
smooth.

I also use Dub integrated into a Xmake implementation and that is
working fine.

Kind regards
Andre

Apr 26 2019
Russel Winder <russel winder.org.uk> writes:
On Fri, 2019-04-26 at 13:06 +0000, Andre Pany via Digitalmars-d wrote:
=20

[=E2=80=A6]
Use Dub for simple cases. Integrate Dub into existing Build tools=20
for complex cases.
=20

[=E2=80=A6]

I started trying to use Dub purely for package fetch in SCons, but I stoppe=
d
using SCons so stopped work on it. Also Dub has a weird way of managing
package build products which was beginning to become a mountain over which =
is
was rather difficult to climb.

If Dub was a library with a Python binding life would be a lot easier.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 26 2019
Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 15 April 2019 at 09:34:00 UTC, Anton Fediushin wrote:
Compilation of dub-registry on a laptop with just 4Gb RAM takes
ages and hangs up the whole system. A tiny edit in any diet
template caused them all to be parsed over and over again.

Yeah, I think the code.dlang.org site is salvageable, but I am
not a vibe.d and especially not a diet template fan. Such a pain.

D is not the right tool for that particular job.

D is amazing for web stuff.

Apr 15 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
I've personally put a lot of effort into trying to fix DUB's problems in
the past. Though I would be happy to be proven wrong, at this point, I'm
all but convinced the problems with DUB are far too fundamental.

I know this is the sort of proposal that makes people cringe, and often
for good reason, but in this case, I really do think it would be
quicker, easier, and produce a better result to simply re-design it from
the ground up (while making sure to leverage the existing code.dlang.org
ecosystem in some say), than to try to wrangle this 600lb gorilla into
something it was never designed to be.

I've long been hoping to take a stab at this myself, and I often find
myself thinking through details of how it would work. I would, however,
need help with the dependency-resolution algorithm (perhaps somebody
could go into DUB and work to decouple its dep resolution algoritms from
the rest DUB as much as possible - or at least document how to interface
with it as a lib).

Apr 15 2019
Guillaume Piolat <first.last gmail.com> writes:
On Monday, 15 April 2019 at 16:37:38 UTC, Nick Sabalausky
(Abscissa) wrote:
I've personally put a lot of effort into trying to fix DUB's
problems in the past. Though I would be happy to be proven
wrong, at this point, I'm all but convinced the problems with
DUB are far too fundamental.

I know this is the sort of proposal that makes people cringe,
and often for good reason, but in this case, I really do think
it would be quicker, easier, and produce a better result to
simply re-design it from the ground up (while making sure to
leverage the existing code.dlang.org ecosystem in some say),
than to try to wrangle this 600lb gorilla into something it was
never designed to be.

I've long been hoping to take a stab at this myself, and I
often find myself thinking through details of how it would
work. I would, however, need help with the
dependency-resolution algorithm (perhaps somebody could go into
DUB and work to decouple its dep resolution algoritms from the
rest DUB as much as possible - or at least document how to
interface with it as a lib).

I've tried paying MrSmith33 to write a DUB replacement for a
while.
I think the "incremental" camp is 80% right saying that a DUB
rewrite is a pipe dream in large parts.

However if you want to build on that small attempt, don't
hesitate to pick what you like:
https://gitlab.com/AuburnSounds/rub (coined "idub" for a pun)

In the end it needs a dedicated programmer with a _principled_
vision, but I'm sure some money would find your way, maybe not
much though.

I think recipe files (which format, we'll disagree on) and the
registry are good and MUST be kept.

Apr 15 2019
Atila Neves <atila.neves gmail.com> writes:
On Monday, 15 April 2019 at 19:52:10 UTC, Guillaume Piolat wrote:
On Monday, 15 April 2019 at 16:37:38 UTC, Nick Sabalausky
(Abscissa) wrote:
I've personally put a lot of effort into trying to fix DUB's
problems in the past. Though I would be happy to be proven
wrong, at this point, I'm all but convinced the problems with
DUB are far too fundamental.

I know this is the sort of proposal that makes people cringe,
and often for good reason, but in this case, I really do think
it would be quicker, easier, and produce a better result to
simply re-design it from the ground up (while making sure to
leverage the existing code.dlang.org ecosystem in some say),
than to try to wrangle this 600lb gorilla into something it
was never designed to be.

I've long been hoping to take a stab at this myself, and I
often find myself thinking through details of how it would
work. I would, however, need help with the
dependency-resolution algorithm (perhaps somebody could go
into DUB and work to decouple its dep resolution algoritms
from the rest DUB as much as possible - or at least document
how to interface with it as a lib).

I've tried paying MrSmith33 to write a DUB replacement for a
while.
I think the "incremental" camp is 80% right saying that a DUB
rewrite is a pipe dream in large parts.

However if you want to build on that small attempt, don't
hesitate to pick what you like:
https://gitlab.com/AuburnSounds/rub (coined "idub" for a pun)

I'll take a look. In the meanwhile I started this:

https://github.com/kaleidicassociates/bud

It uses dub as a library to make sure that it works just as dub
does, but bypasses what it can. The idea is to completely
separate these disparate tasks:

* Dependency version resolution (from dub.{sdl,json} ->
dub.selections.json)
* Fetching dependencies that are not already in the file system
(trivial after dub.selections.json has been generated)
* Extracting information from dub about source files, import
paths, etc.
* Actually using the information from dub to build

In the future dub.selections.json will have to be fixed to have
different entries per configuration, and possibly per platform as
well. Fortunately there's a version field in the current format.
I'm currently concentrating on assuming that dub.selections.json
is generated and taking it from there.

Apr 16 2019
Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 16 April 2019 at 12:12:45 UTC, Atila Neves wrote:
The idea is to completely separate these disparate tasks:

* Dependency version resolution (from dub.{sdl,json} ->
dub.selections.json)
* Fetching dependencies that are not already in the file system
(trivial after dub.selections.json has been generated)
* Extracting information from dub about source files, import
paths, etc.
* Actually using the information from dub to build

It sounds like us dub-skeptics all actually agree on a surprising
number of details.

If we go with a plan like this, we can get more authors on board
and get dub some more serious buy-in.

Apr 16 2019
Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 16 April 2019 at 12:12:45 UTC, Atila Neves wrote:
I'll take a look. In the meanwhile I started this:

https://github.com/kaleidicassociates/bud

It uses dub as a library to make sure that it works just as dub
does, but bypasses what it can. The idea is to completely
separate these disparate tasks:

* Dependency version resolution (from dub.{sdl,json} ->
dub.selections.json)
* Fetching dependencies that are not already in the file system
(trivial after dub.selections.json has been generated)
* Extracting information from dub about source files, import
paths, etc.
* Actually using the information from dub to build

Won't complain about such good news!

dub.json and dub.selections.json must unify though, as
dub.selections.json is not self-sufficient, what if they
contradict themselves?

In the future dub.selections.json will have to be fixed to have
different entries per configuration, and possibly per platform
as well. Fortunately there's a version field in the current
format. I'm currently concentrating on assuming that
dub.selections.json is generated and taking it from there.

I'm not really sure which are the original DUB assumtions.

I think our research concluded that solving dependencies for a
particular platform+configuration is (in an idealized DUB) a
fundamentally different operation than generating a proper
dub.selections.json (for which you would have to find a super-set
for "all conf"). The problem is that configurations/platform
could appear and disappear alongside previous versions.

(For readers: this would lead the path to platform-based
dependencies and configuration-based dependencies which are
currently not doable, such as:

"dependencies-linux": { "x11": "~>1.0" }
)

Apr 16 2019
Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 16 April 2019 at 15:52:57 UTC, Guillaume Piolat wrote:
On Tuesday, 16 April 2019 at 12:12:45 UTC, Atila Neves wrote:
I'll take a look. In the meanwhile I started this:

https://github.com/kaleidicassociates/bud

It uses dub as a library to make sure that it works just as
dub does, but bypasses what it can. The idea is to completely
separate these disparate tasks:

* Dependency version resolution (from dub.{sdl,json} ->
dub.selections.json)
* Fetching dependencies that are not already in the file
system (trivial after dub.selections.json has been generated)
* Extracting information from dub about source files, import
paths, etc.
* Actually using the information from dub to build

Won't complain about such good news!

dub.json and dub.selections.json must unify though, as
dub.selections.json is not self-sufficient, what if they
contradict themselves?

I'm not sure I know what you mean. Unless you edit
dub.selections.json manually, it can't contradict the build
recipe it came from. Version 1 of dub.selections.json isn't
sufficient, but I'm leaving that as a problem to solve later.

I think our research concluded that solving dependencies for a
particular platform+configuration is (in an idealized DUB) a
fundamentally different operation than generating a proper
dub.selections.json (for which you would have to find a
super-set for "all conf").

Could you elaborate on this please? At least for configurations,
I don't see why it wouldn't be possible to generate the current
JSON object in dub.selections.json, but for each configuration
instead. And, possibly platform (linux, x86_64, etc.).

The problem is that configurations/platform could appear and
disappear alongside previous versions.

Could you please give an example of this?

(For readers: this would lead the path to platform-based
dependencies and configuration-based dependencies which are
currently not doable, such as:

"dependencies-linux": { "x11": "~>1.0" }
)

Yep!

Apr 17 2019
Guillaume Piolat <contact youknowwhere.com> writes:
On Wednesday, 17 April 2019 at 08:50:31 UTC, Atila Neves wrote:
Could you please give an example of this?

Sorry I'm not clear.

I'll dig into my conversations with MrSmith33 later and will come
back to you because it's not in my mind anymore (and I wasn't the
one doing the work).

Let's meet at DConf at least to talk about this.

Apr 17 2019
Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 17 April 2019 at 08:50:31 UTC, Atila Neves wrote:
(For readers: this would lead the path to platform-based
dependencies and configuration-based dependencies which are
currently not doable, such as:

"dependencies-linux": { "x11": "~>1.0" }
)

Yep!

I feel I've way overstated the soundness problems of DUB. Why
"dependency-<platform>" and "dependency-<configuration>" is not
supported is explained here, since two years...
https://github.com/dlang/dub/wiki/FAQ

Apr 20 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/16/19 8:12 AM, Atila Neves wrote:

I'll take a look. In the meanwhile I started this:

https://github.com/kaleidicassociates/bud

It uses dub as a library to make sure that it works just as dub does,
but bypasses what it can. The idea is to completely separate these
disparate tasks:

What is the current state of this tool?

Apr 17 2019
Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 17 April 2019 at 19:07:07 UTC, Nick Sabalausky
(Abscissa) wrote:
On 4/16/19 8:12 AM, Atila Neves wrote:

I'll take a look. In the meanwhile I started this:

https://github.com/kaleidicassociates/bud

It uses dub as a library to make sure that it works just as
dub does, but bypasses what it can. The idea is to completely
separate these disparate tasks:

What is the current state of this tool?

In its infancy.

Apr 18 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/15/19 12:37 PM, Nick Sabalausky (Abscissa) wrote:
I've personally put a lot of effort into trying to fix DUB's problems in
the past. Though I would be happy to be proven wrong, at this point, I'm
all but convinced the problems with DUB are far too fundamental.

I know this is the sort of proposal that makes people cringe, and often
for good reason, but in this case, I really do think it would be
quicker, easier, and produce a better result to simply re-design it from
the ground up (while making sure to leverage the existing code.dlang.org
ecosystem in some say), than to try to wrangle this 600lb gorilla into
something it was never designed to be.

I've long been hoping to take a stab at this myself, and I often find
myself thinking through details of how it would work. I would, however,
need help with the dependency-resolution algorithm (perhaps somebody
could go into DUB and work to decouple its dep resolution algoritms from
the rest DUB as much as possible - or at least document how to interface
with it as a lib).

To clarify, I'm mainly referring to the "package manager" aspect of DUB
here. I'm less concerned with the "buildsystem" part simply because with
a proper package manager, individual projects and even libraries will be
free to use whatever buildsystem they want, even if that happens to be
"dub build" (which *is* an entirely reasonable choice for many simpler
projects...in large part because that's specifically the use-case it was
primarily designed for.).

Apr 15 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 15, 2019 at 06:56:24PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
[...]
To clarify, I'm mainly referring to the "package manager" aspect of
DUB here. I'm less concerned with the "buildsystem" part simply
because with a proper package manager, individual projects and even
libraries will be free to use whatever buildsystem they want, even if
that happens to be "dub build" (which *is* an entirely reasonable
choice for many simpler projects...in large part because that's
specifically the use-case it was primarily designed for.).

Y'know, the idea just occurred to me: why even bind the package manager
to the build system at all?  Why not make the build system one of the
dependencies of the project?  So you could have library A that is
written to be built using dub build, and library B that depends on
'make', say (where 'make' is some dub-ified description of make as a
build system), and A depends on B.  So to build A, dub would fetch the
necessary sources to build 'make', then use 'make' to build B, and then
build A, and so forth.

This way, you can have multiple build systems coexisting peacefully with
each other, without requiring the package manager to roll its own build
system.  You could have library C depend on 'scons' and library D depend
on 'gradle' or whatever, as long as there's a suitably dub-ified
description of these build systems, dub could fetch and build them as
dependencies and then use them to build whatever targets are needed.
'dub' itself (or maybe we should call it 'dub-build' or something) would
be its own, standalone build module that projects could depend on by
default.  So if you don't like dub's built-in build system, just
override it to depend on 'make' or whatever your choice of build poison
is.

For backward compatibility, 'dub-build' would be assumed if no build
system is specified.

If we were to pull this off, I might even consider using dub in a
non-toy way. (Right now the only way I can get it to work sanely with my
vibe.d project is to use the hack of an empty dub project whose sole
purpose is to declare dub dependencies. It's ugly, and annoying.)

T

--
Без труда не выловишь и рыбку из пруда.

Apr 15 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/15/19 7:17 PM, H. S. Teoh wrote:
On Mon, Apr 15, 2019 at 06:56:24PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
[...]
To clarify, I'm mainly referring to the "package manager" aspect of
DUB here. I'm less concerned with the "buildsystem" part simply
because with a proper package manager, individual projects and even
libraries will be free to use whatever buildsystem they want, even if
that happens to be "dub build" (which *is* an entirely reasonable
choice for many simpler projects...in large part because that's
specifically the use-case it was primarily designed for.).

Y'know, the idea just occurred to me: why even bind the package manager
to the build system at all?  Why not make the build system one of the
dependencies of the project?  So you could have library A that is
written to be built using dub build, and library B that depends on
'make', say (where 'make' is some dub-ified description of make as a
build system), and A depends on B.  So to build A, dub would fetch the
necessary sources to build 'make', then use 'make' to build B, and then
build A, and so forth.

Yup, from the sound of it, I think we're on exactly the same page here.
Basically, a package manager's config should need exactly three things
from your project:

1. What dependencies.
2. The command(s) to (build, test, etc.)
3. Name(s)/Location(s) of the build's output.

Then, with that information, a package manager provides services such as
(but not necessarily limited to):

1. A simple, standardized way for you and your users to obtain/build the
dependencies.

2. A simple, standardized way for buildscripts/buildsystems to obtain
the information needed to include the dependencies in their own build
(such as -I... include directories, paths to the now-already-built
lib/exec binaries, etc.)

From this, each project can naturally either just roll its own
buildscripts, or depend on another package providing a builsystem.

Some of the *details* can be quite nontrivial...like dependency
resolution algorithms, or designing the interactions between package
manager and buildsystem to be simple, yet effective enough to suit all
parties needs. But ultimately, it boils down conceptually to be very simple.

I tried to argue in favor of an approach like this earlier on in DUB's
development, but the author maintained that making a package ecosystem
all work together requires all packages using the same buildsystem. I
wasn't able to convince otherwise.

Apr 15 2019
rikki cattermole <rikki cattermole.co.nz> writes:
On 16/04/2019 11:52 AM, Nick Sabalausky (Abscissa) wrote:
I tried to argue in favor of an approach like this earlier on in DUB's
development, but the author maintained that making a package ecosystem
all work together requires all packages using the same buildsystem. I
wasn't able to convince otherwise.

I can understand that argument. Regarding targets same compiler ext.

But I do agree that it would be the better design if they could all be
worked out.

Apr 15 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Apr 15, 2019 at 07:52:07PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
[...]
[...]  Basically, a package manager's config should need exactly three
things from your project:

1. What dependencies.
2. The command(s) to (build, test, etc.)
3. Name(s)/Location(s) of the build's output.

Then, with that information, a package manager provides services such
as (but not necessarily limited to):

1. A simple, standardized way for you and your users to obtain/build
the dependencies.

2. A simple, standardized way for buildscripts/buildsystems to obtain
the information needed to include the dependencies in their own build
(such as -I... include directories, paths to the now-already-built
lib/exec binaries, etc.)

I'd add:

3. A standard query interface for querying a remote repository for
packages (matching some names / patterns) and version numbers.

From this, each project can naturally either just roll its own
buildscripts, or depend on another package providing a builsystem.

That's a good idea.  Completely decouple package management from
building.  Let the package manager do what it does best: managing
packages, and leave the compilation to another tool more suited for the
job.  Of course, the default build command can be set to dub build to
keep existing users happy.

I was thinking the build/test/etc. command can itself be defined by a
dependency.  For example, if your project uses make for builds, there
could be a make dub package that defines command-line templates used
for generating platform-specific build commands. Your package would
simply specify:

"build-system": "make"

and the "make" package would define the exact command(s) for invoking
make (e.g., if gmake is picked up as a resolution, it could define the
executable name as gmake, or it could define different CLI syntax for
Windows vs. Linux, etc.).  The package manager would then recognize
"make" as a dependency, and would download and install that package
(which presumably would install an appropriate version of 'make') if it
hasn't already been installed, then invoke it to build the project.

The "make" package itself would be presumably some kind of wrapper
around a system utility, or an installer of some sort that installs an
appropriate version of make on the system.  It could be implemented as a
normal dub package whose "build" command is something that runs an
attached installer.

Some of the *details* can be quite nontrivial...like dependency
resolution algorithms, or designing the interactions between package
manager and buildsystem to be simple, yet effective enough to suit all
parties needs.  But ultimately, it boils down conceptually to be very
simple.

[...]

If done correctly, dependency resolution is just a topological walk on a
DAG.  Get this right, and everything else falls into place.

To generate the DAG, there would need to be some kind of list of
repositories, which defaults to code.dlang.org (but which can be
customized if necessary -- e.g., add local filesystem repos for local
packages, or a URL to a private additional repository).  There would
also need to be some kind of version resolution scheme.  It shouldn't be
too hard to do: all you need is for repositories to support three
queries:

1) Fetch a list of packages matching the given name(s);

2) Filter said list by a given OS/platform identifier.

3) Filter said list by zero or more given version specs (either an exact
match, or a <= or >= match, or whatever else you may want to filter
version strings by.).

Repositories ought to support such queries efficiently, i.e., the
package manager shouldn't have to download the entire list of 1000
packages each with 50 different version identifiers, and then perform an
O(n) search on the list to find the necessary matching package.  It
should behave like a database: you send it an OS identifier, a list of
desired package names, each with one or more version filters, and it
returns a list of matching package descriptions, including URLs from
which you may download the package(s).  It should take only 1 network
roundtrip to get this information.

Then once the package manager receives the URLs, it can use whatever
method necessary to download said URLs (or just cache it somewhere if
it's a pathname on the local filesystem), and read the package
description(s) to find any additional dependencies that it may need.

T

--
A program should be written to model the concepts of the task it performs
rather than the physical world or a process because this maximizes the
potential for it to be applied to tasks that are conceptually similar and, more
important, to tasks that have not yet been conceived. -- Michael B. Allen

Apr 16 2019
drug <drug2004 bk.ru> writes:
On 16.04.2019 20:39, H. S. Teoh wrote:
If done correctly, dependency resolution is just a topological walk on a
DAG.  Get this right, and everything else falls into place.

How do you resolve conflicts using DAG only?

Apr 17 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 17, 2019 at 10:40:29AM +0300, drug via Digitalmars-d wrote:
On 16.04.2019 20:39, H. S. Teoh wrote:
If done correctly, dependency resolution is just a topological walk
on a DAG.  Get this right, and everything else falls into place.

How do you resolve conflicts using DAG only?

What kind of conflicts?

T

--
Too many people have open minds but closed eyes.

Apr 17 2019
drug <drug2004 bk.ru> writes:
On 17.04.2019 18:53, H. S. Teoh wrote:

What kind of conflicts?

To be more correct I mean version constraints.

Apr 17 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 17, 2019 at 07:03:33PM +0300, drug via Digitalmars-d wrote:
On 17.04.2019 18:53, H. S. Teoh wrote:

What kind of conflicts?

To be more correct I mean version constraints.

Ah, I see what you mean.  So package xyz may have, say, 10 different
versions, and each version could potentially have different dependencies
(since dependencies can change over time).  So after narrowing down to
the subset of xyz versions that satisfy the given version criteria, you
still have to decide which of the remaining versions to use, since they
may depend on other packages that are subject to other version
constraints.  So you'll need some graph algorithms to resolve these
constraints.

Interesting, I've never thought about this before.  I'll have to think
about this some more.  Thanks!

T

--
Not all rumours are as misleading as this one.

Apr 17 2019
Paul Backus <snarwin gmail.com> writes:
On 4/17/19 12:18 PM, H. S. Teoh wrote:
On Wed, Apr 17, 2019 at 07:03:33PM +0300, drug via Digitalmars-d wrote:
To be more correct I mean version constraints.

Ah, I see what you mean.  So package xyz may have, say, 10 different
versions, and each version could potentially have different dependencies
(since dependencies can change over time).  So after narrowing down to
the subset of xyz versions that satisfy the given version criteria, you
still have to decide which of the remaining versions to use, since they
may depend on other packages that are subject to other version
constraints.  So you'll need some graph algorithms to resolve these
constraints.

Interesting, I've never thought about this before.  I'll have to think
about this some more.  Thanks!

T

Dependency resolution is actually NP-complete in the general case:

https://research.swtch.com/version-sat

Of course, in practice, there are solutions that work well enough to be
usable; several are discussed in the linked article.

Apr 17 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 17, 2019 at 07:29:39PM -0400, Paul Backus via Digitalmars-d wrote:
On 4/17/19 12:18 PM, H. S. Teoh wrote:
On Wed, Apr 17, 2019 at 07:03:33PM +0300, drug via Digitalmars-d wrote:
To be more correct I mean version constraints.

Ah, I see what you mean.  So package xyz may have, say, 10 different
versions, and each version could potentially have different
dependencies (since dependencies can change over time).  So after
narrowing down to the subset of xyz versions that satisfy the given
version criteria, you still have to decide which of the remaining
versions to use, since they may depend on other packages that are
subject to other version constraints.  So you'll need some graph
algorithms to resolve these constraints.

Interesting, I've never thought about this before.  I'll have to
think about this some more.  Thanks!

[...]
Dependency resolution is actually NP-complete in the general case:

https://research.swtch.com/version-sat

Of course, in practice, there are solutions that work well enough to
be usable; several are discussed in the linked article.

Ouch.  The term "dependency hell" never seemed more apt, after reading
the article. :-/

Interestingly, NP completeness can be avoided by only allowing packages
to specify minimum versions, rather than a specific version (or
arbitrarily complex version conditions).  Allowing installation of
multiple versions of the same package also gets away from NP
completeness, though in practice it's probably a lot harder to pull off.
An interesting hybrid approach is to combine both using semver, which
sorta makes sense in some way. Though it's unclear how well this works
in practice.

Time to bust out that SAT solver... :-/

T

--
Tell me and I forget. Teach me and I remember. Involve me and I understand. --
Benjamin Franklin

Apr 17 2019
Russel Winder <russel winder.org.uk> writes:
On Wed, 2019-04-17 at 22:27 -0700, H. S. Teoh via Digitalmars-d wrote:
[=E2=80=A6]
=20
Ouch.  The term "dependency hell" never seemed more apt, after
reading
the article. :-/

[=E2=80=A6]

The Gradle people have been through all this over the years, and now
have an excellent dependency management system including allowing quite
fine grain manual specifications.

Gradle now has a C++ build as well as the whol gamut of JVM-based
languages. An alternative is to add D to this.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 18 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/18/19 5:40 AM, Russel Winder wrote:

The Gradle people have been through all this over the years, and now
have an excellent dependency management system including allowing quite
fine grain manual specifications.

Gradle now has a C++ build as well as the whol gamut of JVM-based
languages. An alternative is to add D to this.

My old JVM days predate Gradle, but at a glance it looks like Gradle is
mainly a buildsystem. As the key problem we're facing is needing to
separate package management from buildsystem, utilizing Gradle seems
like a tough sell (tough to sell to me, included), even if only utilized
for its package management.

Although I'm personally not entirely against the idea of utilizing an
existing cross-platform language-agnostic package manager behind-the-scenes.

For example: What do you think of 0install?
<https://github.com/Abscissa/DPak/issues/1>  I haven't personally used
it, but I've had my eye on it for quite a number of years. Seems similar
to Nix, but without the difficult Nix expression language, and without
the difficulty of installing older package versions.

Apr 18 2019
Russel Winder <russel winder.org.uk> writes:
On Thu, 2019-04-18 at 12:01 -0400, Nick Sabalausky (Abscissa) via Digitalma=
rs-
d wrote:
[=E2=80=A6]
=20
My old JVM days predate Gradle, but at a glance it looks like Gradle is=

=20
mainly a buildsystem. As the key problem we're facing is needing to=20
separate package management from buildsystem, utilizing Gradle seems=20
like a tough sell (tough to sell to me, included), even if only utilized=

=20
for its package management.

[=E2=80=A6]

Gradle has always been an integrated dependency management and build system
ever since its genesis in the Conservatory of Barbican Centre in 2007.

Looking at Gradle, Maven, Cargo, Go, Conan, the choice is for integrated
dependency management and build. Dub it would seem has taken the right over=
all
approach, just not gone an implementation route that serves D the way Cargo
serves Rust.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 18 2019
Guillaume Piolat <first.last gmail.com> writes:
On Friday, 19 April 2019 at 06:19:42 UTC, Russel Winder wrote:
Gradle has always been an integrated dependency management and
build system ever since its genesis in the Conservatory of
Barbican Centre in 2007.

Looking at Gradle, Maven, Cargo, Go, Conan, the choice is for
integrated dependency management and build. Dub it would seem
has taken the right overall approach, just not gone an
implementation route that serves D the way Cargo serves Rust.

+1 (for everything that Russel said in this thread!)

Apr 19 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/19/19 2:19 AM, Russel Winder wrote:

Looking at Gradle, Maven, Cargo, Go, Conan, the choice is for integrated
dependency management and build. Dub it would seem has taken the right overall
approach,

Well, it certainly seems to have taken a *popular* approach. I wouldn't
necessarily take that as implying the "right" approach (for us).

just not gone an implementation route that serves D the way Cargo
serves Rust.

Maybe. But there's two things I think you're overlooking:

1. Differences in manpower.
2. Differences in situation/requirements.

1: Getting something like that right, without being overly-limited,
amounts to quite a significant project in terms of manpower and, though
I could be wrong, my perspective is that those communities have had more
manpower to put into such a project than we have. At the very least, I
think DUB has proven it's a poor approach for our current level of manpower.

2: D currently has plenty of buildsystem options to choose from, there's
likely to be something for everyone. BUT, what D completely lacks right
now, and desperately needs IMO, is a good package manager that's
non-restrictive enough to suit everyone's needs. Making the D community
wait to get that until we *finally* manage to work out our own
one-buildsystem-to-rule-them-all (which we're currently nowhere near) is
a terrible strategic approach for our current position.

Later on, if a D buildsystem that works great for everyone's needs
actually manages to emerge...THEN we can integrate our package manager
with it.

Summary: There is FAR more to good design and management than playing
follow-the-leader.

Apr 19 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Apr 19, 2019 at 03:54:45PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
On 4/19/19 2:19 AM, Russel Winder wrote:

Looking at Gradle, Maven, Cargo, Go, Conan, the choice is for
integrated dependency management and build. Dub it would seem has
taken the right overall approach,

Well, it certainly seems to have taken a *popular* approach. I
wouldn't necessarily take that as implying the "right" approach (for
us).

[...]

Yes. I for one dumped Gradle shortly after starting my Android project,
because it just didn't let me do what I need to do, or at least not
easily.  Gradle expects your typical Java codebase with standard source
tree structure.  I needed D codegen and cross-compilation as integral
parts of my build.  The two ... let's just say, don't work very well
together.  It's the "my way or the highway" philosophy all over again.
Yes it hides away a lot of complexity, and does a lot of nice things
automatically -- when what you need to do happens to match the narrow
use case Gradle was designed to do.  But when you need to do something
*other* than the One Accepted Way, you're in for a long uphill battle --
assuming it's even possible in the first place.  To that, I say, No
Thank You, I have other tools that fit better with how I work.

(Not to mention, I have a hard time justifying to myself installing a
multi-GB program just to be able to compile a tiny bit of code. And the
bare act of invoking Gradle soaks up GBs of RAM and takes forever and 64
days just to decide what it should be doing.  Perhaps I'm just a cranky
old antiquated ape who needs to just get with the times, but I seriously
cannot swallow that this is now the Accepted Way of Doing Things.  And
with D compile times supposedly being one of its top selling points, I
seriously cannot see how such an approach will work well in the long
run.  (Though OTOH perhaps it will finally get rid of that
cringe-inspiring fast-fast-fast motto.))

In any case, with all the work being poured into betterC and C++
interop, any proposed universal D build / package system MUST take into
account interacting with C/C++ codebases, which generally do *not* use
this integrated package/build approach.  At the very least, a D build
system that has pretensions of being universal must allow easy
integration of C/C++ codebases, and that means accepting the possibility
of other build systems, and at least making an effort to work with them
nicely.

This is why I proposed a standard "API" for the package manager to
interact with arbitrary codebases with arbitrary build systems.  Rather
than imposing arbitrary restrictions on how packages ought to be built,
it should acknowledge that other build systems exist, and provide a
simple, consistent way of interacting with them that will *still* work
well with those who prefer the integrated approach.

T

--
If creativity is stifled by rigid discipline, then it is not true creativity.

Apr 19 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/19/19 5:58 PM, H. S. Teoh wrote:
On Fri, Apr 19, 2019 at 03:54:45PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
On 4/19/19 2:19 AM, Russel Winder wrote:
Looking at Gradle, Maven, Cargo, Go, Conan, the choice is for
integrated dependency management and build. Dub it would seem has
taken the right overall approach,

Well, it certainly seems to have taken a *popular* approach. I
wouldn't necessarily take that as implying the "right" approach (for
us).

[...]

Yes. I for one dumped Gradle shortly after starting my Android project,
because it just didn't let me do what I need to do, or at least not
easily.  Gradle expects your typical Java codebase with standard source
tree structure.  I needed D codegen and cross-compilation as integral
parts of my build.  The two ... let's just say, don't work very well
together.  It's the "my way or the highway" philosophy all over again.

No big surprise why a "my way or the highway philosophy" is more
successful with a JVM audience than a D audience ;)

Like I alluded to: different language -> different audience -> different
needs and requirements -> different "right" answers.

Apr 19 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Apr 20, 2019 at 01:14:12AM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
On 4/19/19 5:58 PM, H. S. Teoh wrote:

[...]
Yes. I for one dumped Gradle shortly after starting my Android
project, because it just didn't let me do what I need to do, or at
least not easily.  Gradle expects your typical Java codebase with
standard source tree structure.  I needed D codegen and
cross-compilation as integral parts of my build.  The two ... let's
just say, don't work very well together.  It's the "my way or the
highway" philosophy all over again.

No big surprise why a "my way or the highway philosophy" is more
successful with a JVM audience than a D audience ;)

Like I alluded to: different language -> different audience ->
different needs and requirements -> different "right" answers.

What I have trouble comprehending is, the technology needed to support
*both* use cases already exists.  So why not use it??  Why arbitrarily
exclude certain use cases in favor of one specific one, when there is no
technological reason not to support all use cases?  We already have the
tech to fly to the moon and back, yet we arbitrarily impose the
restriction that the aircraft must remain in contact with the runway at
all times, because some grandma on the plane is scared of heights (aka
codebases that don't conform to the One True Way To Organize And Build
Source Code).  It doesn't make any sense to me.

T

--
It's bad luck to be superstitious. -- YHL

Apr 19 2019
Russel Winder <russel winder.org.uk> writes:
On Fri, 2019-04-19 at 14:58 -0700, H. S. Teoh via Digitalmars-d wrote:
=20

[=E2=80=A6]
Yes. I for one dumped Gradle shortly after starting my Android project,
because it just didn't let me do what I need to do, or at least not
easily.  Gradle expects your typical Java codebase with standard source
tree structure.  I needed D codegen and cross-compilation as integral
parts of my build.  The two ... let's just say, don't work very well
together.  It's the "my way or the highway" philosophy all over again.
Yes it hides away a lot of complexity, and does a lot of nice things
automatically -- when what you need to do happens to match the narrow
use case Gradle was designed to do.  But when you need to do something
*other* than the One Accepted Way, you're in for a long uphill battle --
assuming it's even possible in the first place.  To that, I say, No
Thank You, I have other tools that fit better with how I work.
=20

Gradle is definitely not rigid as implied in the above: it can work with an=
y
source structure. True there is a default, and "convention over configurati=
on"
is the main philosophy, but this can be easily overiden by those who need t=
o.=20

There are hooks for doing pre-compilation code generation, though I suspect
whilst there is support for C++, there is no ready made support for D.

That you chose to ditch Gradle and go another way is entirely fine, but to
denigrate Gradle as above based on what appears to be a single episode quic=
kly
abandoned, is a bit unfair to Gradle.

=20

[=E2=80=A6]

The elided comments on Gradle requiring JVM and thus lots of memory and
relatively slow startup is very fair. Much better grounds for your ditching=
of
Gradle than not finding the Gradle way of doing things you needed to do.

I feel Gradle is probably not a good choice for D builds now, but it could =
be.
This is because no-one is using for that purpose and so the easy to use, D
oriented tools are not available, everything has to be done with the base
Gradle tools.=20

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 26 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Apr 26, 2019 at 02:37:31PM +0100, Russel Winder via Digitalmars-d wrote:
On Fri, 2019-04-19 at 14:58 -0700, H. S. Teoh via Digitalmars-d wrote:

[…]
Yes. I for one dumped Gradle shortly after starting my Android
project, because it just didn't let me do what I need to do, or at
least not easily.  Gradle expects your typical Java codebase with
standard source tree structure.  I needed D codegen and
cross-compilation as integral parts of my build.  The two ... let's
just say, don't work very well together.  It's the "my way or the
highway" philosophy all over again.  Yes it hides away a lot of
complexity, and does a lot of nice things automatically -- when what
you need to do happens to match the narrow use case Gradle was
designed to do.  But when you need to do something *other* than the
One Accepted Way, you're in for a long uphill battle -- assuming
it's even possible in the first place.  To that, I say, No Thank
You, I have other tools that fit better with how I work.

Gradle is definitely not rigid as implied in the above: it can work
with any source structure. True there is a default, and "convention
over configuration" is the main philosophy, but this can be easily
overiden by those who need to.

There are hooks for doing pre-compilation code generation, though I
suspect whilst there is support for C++, there is no ready made
support for D.

Here's a question, since you obviously know Gradle far better than my
admitted ignorance: does Gradle support the following?

- Compile a subset of source files into an executable, and run said
executable on some given set of input files in order to produce a set
of auto-generated source files. Then compile the remaining source
files along with the auto-generated ones to produce the final product.

- Compile a subset of source files into an executable, then run said
executable on a set of data files to transform them in some way, then
invoke a second program on the transformed data files to produce a set
of images that are then post-processed by image-processing tools to
produce the final outputs.

- Given a set of build products (executables, data files, etc.), pack
them into an archive, compress, sign, etc., to produce a deliverable.

- Have all of the above work correctly and minimally whenever one or
more input files (either data files or source files) changes.
Minimally meaning if a particular target doesn't depend on the changed
input files, it doesn't get rebuilt, and if an intermediate build
product is identical to the previous build, subsequent steps are
elided.

If Gradle can support all of the above, then it may be worthy of
consideration.

Of course, the subsequent question would then be, how easy is it to
accomplish the above, i.e., is it just a matter of adding a few lines to
the build description, or does it involve major build system hackery.

That you chose to ditch Gradle and go another way is entirely fine,
but to denigrate Gradle as above based on what appears to be a single
episode quickly abandoned, is a bit unfair to Gradle.

I did not intend to denigrate Gradle; I was merely relating my own
experience with it.  Clearly, it works very well for many people,
otherwise it wouldn't even be on the radar in the first place.  But it
took far too much effort than I was willing to put in to get it to do
what I need to do, esp. when I already have an SCons system that I'm
already familiar with, and can get up and running with minimal effort.
Given such a choice, it should be obvious why I decided against using
Gradle.

[...]
The elided comments on Gradle requiring JVM and thus lots of memory
and relatively slow startup is very fair. Much better grounds for your
ditching of Gradle than not finding the Gradle way of doing things you
needed to do.

See, this is the problem I have with many supposedly "modern" tools.
They are giant CPU and memory hogs, require tons of disk space for who
knows what, take forever to start up, and chug along taking their sweet
time just to get the simplest of tasks done, where supposedly "older" or
"antiquated" tools fire up in an instant, get right to work, and get the
same task done in a fraction of the time.  This in itself may have been
excusable, but then these "modern" tools also come with a whole bunch of
red tape, arbitrary limitations, and the philosophy of "follow our way
or walk there on foot yourself".  IOW they do what "antiquated" tools do
poorly, and can't even do advanced things said "antiquated" tools can do
without any fuss -- and this not because of *technical* limitations, but
because it was arbitrarily decided at some point that use cases X and Y
will not be supported, just 'cuz.

So somebody please enlighten this here dinosaur of *why* I should choose
these "modern" tools over my existing, and IMO better, ones?

This is why I'm a fan of empowering the user.  Instead of being obsessed
over how to package your software in a beautiful jewel-encrusted box
with a gold ribbon on top, (which IMO is an utter waste of time and only
attracts those with an inflated sense of entitlement), give the user the
tools to do what the software can do. Instead of delivering a magic
black box that does everything but with the hood welded shut, give them
the components within the black box with which they can build something
far beyond what you may have initially conceived.  I believe *that* is
the way to progress, not this "hand me my software on a silver platter"
philosophy so pervasive nowadays.

(The jewel-encrusted box is still nice, BTW -- but only as long as it's
not at the expense of welding the hood shut.)

I feel Gradle is probably not a good choice for D builds now, but it
could be.  This is because no-one is using for that purpose and so the
easy to use, D oriented tools are not available, everything has to be
done with the base Gradle tools.

[...]

If Gradle is dependent on Java, then using it for D builds would be kind
of ... anticlimactic. :-D  But probably the primary issue would be the
slow startup times and heavy storage requirements that goes against the
grain of our present (cringe-worthy) fast-fast-fast slogan. I suppose
it's no coincidence that the Gradle logo is ... just as slow and mellow
as the tool itself. ;-)  (I love RL elephants, BTW, so this isn't meant
in a denigrating way. But I'd have a very hard time justifying the
order-of-magnitude increase in my edit-compile-run cycle turnaround
times.  Life is far too short to have to wait for such things.)

T

--
Talk is cheap. Whining is actually free. -- Lars Wirzenius

Apr 26 2019
Gilter <Gilter gmall.com> writes:
On Friday, 26 April 2019 at 17:30:50 UTC, H. S. Teoh wrote:
On Fri, Apr 26, 2019 at 02:37:31PM +0100, Russel Winder via
Digitalmars-d wrote:
On Fri, 2019-04-19 at 14:58 -0700, H. S. Teoh via
Digitalmars-d wrote:

[…]
Yes. I for one dumped Gradle shortly after starting my
Android project, because it just didn't let me do what I
need to do, or at least not easily.  Gradle expects your
typical Java codebase with standard source tree structure.
I needed D codegen and cross-compilation as integral parts
of my build.  The two ... let's just say, don't work very
well together.  It's the "my way or the highway" philosophy
all over again.  Yes it hides away a lot of complexity, and
does a lot of nice things automatically -- when what you
need to do happens to match the narrow use case Gradle was
designed to do.  But when you need to do something *other*
than the One Accepted Way, you're in for a long uphill
battle -- assuming it's even possible in the first place.
To that, I say, No Thank You, I have other tools that fit
better with how I work.

Gradle is definitely not rigid as implied in the above: it can
work with any source structure. True there is a default, and
"convention over configuration" is the main philosophy, but
this can be easily overiden by those who need to.

There are hooks for doing pre-compilation code generation,
though I suspect whilst there is support for C++, there is no
ready made support for D.

Here's a question, since you obviously know Gradle far better
than my admitted ignorance: does Gradle support the following?

- Compile a subset of source files into an executable, and run
said
executable on some given set of input files in order to
produce a set
of auto-generated source files. Then compile the remaining
source
files along with the auto-generated ones to produce the final
product.

- Compile a subset of source files into an executable, then run
said
executable on a set of data files to transform them in some
way, then
invoke a second program on the transformed data files to
produce a set
of images that are then post-processed by image-processing
tools to
produce the final outputs.

Not sure what you find difficult about this. You create 2 tasks,
one task generates the executable. The second task runs the
executable, add the first task as a dependency of the second
task. It really just boils down to, do this task before this
other task and a task should be able to be just a command you can
run. Which is basically what every build system supports.

https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:adding_dependencies_to_tasks

- Given a set of build products (executables, data files,
etc.), pack
them into an archive, compress, sign, etc., to produce a
deliverable.

This really is just a task where you run a command. Signing is
different for basically every system, and depending on the system
it can be different based on what program you are using to sign.
If you expect the build system to know how to sign an executable
by just doing "sign: true" then you aren't going to find a build
system like that unless it only works on one system.

For zip files you can easily archive them:

https://docs.gradle.org/current/userguide/working_with_files.html#sec:archives

- Have all of the above work correctly and minimally whenever
one or
more input files (either data files or source files) changes.
Minimally meaning if a particular target doesn't depend on
the changed
input files, it doesn't get rebuilt, and if an intermediate
build
product is identical to the previous build, subsequent steps
are
elided.

This has been around for quite a while, since 2.5 by the looks of
it. You have to specify the input and output files so it can
detect if they were changed.

https://docs.gradle.org/2.5/userguide/more_about_tasks.html#sec:up_to_date_checks

If Gradle can support all of the above, then it may be worthy
of consideration.

Of course, the subsequent question would then be, how easy is
it to accomplish the above, i.e., is it just a matter of adding
a few lines to the build description, or does it involve major
build system hackery.

What build systems do you know that support the above. Like I
said somethings are kind of unreasonable, like expecting a build
system to be able to sign your executables with just a simple
flag. The signing process I have to do doesn't actually allow us
to automate it. For some reason it won't work, you have to be
physically be at the server for it to sign the executable. Seems
like it might be a security feature, can't even remote into the
server to sign it manually like that either.

That you chose to ditch Gradle and go another way is entirely
fine, but to denigrate Gradle as above based on what appears
to be a single episode quickly abandoned, is a bit unfair to
Gradle.

I did not intend to denigrate Gradle; I was merely relating my
own experience with it.  Clearly, it works very well for many
people, otherwise it wouldn't even be on the radar in the first
place.  But it took far too much effort than I was willing to
put in to get it to do what I need to do, esp. when I already
have an SCons system that I'm already familiar with, and can
get up and running with minimal effort. Given such a choice, it
should be obvious why I decided against using Gradle.

Imagine that, learning something new takes a lot more effort than
using something you already know. Who would have thought?

[...]
The elided comments on Gradle requiring JVM and thus lots of
memory and relatively slow startup is very fair. Much better
grounds for your ditching of Gradle than not finding the
Gradle way of doing things you needed to do.

See, this is the problem I have with many supposedly "modern"
tools. They are giant CPU and memory hogs, require tons of disk
space for who knows what, take forever to start up, and chug
along taking their sweet time just to get the simplest of tasks
done, where supposedly "older" or "antiquated" tools fire up in
an instant, get right to work, and get the same task done in a
fraction of the time.  This in itself may have been excusable,
but then these "modern" tools also come with a whole bunch of
red tape, arbitrary limitations, and the philosophy of "follow
our way or walk there on foot yourself".  IOW they do what
"antiquated" tools do poorly, and can't even do advanced things
said "antiquated" tools can do without any fuss -- and this not
because of *technical* limitations, but because it was
arbitrarily decided at some point that use cases X and Y will
not be supported, just 'cuz.

So somebody please enlighten this here dinosaur of *why* I
should choose these "modern" tools over my existing, and IMO
better, ones?

What do you consider giant CPU and memory hogs that require a ton
of disk space? Cause from my experience the people here complain
about VS taking up so much space, end up doing development work
on some shitty tablet with 32 GB of HDD space.

I just use a python script, I don't feel like bothering with
things like SCons. I execute exactly what I need and it is easy
to see what it is doing and what is happening. It doesn't take
long to compile either though, I usually only need to re-compile
one component as well.

Use whatever you want, just don't go condemning something cause
you never bothered to learn how to use it.

Apr 27 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/27/19 10:25 AM, Gilter wrote:
Use whatever you want, just don't go condemning something cause you
never bothered to learn how to use it.

As he already said, he wasn't condemning it, just relaying his
experience with it.

Apr 27 2019
Dibyendu Majumdar <d.majumdar gmail.com> writes:
Hi,

I think there is a lot to be said for using Gradle / and Maven
repository as the package management tool. Why reinvent something
that already works.

Gradle is a task dependency management tool at its core - it
doesn't care what these tasks are, and you can write plugins to
create any tasks you want. I have seen Gradle used successfully
for C++ projects.

Regards
Dibyendu

Apr 27 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/26/19 1:30 PM, H. S. Teoh wrote:
See, this is the problem I have with many supposedly "modern" tools.
They are giant CPU and memory hogs, require tons of disk space for who
knows what, take forever to start up, and chug along taking their sweet
time just to get the simplest of tasks done, where supposedly "older" or
"antiquated" tools fire up in an instant, get right to work, and get the
same task done in a fraction of the time.  This in itself may have been
excusable, but then these "modern" tools also come with a whole bunch of
red tape, arbitrary limitations, and the philosophy of "follow our way
or walk there on foot yourself".  IOW they do what "antiquated" tools do
poorly, and can't even do advanced things said "antiquated" tools can do
without any fuss -- and this not because of *technical* limitations, but
because it was arbitrarily decided at some point that use cases X and Y
will not be supported, just 'cuz.

+1

So somebody please enlighten this here dinosaur of *why* I should choose
these "modern" tools over my existing, and IMO better, ones?

Because "you just have to believe" and join the newer-is-always-better
cult? Because we're told by everyday joe that we always have to keep up
with new technology or get left behind, so by golly that must be true,
after all, who are we to question those smarter than us, those from whom
everyday joe obtained the "keep up or left behind" Truth?

I think THAT'S why you have to. ;)

This is why I'm a fan of empowering the user.  Instead of being obsessed
over how to package your software in a beautiful jewel-encrusted box
with a gold ribbon on top, (which IMO is an utter waste of time and only
attracts those with an inflated sense of entitlement), give the user the
tools to do what the software can do. Instead of delivering a magic
black box that does everything but with the hood welded shut, give them
the components within the black box with which they can build something
far beyond what you may have initially conceived.  I believe *that* is
the way to progress, not this "hand me my software on a silver platter"
philosophy so pervasive nowadays.

(The jewel-encrusted box is still nice, BTW -- but only as long as it's
not at the expense of welding the hood shut.)

Unix philosophy. Yup. *nods*. "A tool should do one thing, and do it
well." The Lego approach to software.

I grew up on MS-DOS and Windows (well, ok, Apple 2...then DOS/Win). And
hey, they served their purposes for me. Fast-forward some years later
and I was talking to a game programmer I had previously known (gamedev
circles tend to be fairly Win/MS-heavy - and there's admittedly been
reasons for that). At one point he stopped and said, "Hey, wait a
minute, since when did you become a Linux guy?" Well, since I grew up
and learned that I like being able to automate any repetitive sequence I
need to, to aid my productivity and help manage cognitive load. The unix
philosophy is key in enabling that.

It's also why a world of random hackers have been able to build and
maintain a system that, in terms of capabilities, even giants like
Microsoft and Google can't keep up with (although, certain business
realities actually make it much more lucrative for large companies to
keep their products artificially limited - so it's not as if they're
especially motivated to compete with *nix on capabilities).

Apr 27 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Apr 27, 2019 at 04:41:56PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
On 4/26/19 1:30 PM, H. S. Teoh wrote:

[...]
This is why I'm a fan of empowering the user.  Instead of being
obsessed over how to package your software in a beautiful
jewel-encrusted box with a gold ribbon on top, (which IMO is an
utter waste of time and only attracts those with an inflated sense
of entitlement), give the user the tools to do what the software can
do. Instead of delivering a magic black box that does everything but
with the hood welded shut, give them the components within the black
box with which they can build something far beyond what you may have
initially conceived.  I believe *that* is the way to progress, not
this "hand me my software on a silver platter" philosophy so
pervasive nowadays.

[...]
Unix philosophy. Yup. *nods*. "A tool should do one thing, and do it
well." The Lego approach to software.

It's not so much doing one thing and doing it well (even though that is
a natural consequence), but (1) giving the user access to the tools that
*you* have at your disposal, and (2) making the interface *composable*,
so that the user is empowered to put the blocks together in brand new
ways that you've never thought of.

[...]
"Hey, wait a minute, since when did you become a Linux guy?" Well,
since I grew up and learned that I like being able to automate any
repetitive sequence I need to, to aid my productivity and help manage
cognitive load. The unix philosophy is key in enabling that.

[...]

Yes, the ability to automate *any arbitrary task* is a key functionality
that modern designs seem to have overlooked. Automation is the raison
d'etre of the first machines.  Humans are bad at repetitive tasks, and
machines are supposed to take over these tasks so that humans can excel
at the other things they are good at.

Yet the UIs of the modern day force you to click through endless nested
levels of submenus and provide very little (if any at all) way to
automate things.  And when such is provided, it's usually arbitrarily
limited or encumbered in some way, and usually cannot come up to the
full functionality accessible when you do things the manual way.  It
doesn't make any sense.  Machines are supposed to abstract away the
repetitive tasks and allow you to program them to do arbitrary things on
the click of a button, not to force you to make repetitive gestures and
get a wrist aneurysm because the GUI designer didn't think scripting was
an important use case.  The whole point of a general-purpose computing
machine is to be able to apply this automation to ANY ARBITRARY TASK,
not merely some arbitrarily limited subset that the designers deigned
worthy to be included.

Making your interface composable is what allows your code to be
composable with any other code in a painless way.

This is why more and more in my own designs I'm leaning towards the
guiding principle of programs being thin wrappers around reusable
library modules that provide the real functionality. And the programs
themselves should as much as possible provide the simplest, most
straightforward interface that makes it composable with any other
arbitrary program.

Composability, because the user is smarter than you and will think of
use cases far beyond your wildest imaginations, so they should not be
arbitrarily limited to only what you can think of.

Libraries, because no matter how awesome your program interface is,
eventually somebody will want to reuse your functionality from their
*own* program.  Allowing them to do that without needlessly cumbersome
workarounds like spawning a subprocess and contorting their code to your
quirky CLI syntax or method of invocation will make them very happy and
loyal users, and will increase the scope of applicability of your code
far beyond what you may have conceived.

(And no, remote procedure call is not an excuse to avoid providing a
library API to your functionality.  RPC has its uses, but the user
should be in control of how they wish your code to run. They should not
need to spawn a daemon via JSON over HTTP to a remote server just to be
able to call a single function. They *can* do this *if they want to*,
but this should not be the *only* way of interfacing with your code. The
user should be empowered to deploy your code according to *their* needs,
not yours. They should be able to choose whether to call your function
as a local function call to a DLL / .so on the local filesystem, or to
run your library on a remote server accessed via RPC over the network.)

Empower the user to do what *they* want, how they want it, rather than
straitjacket them to do what *you* want, the way you dictate it. This
principle applies to all levels of code, from not writing a class that
requires its methods to be called in a specific order otherwise it
crashes or gets into an inconsistent state, to encapsulating your
primary functionality in a library API that can be used anywhere in any
way.

Life is too short to be wasted inventing shims and workarounds to
functionalities that *should* have been made available generally with no
strings attached and no straitjackets imposing arbitrary limitations.
Empowerment, composability, automatability.  I'm tired of trying to work
with software that doesn't provide all three.  It's just not worth my
bother anymore.

T

--
MSDOS = MicroSoft's Denial Of Service

Apr 28 2019
Russel Winder <russel winder.org.uk> writes:
On Fri, 2019-04-26 at 10:30 -0700, H. S. Teoh via Digitalmars-d wrote:
=20

[=E2=80=A6]
Here's a question, since you obviously know Gradle far better than my
admitted ignorance: does Gradle support the following?
=20
- Compile a subset of source files into an executable, and run said
executable on some given set of input files in order to produce a set
of auto-generated source files. Then compile the remaining source
files along with the auto-generated ones to produce the final product.

Yes.

- Compile a subset of source files into an executable, then run said
executable on a set of data files to transform them in some way, then
invoke a second program on the transformed data files to produce a set
of images that are then post-processed by image-processing tools to
produce the final outputs.

Yes.

- Given a set of build products (executables, data files, etc.), pack
them into an archive, compress, sign, etc., to produce a deliverable.

Yes.

- Have all of the above work correctly and minimally whenever one or
more input files (either data files or source files) changes.
Minimally meaning if a particular target doesn't depend on the changed
input files, it doesn't get rebuilt, and if an intermediate build
product is identical to the previous build, subsequent steps are
elided.

Yes.

If Gradle can support all of the above, then it may be worthy of
consideration.
=20
Of course, the subsequent question would then be, how easy is it to
accomplish the above, i.e., is it just a matter of adding a few lines to
the build description, or does it involve major build system hackery.

Not all the things are part of Gradle as distributed, you have to write som=
e
tasks. However the infrastructure is there to make writing the tasks
straightforward. This not one or two lines, but not major hackery.

[...]
=20
See, this is the problem I have with many supposedly "modern" tools.
They are giant CPU and memory hogs, require tons of disk space for who
knows what, take forever to start up, and chug along taking their sweet
time just to get the simplest of tasks done, where supposedly "older" or
"antiquated" tools fire up in an instant, get right to work, and get the
same task done in a fraction of the time.  This in itself may have been
excusable, but then these "modern" tools also come with a whole bunch of
red tape, arbitrary limitations, and the philosophy of "follow our way
or walk there on foot yourself".  IOW they do what "antiquated" tools do
poorly, and can't even do advanced things said "antiquated" tools can do
without any fuss -- and this not because of *technical* limitations, but
because it was arbitrarily decided at some point that use cases X and Y
will not be supported, just 'cuz.

This is a general problem of software, not just build systems such as Gradl=
e.
It does seem though that Meson + Ninja are much lighter weight.

So somebody please enlighten this here dinosaur of *why* I should choose
these "modern" tools over my existing, and IMO better, ones?

What are the better ones, one cannot debate without knowledge. ;-)

This is why I'm a fan of empowering the user.  Instead of being obsessed
over how to package your software in a beautiful jewel-encrusted box
with a gold ribbon on top, (which IMO is an utter waste of time and only
attracts those with an inflated sense of entitlement), give the user the
tools to do what the software can do. Instead of delivering a magic
black box that does everything but with the hood welded shut, give them
the components within the black box with which they can build something
far beyond what you may have initially conceived.  I believe *that* is
the way to progress, not this "hand me my software on a silver platter"
philosophy so pervasive nowadays.
=20
(The jewel-encrusted box is still nice, BTW -- but only as long as it's
not at the expense of welding the hood shut.)

UI and UX are important, definitely. The Groovy DSL and the Kotlin DSL for
writing build are focussed on convention over configuration so as to make
things as simple as possible for the straightforward cases. But the tools a=
re
there to configure and programme the build for complicated cases. In this
Gradle and SCons have similar capabilities though SCons lack many things bu=
ilt
in to Gradle.

There are probably 10x as many build requirements as there are programmers.=
A
single unopenable black box build system simply has no chance.=20

[=E2=80=A6]
=20
If Gradle is dependent on Java, then using it for D builds would be kind
of ... anticlimactic. :-D  But probably the primary issue would be the
slow startup times and heavy storage requirements that goes against the
grain of our present (cringe-worthy) fast-fast-fast slogan. I suppose
it's no coincidence that the Gradle logo is ... just as slow and mellow
as the tool itself. ;-)  (I love RL elephants, BTW, so this isn't meant
in a denigrating way. But I'd have a very hard time justifying the
order-of-magnitude increase in my edit-compile-run cycle turnaround
times.  Life is far too short to have to wait for such things.)

Gradle is JVM-based, but they have build servers and caching to make builds
faster than you might think. Clearly the bulk of build focus is on JVM-rela=
ted=20
stuff, but C and C++ got added because someone paid for it to be added.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 27 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/16/19 1:39 PM, H. S. Teoh wrote:
On Mon, Apr 15, 2019 at 07:52:07PM -0400, Nick Sabalausky (Abscissa) >>
Then, with that information, a package manager provides services such
as (but not necessarily limited to):

1. A simple, standardized way for you and your users to obtain/build
the dependencies.

2. A simple, standardized way for buildscripts/buildsystems to obtain
the information needed to include the dependencies in their own build
(such as -I... include directories, paths to the now-already-built
lib/exec binaries, etc.)

I'd add:

3. A standard query interface for querying a remote repository for
packages (matching some names / patterns) and version numbers.

Oh, don't get me wrong, there's a whole BUNCH of things I'd like to see
on this particular list, and that definitely includes your suggestion. I
left it at "not necessarily limited to..." for good reason ;)

From this, each project can naturally either just roll its own
buildscripts, or depend on another package providing a builsystem.

That's a good idea.  Completely decouple package management from
building.  Let the package manager do what it does best: managing
packages, and leave the compilation to another tool more suited for the
job.

Exactly.

I was thinking the build/test/etc. command can itself be defined by a
dependency.  For example, if your project uses make for builds, there
could be a make dub package that defines command-line templates used
for generating platform-specific build commands. Your package would
simply specify:

"build-system": "make"

and the "make" package would define the exact command(s) for invoking
make [...etc...]

Interesting idea. I definitely like it, though I would consider it not a
"basic fundamental" mechanism, but rather a really nice bonus feature
that provides a convenient shortcut for having to specifying certain
things manually...IF you happen to be using a buildsystem known by the
package manager...or (better yet, if possible) a buildsystem that's able
to tell the package manager enough about itself to make this happen.

Some of the *details* can be quite nontrivial...like dependency
resolution algorithms, or designing the interactions between package
manager and buildsystem to be simple, yet effective enough to suit all
parties needs.  But ultimately, it boils down conceptually to be very
simple.

[...]

If done correctly, dependency resolution is just a topological walk on a
DAG.  Get this right, and everything else falls into place.

Yes, but therin lies the rub - "getting this right". AFAICT, Sonke seems
to have a very good grasp on dependency resolution via DAG (or at least,
certainly a far better grasp than I do). But yet, even his algorithms in
DUB, he needed to occasionally patch with common-use-case
short-circuiting logic (and I don't even know what else) just to fix
various seemingly-infinite-loop issues (not technically infinite, but
appeared to be to the user) that people were still reporting some years
into DUBs lifetime.

'Course, if it really is simpler than it seems to me (and *especially*
is it *isn't*), then I'd certainly be more than happy to delegate
dependency resolution out to somebody else.

Apr 17 2019
Russel Winder <russel winder.org.uk> writes:
On Wed, 2019-04-17 at 03:48 -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
=20

[=E2=80=A6]
=20
Oh, don't get me wrong, there's a whole BUNCH of things I'd like to
see=20
on this particular list, and that definitely includes your
suggestion. I=20
left it at "not necessarily limited to..." for good reason ;)

[=E2=80=A6]

The problem here is that people are having lots of ideas on this email
list, but no-one is choosing to collate them into a an actual "call to
action". As ever in an ever extending email thread the energy to do
something within the D community dissipates and nothing ends up
happening.

Perhaps we should make the hack day at DConf 2019 "get something
actually moving on this" day.

Or perhaps people will continue to say stuff on very long email threads
and end up doing nothing. This is the third such thread on Dub I can
remember, the previous two ended up with no  outcomes other than lots
of opinions in very long email threads.
=20
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

Apr 17 2019
NaN <divide by.zero> writes:
On Wednesday, 17 April 2019 at 08:06:13 UTC, Russel Winder wrote:
On Wed, 2019-04-17 at 03:48 -0400, Nick Sabalausky (Abscissa)
via
Digitalmars-d wrote:

[…]

Oh, don't get me wrong, there's a whole BUNCH of things I'd
like to
see
on this particular list, and that definitely includes your
suggestion. I
left it at "not necessarily limited to..." for good reason ;)

[…]

The problem here is that people are having lots of ideas on
this email list, but no-one is choosing to collate them into a
an actual "call to action". As ever in an ever extending email
thread the energy to do something within the D community
dissipates and nothing ends up happening.

Perhaps we should make the hack day at DConf 2019 "get
something actually moving on this" day.

Or perhaps people will continue to say stuff on very long email
threads
and end up doing nothing. This is the third such thread on Dub
I can
remember, the previous two ended up with no  outcomes other
than lots
of opinions in very long email threads.

The people who are actually going to do the work should setup a
private group somewhere. These sprawling threads on the public
newsgroup dilute not just the vision but also the motivation.

Apr 17 2019
rikki cattermole <rikki cattermole.co.nz> writes:
On 17/04/2019 8:27 PM, NaN wrote:
On Wednesday, 17 April 2019 at 08:06:13 UTC, Russel Winder wrote:
On Wed, 2019-04-17 at 03:48 -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:

[…]
Oh, don't get me wrong, there's a whole BUNCH of things I'd like to
see
on this particular list, and that definitely includes your
suggestion. I
left it at "not necessarily limited to..." for good reason ;)

[…]

The problem here is that people are having lots of ideas on this email
list, but no-one is choosing to collate them into a an actual "call to
action". As ever in an ever extending email thread the energy to do
something within the D community dissipates and nothing ends up
happening.

Perhaps we should make the hack day at DConf 2019 "get something
actually moving on this" day.

Or perhaps people will continue to say stuff on very long email threads
and end up doing nothing. This is the third such thread on Dub I can
remember, the previous two ended up with no  outcomes other than lots
of opinions in very long email threads.

The people who are actually going to do the work should setup a private
group somewhere. These sprawling threads on the public newsgroup dilute
not just the vision but also the motivation.

Over on Discord we have the Graphics work group which is working out
quite well. Still very early on work wise so we've kept quite quiet
about it.

We have done our best to set aside our opinions and do research to find
out the requirements before writing code with a majority in agreement
before decisions / pulling of code. It would be a good model to
replicate if a few people came together and decided to start work on a
new (although compatible) build system.

Apr 17 2019
bachmeier <no spam.net> writes:
On Wednesday, 17 April 2019 at 08:06:13 UTC, Russel Winder wrote:

Or perhaps people will continue to say stuff on very long email
threads
and end up doing nothing. This is the third such thread on Dub
I can
remember, the previous two ended up with no  outcomes other
than lots
of opinions in very long email threads.

This is a leadership problem. Notably absent from these
discussions are Andrei and Walter. This is a critical piece of
infrastructure that's not working, it's distributed with DMD, and
they're completely silent because it's not something they find
interesting.

If you're the leader of a large open source project, now is when
you step in and ask who is willing to put in the time to take
this on, so you can give one or more of them the authority to
make decisions. There should be a formal proposal that can be
commented on as we do with a DIP. Work begins, with a formal
definition of when it's ready to go live guiding those doing the
work.

Maybe we'll see something like that happen. Based on what I've
seen over the last six years, it seems unlikely. Instead there
will be long email threads by individuals that have no
decisionmaking authority, and everyone will conclude that they'd
be crazy to dump hundreds of hours into such a project. In a
couple months we'll have a fourth long email thread on ways to
fix Dub.

Apr 17 2019
Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 17 April 2019 at 13:41:43 UTC, bachmeier wrote:
On Wednesday, 17 April 2019 at 08:06:13 UTC, Russel Winder
wrote:

Or perhaps people will continue to say stuff on very long
email threads
and end up doing nothing. This is the third such thread on Dub
I can
remember, the previous two ended up with no  outcomes other
than lots
of opinions in very long email threads.

This is a leadership problem. Notably absent from these
discussions are Andrei and Walter. This is a critical piece of
infrastructure that's not working, it's distributed with DMD,
and they're completely silent because it's not something they
find interesting.

If you're the leader of a large open source project, now is
when you step in and ask who is willing to put in the time to
take this on, so you can give one or more of them the authority
to make decisions. There should be a formal proposal that can
be commented on as we do with a DIP. Work begins, with a formal
definition of when it's ready to go live guiding those doing
the work.

Maybe we'll see something like that happen. Based on what I've
seen over the last six years, it seems unlikely. Instead there
will be long email threads by individuals that have no
decisionmaking authority, and everyone will conclude that
they'd be crazy to dump hundreds of hours into such a project.
In a couple months we'll have a fourth long email thread on
ways to fix Dub.

Please be more precise. Dub does a perfect job if you have
standalone D applications. There are issues, as I understand, if
you have complex integration scenarios with C/C++.

Dub is built by the community for the community. I also had to
create some pull reduests to get all my scenarios working and now
everything works for my scenarios like a charme.

Maybe for complex integration scenarios another build tool has to
be used. Or maybe Dub has to be improved to support complex
scenarios.

I do not see any management issue here.
If you (to all) have issues, please consider creating pull
requests.

Kind regards
Andre

Apr 17 2019
drug <drug2004 bk.ru> writes:
On 17.04.2019 17:09, Andre Pany wrote:

Please be more precise. Dub does a perfect job if you have standalone D
applications. There are issues, as I understand, if you have complex
integration scenarios with C/C++.

Dub is built by the community for the community. I also had to create
some pull reduests to get all my scenarios working and now everything
works for my scenarios like a charme.

Maybe for complex integration scenarios another build tool has to be
used. Or maybe Dub has to be improved to support complex scenarios.

I do not see any management issue here.
If you (to all) have issues, please consider creating pull requests.

Kind regards
Andre

I agree with you totally. I don't understand why W&A should appear in
this thread. dub wasn't intended to be standard package manager. And
it's absolutely normally that now community needs to improve it. There
is no someone fault at all.

Apr 17 2019
bachmeier <no spam.net> writes:
On Wednesday, 17 April 2019 at 14:09:06 UTC, Andre Pany wrote:

Please be more precise. Dub does a perfect job if you have
standalone D applications. There are issues, as I understand,
if you have complex integration scenarios with C/C++.

Dub is built by the community for the community. I also had to
create some pull reduests to get all my scenarios working and
now everything works for my scenarios like a charme.

Maybe for complex integration scenarios another build tool has
to be used. Or maybe Dub has to be improved to support complex
scenarios.

I do not see any management issue here.
If you (to all) have issues, please consider creating pull
requests.

Well clearly others disagree with you. You are saying "it works
for me - make changes to the source code and submit a PR if you
don't like it."

That is not an answer. It excludes anyone new to D, anyone that
uses D but does not have time to work on Dub, or anyone that
doesn't have the ability to make changes to Dub as needed for
acceptance. No such requirement exists for using any other
language to my knowledge, and I've used a lot of languages over
the years. If this is the path that will be taken, there is no
point speculating why the language isn't used more, especially in
commercial settings. There's not even a starting point for using
the language if these issues are not resolved.

Dub is an official project because it is included with DMD and it
is the recommended (and for the most part only) way to distribute
D code. It needs to be taken seriously.

Apr 17 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/17/19 4:06 AM, Russel Winder wrote:
On Wed, 2019-04-17 at 03:48 -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:

[…]
Oh, don't get me wrong, there's a whole BUNCH of things I'd like to
see
on this particular list, and that definitely includes your
suggestion. I
left it at "not necessarily limited to..." for good reason ;)

[…]

The problem here is that people are having lots of ideas on this email
list, but no-one is choosing to collate them into a an actual "call to
action".

FWIW, the only reason I was being non-canonical about desired features
in a package manager is that (from experience) I wanted to avoid
pie-in-the-sky up-front design and focus first on the core necessities -
additional niceties can come later, on top of a solid working core.

As ever in an ever extending email thread the energy to do
something within the D community dissipates and nothing ends up
happening.

Actually, I've been fired up enough about this for a good while to put
my code where my mouth is (I just have a bad habit of spreading myself
too thin dev-wise, and having too much other life crap getting in the
way). And, personally, I find this to be D's biggest current stumbling
block, even moreso than anything in the language or compiler (which is
probably good that I feel that way, because I'm pretty much
out-of-my-league when it comes to hacking on the compiler). So note my
next post in this thread...

Apr 17 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 17, 2019 at 03:48:36AM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
On 4/16/19 1:39 PM, H. S. Teoh wrote:

[...]
I was thinking the build/test/etc. command can itself be defined by
a dependency.  For example, if your project uses make for builds,
there could be a make dub package that defines command-line
templates used for generating platform-specific build commands. Your
package would simply specify:

"build-system": "make"

and the "make" package would define the exact command(s) for
invoking make [...etc...]

Interesting idea. I definitely like it, though I would consider it not
a "basic fundamental" mechanism, but rather a really nice bonus
feature that provides a convenient shortcut for having to specifying
certain things manually...IF you happen to be using a buildsystem
known by the package manager...or (better yet, if possible) a
buildsystem that's able to tell the package manager enough about
itself to make this happen.

The idea behind this is that in order to avoid baking in support for a
multitude of build systems (make, scons, meson, gradle, whatever) into
dub, which will inevitably be unmaintained and out-of-date 'cos is too
much work to keep up with everything, we delegate the job to individual
packages.  There would be one package per external build system, e.g., a
'make' package for make support, an 'scons' package for scons support,
and so on.  These packages don't have to do very much, all they need is
to (1) define a way of installing said build system, in a way that can
be used by dub via a standard interface, and (2) export various
command-line templates that dub will use for invoking the build system.

So for example, the 'make' package might contain a bunch of URLs for
downloading a 'make' executable and any associated libraries and data
files, and then export /path/to/dub/cache/bin/make as the command
template for invoking the build system.  There could be OS-dependent
variations, such as C:\path\to\dub\cache\make\gmake.exe if you're on
Windows, for example.  These command flavors are defined within the
'make' dub package, and then any project that uses make can just declare
a dependency on 'make', and dub will know how to do the Right Thing(tm).
Dub itself wouldn't have to know the specifics of how to work with a
make-based source tree; it will just execute the commands defined by the
'make' dub package.

[...]
If done correctly, dependency resolution is just a topological walk
on a DAG.  Get this right, and everything else falls into place.

Yes, but therin lies the rub - "getting this right". AFAICT, Sonke
seems to have a very good grasp on dependency resolution via DAG (or
at least, certainly a far better grasp than I do). But yet, even his
algorithms in DUB, he needed to occasionally patch with
common-use-case short-circuiting logic (and I don't even know what
else) just to fix various seemingly-infinite-loop issues (not
technically infinite, but appeared to be to the user) that people were
still reporting some years into DUBs lifetime.

'Course, if it really is simpler than it seems to me (and *especially*
is it *isn't*), then I'd certainly be more than happy to delegate
dependency resolution out to somebody else.

It would help if somebody can provide a concrete example of a complex
dependency graph that requires anything more than a straightforward
topological walk.

T

--
MS Windows: 64-bit rehash of 32-bit extensions and a graphical shell for a
16-bit patch to an 8-bit operating system originally coded for a 4-bit
microprocessor, written by a 2-bit company that can't stand 1-bit of
competition.

Apr 17 2019
drug <drug2004 bk.ru> writes:
On 17.04.2019 19:03, H. S. Teoh wrote:
If done correctly, dependency resolution is just a topological walk
on a DAG.  Get this right, and everything else falls into place.

Yes, but therin lies the rub - "getting this right". AFAICT, Sonke
seems to have a very good grasp on dependency resolution via DAG (or
at least, certainly a far better grasp than I do). But yet, even his
algorithms in DUB, he needed to occasionally patch with
common-use-case short-circuiting logic (and I don't even know what
else) just to fix various seemingly-infinite-loop issues (not
technically infinite, but appeared to be to the user) that people were
still reporting some years into DUBs lifetime.

'Course, if it really is simpler than it seems to me (and *especially*
is it *isn't*), then I'd certainly be more than happy to delegate
dependency resolution out to somebody else.

It would help if somebody can provide a concrete example of a complex
dependency graph that requires anything more than a straightforward
topological walk.

T

Couple of years ago I've made an example -
https://github.com/drug007/dub-reduced-case
Now dub works in other way than that time but nevertheless.
Now it says:

drug drug:/tmp/dub-reduced-case/rdpl$dub build :inspector Building package rdpl:inspector in /tmp/dub-reduced-case/rdpl/inspector/ Root package rdpl:core reference gfm:math ~>6.1.4 cannot be satisfied. Packages causing the conflict: rdpl:core depends on ~>6.1.4 timespatial depends on ~>6.0.0 timespatial depends on ~>6.0.0 timespatial depends on ~>6.0.0  shouldn't dub select gfm:math of version 6.1.4 here?  Apr 17 2019 Dragos Carp <dragoscarp gmail.com> writes: On Wednesday, 17 April 2019 at 16:03:24 UTC, H. S. Teoh wrote: The idea behind this is that in order to avoid baking in support for a multitude of build systems (make, scons, meson, gradle, whatever) into dub, which will inevitably be unmaintained and out-of-date 'cos is too much work to keep up with everything, we delegate the job to individual packages. There would be one package per external build system, e.g., a 'make' package for make support, an 'scons' package for scons support, and so on. These packages don't have to do very much, all they need is to (1) define a way of installing said build system, in a way that can be used by dub via a standard interface, and (2) export various command-line templates that dub will use for invoking the build system. [...] Invoking the external tools is probably doable, the problem is consuming artifacts generated by these tools. Then there are the specialties of each tool: some distinguish between configuration (done once for a fresh build) and build, some are very good at tracking the dependencies and incremental builds, some rediscover the world on each invocation, some support cross-compiling, some use command line parameter for customizing the build, other use configuration files, some support debug builds, some support listing the build targets, some support a server mode, some require a server mode, some deprecate features over time, etc., etc. Finally, it's a mess. There is an effort in Scala world to do something in the same direction (they have a lot of build systems): https://github.com/scalacenter/bsp/blob/master/docs/bsp.md. But this effort resumes itself to just IDE integration. What I'm missing is the tool for building D projects, do you suggest that each project should use what it seams fit, or just stick with dub? It would help if somebody can provide a concrete example of a complex dependency graph that requires anything more than a straightforward topological walk. I don't know if I understand you right, but probably having some generated code, or calling shell scripts with side-effects already complicates the problem enough.  Apr 17 2019 "H. S. Teoh" <hsteoh quickfur.ath.cx> writes: On Wed, Apr 17, 2019 at 04:51:57PM +0000, Dragos Carp via Digitalmars-d wrote: On Wednesday, 17 April 2019 at 16:03:24 UTC, H. S. Teoh wrote: The idea behind this is that in order to avoid baking in support for a multitude of build systems (make, scons, meson, gradle, whatever) into dub, which will inevitably be unmaintained and out-of-date 'cos is too much work to keep up with everything, we delegate the job to individual packages. There would be one package per external build system, e.g., a 'make' package for make support, an 'scons' package for scons support, and so on. These packages don't have to do very much, all they need is to (1) define a way of installing said build system, in a way that can be used by dub via a standard interface, and (2) export various command-line templates that dub will use for invoking the build system. [...] Invoking the external tools is probably doable, the problem is consuming artifacts generated by these tools. It's not really a problem, as far as dub is concerned. All we need is some way to export the list of generated artifacts. E.g., if it's a library package, all you really need is to export (1) the import paths and (2) the path(s) to the generated .so / .a / .lib / .dll files, and dub should be able to figure out how to make use of this library without actually knowing what the build command does. In other words, we hide the complexity of the build system under one or more canned invocation commands (e.g., 'make', 'make clean', 'make docs', 'make test', for a standard set of actions you might commonly want to do), and have the wrapper package tell us what the build artifacts are. Then there are the specialties of each tool: some distinguish between configuration (done once for a fresh build) and build, some are very good at tracking the dependencies and incremental builds, some rediscover the world on each invocation, some support cross-compiling, some use command line parameter for customizing the build, other use configuration files, some support debug builds, some support listing the build targets, some support a server mode, some require a server mode, some deprecate features over time, etc., etc. Finally, it's a mess. I don't think we need to support all of those cases. As far as dub-as-a-package-manager is concerned, all it really cares about is that each dependency is some black box amorphous blob of files, with an associated set of arbitrary shell commands for each standard action: build debug, build release, test, query import paths, query library paths, etc.. These actions should be standard across all codebases, and it doesn't matter how they are implemented underneath. They can be arbitrarily complex shell commands. The dirty details are abstracted away by the build system wrapper packages. As for cross-compilation, IMO this should have built-in support as an optional feature. I.e., there should be a standard command for "cross-compile to architecture XYZ", and packages can either support this or not. If one or more dependencies don't support cross-compilation, then you cannot cross-compile (with dub). There is an effort in Scala world to do something in the same direction (they have a lot of build systems): https://github.com/scalacenter/bsp/blob/master/docs/bsp.md. But this effort resumes itself to just IDE integration. What I'm missing is the tool for building D projects, do you suggest that each project should use what it seams fit, or just stick with dub? If you're asking me about the *current* state of dub, I have to say I don't use it except to pull in code.dlang.org dependencies. I.e., I find it only useful as a package manager and nothing else. As a build system it does not support critical use cases that I need,[*] so I find it completely unusable. [*] Specifically, these are not supported by the current version of dub AFAIK: (1) Arbitrary codegen (i.e., build a subset of D files into helper program H1, then invoke H1 on some data files to generate D code, then compile the resulting D code into the final executable); (2) Cross-compilation (e.g., Linux x86 host -> Android/ARM); (3) Arbitrary preprocessing of data files (build a subset of D files into helper program H2, then invoke H2 on some data input files to produce final data files -- e.g., compress HTML/Javascript, rescale images, etc.); (4) Building of deliverables (e.g., pack build products X,Y,Z into a .zip file, or build/sign an APK, or build a Debian .deb package, etc.). These are all non-trivial tasks, and I do not expect that dub will ever support all of them natively (though it would be nice if it allowed me to specify custom commands to accomplish such tasks!). That's why I'm pushing for separating dub as a package manager from dub as a build tool. The package manager's responsibility is to resolve dependencies, resolve version constraints, and download dependencies. How these dependencies are built should be handled by a *real* build system that can handle (1) to (4) above. That's why I'm proposing to encapsulate the task of building to a wrapper dub package that abstracts away the complexity. At the end of the day, it just boils down to (1) you have a bunch of input files, (2) there's some command that transforms these input files into output files, and (3) these output files reside somewhere in the filesystem. As long as you're told where the output files in (3) are, you know enough to use this package. What goes on inside (2) is irrelevant; said command can be arbitrarily complex -- it can be an entire build system, for example, that consists of many subcommmands -- but a package manager doesn't need to (and shouldn't) care about such details. It would help if somebody can provide a concrete example of a complex dependency graph that requires anything more than a straightforward topological walk. I don't know if I understand you right, but probably having some generated code, or calling shell scripts with side-effects already complicates the problem enough. This is only a problem if you're conflating package management with a build system. If you tackle solely the problem of package management, then these are not problems at all. As far as a package manager is concerned, you have a root package P that has dependencies on some packages A, B, C, that in turn depend on other packages D, E, F. Your job as package manager is to figure out which version(s) of A, B, C, D, E, F to download. Each package is an arbitrary collection of files, with an associated command that transforms these files into build products. All you need to know is a string containing the command that performs this transformation, and a list of paths to the build product(s). You don't care what files are in the package, really. For all you care they could be a bunch of URIs that point to remote objects, or a bunch of encrypted data files, or a .zip file of an entire OS. None of that is relevant. All that matters is that there's a command (specified within the package itself) that tells you how to transform said files into the build products. You just run this command and let it do its job -- how, you don't care. As long as the command produces the build products, that's good enough. Once it's done, you just query for the path(s) to the build products, and use that in the next node up the dependency tree. It's not the package manager's job to stick its grubby hands into the dirty details of the actual build process. T -- ASCII stupid question, getty stupid ANSI.  Apr 17 2019 Julian <julian.fondren gmail.com> writes: On Wednesday, 17 April 2019 at 17:53:35 UTC, H. S. Teoh wrote: In other words, we hide the complexity of the build system under one or more canned invocation commands (e.g., 'make', 'make clean', 'make docs', 'make test', for a standard set of actions you might commonly want to do), and have the wrapper package tell us what the build artifacts are. What you lose with this is dub's current platform independence. That's not always present anyway, for example a project could rely on specific cPanel or macOS features, but it would be nice to retain it as the default. Ada's gprbuild has project files like this: project Packetbl is for Object_Dir use "build"; for Exec_Dir use "bin"; for Source_Dirs use ("src"); for Languages use ("Ada", "C"); for Main use ("packetbl.adb"); package Compiler is for Default_Switches ("Ada") use ("-O3", "-gnata", "-gnaty-m", "-gnatwa"); for Default_Switches ("C") use ("-O3", "-Wall"); end Compiler; package Linker is for Default_Switches ("Ada") use ("-lnfnetlink", "-lnetfilter_queue", "-lanl"); end Linker; end Packetbl; So, this project uses Ada and C languages, a binary should be built with packetbl.adb as the main, and other language files under src/ are linked in. This project doesn't bother to list source files, but gprbuild knows what .adb/.ads and .c files are, so it builds them all with the appropriate compiler and then links them together. To know how to compile, gprbuild has a system configuration that includes the exact compilers to use for each language, their paths, etc. The sort of thing you'd set with 'dub init', with probable defaults, and with compiler knowledge (of how to invoke "gcc", "clang", etc.) built into dub.  Apr 17 2019 "H. S. Teoh" <hsteoh quickfur.ath.cx> writes: On Wed, Apr 17, 2019 at 06:17:25PM +0000, Julian via Digitalmars-d wrote: On Wednesday, 17 April 2019 at 17:53:35 UTC, H. S. Teoh wrote: In other words, we hide the complexity of the build system under one or more canned invocation commands (e.g., 'make', 'make clean', 'make docs', 'make test', for a standard set of actions you might commonly want to do), and have the wrapper package tell us what the build artifacts are. What you lose with this is dub's current platform independence. That's not always present anyway, for example a project could rely on specific cPanel or macOS features, but it would be nice to retain it as the default. [...] Actually, this is precisely why I said that build systems would have their own wrapper packages. The wrapper packages would specify potentially different commands to run, depending on the current platform. Since the wrapper package would have specific knowledge about the build system it's wrapping, it could craft the build command such that it sets up the appropriate platform-dependent settings. E.g., the 'make' wrapper would specify a command that sets up INCPATH to /usr/include on Linux, for example, and C:\blah\blah\include on Windows. In fact, we could even use this as a mechanism to support cross compilation. E.g., given the current platform and the target platform as parameters, the wrapper package would craft a build command that sets up the appropriate path(s) and environment variables for cross compilation. This is an optional bonus feature, of course. The point is that the build system wrapper packages would hide away these details from dub, so dub doesn't have to know about how to set up the build for Windows vs. Linux. It just installs the wrapper package and queries it for the right command to use. T -- Frank disagreement binds closer than feigned agreement.  Apr 17 2019 "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes: On 4/17/19 1:53 PM, H. S. Teoh wrote: At the end of the day, it just boils down to (1) you have a bunch of input files, (2) there's some command that transforms these input files into output files, and (3) these output files reside somewhere in the filesystem. As long as you're told where the output files in (3) are, you know enough to use this package. What goes on inside (2) is irrelevant; said command can be arbitrarily complex -- it can be an entire build system, for example, that consists of many subcommmands -- but a package manager doesn't need to (and shouldn't) care about such details. +1, exactly correct  Apr 17 2019 JN <666total wp.pl> writes: On Wednesday, 17 April 2019 at 19:15:40 UTC, Nick Sabalausky (Abscissa) wrote: On 4/17/19 1:53 PM, H. S. Teoh wrote: At the end of the day, it just boils down to (1) you have a bunch of input files, (2) there's some command that transforms these input files into output files, and (3) these output files reside somewhere in the filesystem. As long as you're told where the output files in (3) are, you know enough to use this package. What goes on inside (2) is irrelevant; said command can be arbitrarily complex -- it can be an entire build system, for example, that consists of many subcommmands -- but a package manager doesn't need to (and shouldn't) care about such details. +1, exactly correct In the ideal world, perhaps. I am just worried opening dub to other build systems will just open a can of worms. Packages will use different build systems, which will be a pain to set up. Right now, the advice is "to get started with D, install official DMD distribution and set up your project with dub". Afterwards, the advice may be "to get started with D, install official DMD distribution, set up your project with dub, and install git,svn,meson,scons,make,cmake so that you can build some packages. Oh and set these environmental variables for these tools. Oh and if you're on Windows you're screwed". While dub may seem limiting to some, it offers standarization and unification. I know these are forbidden words in D community, but standards and forcing people to do things certain way are often beneficial and boost the cohesion of the ecosystem. "I find this to be D's biggest current stumbling block, even moreso than anything in the language or compiler" - really? The biggest D stumbling block is the fact that it's not easy to set up a mixed C/C++ project with D? How many projects of that kind are there even? How about issues such as crappy std.xml long overdue for replacement (I could list more but I don't want to derail this thread)?  Apr 17 2019 Julian <julian.fondren gmail.com> writes: On Wednesday, 17 April 2019 at 20:10:08 UTC, JN wrote: "I find this to be D's biggest current stumbling block, even moreso than anything in the language or compiler" - really? The biggest D stumbling block is the fact that it's not easy to set up a mixed C/C++ project with D? How many projects of that kind are there even? How about issues such as crappy std.xml long overdue for replacement (I could list more but I don't want to derail this thread)? Is there no good C or C++ XML library that you could make a std.xml alternative out of? Maybe there'd be a dub package that you could just grab and use for now, that wrapped one, if this were an option for dub. Consider std.regex and https://github.com/jrfondren/topsender-bench The fastest std.regex option is more than 10x slower than libc regex, which is already too slow to seriously use for anything but once-off tasks. (I put a thread in learn about this. None of the other suggestions there made any significant change to performance.) When the first thing someone'd try is ** 140x ** slower than the Python script they didn't even think about optimizing, I can't say it benefits D that nobody can say "oh yeah just dub add pcre and change your import"  Apr 17 2019 Julian <julian.fondren gmail.com> writes: On Wednesday, 17 April 2019 at 20:32:28 UTC, Julian wrote: When the first thing someone'd try is ** 140x ** slower than the Python script they didn't even think about optimizing 37x, rather. It's still 2s ("I can hit enter and wait for the output") vs. over a minute ("I'd better take a 10min break while this runs.")  Apr 17 2019 Russel Winder <russel winder.org.uk> writes: On Wed, 2019-04-17 at 20:32 +0000, Julian via Digitalmars-d wrote: =20 [=E2=80=A6] Is there no good C or C++ XML library that you could make a=20 std.xml alternative out of? Maybe there'd be a dub package that you could just grab and use for now, that wrapped one, if this were an=20 option for dub. [=E2=80=A6] Isn't the current done thing just to make use of libxml2 ? --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk  Apr 18 2019 Dmitry Olshansky <dmitry.olsh gmail.com> writes: On Wednesday, 17 April 2019 at 20:32:28 UTC, Julian wrote: On Wednesday, 17 April 2019 at 20:10:08 UTC, JN wrote: Consider std.regex and https://github.com/jrfondren/topsender-bench The fastest std.regex option is more than 10x slower than libc regex, which is already too slow to seriously use for anything but once-off tasks. Author of std.regex here. It's been awhile since I monitored its performance. Still, let's drill down. Caveats emptor: I do not have your datafile, but I produced one by sampling example lines from the web. First, your flags are a bit off for DMD, use the following: dmd -release -inline -O I know, not very intuitive. This more then doubles performance with dmd, std.regex is templated library (sadly) so it heavily depends on passing the right flags at the application level :( For LDC I used the following (can't remember if -O implies -release): ldc2 -release -O Second, if doing match per line regex it's best to use matchFirst instead of match which caches the engine in between calls. match is intended to plow through large chunks such as iterating over matches in memmory-mapped file and therefore creates new engine on each call to match. With these two tweaks I get a respectful speed of 1.5x of PCRE/JIT. IIRC PCRE_JIT option doesn't work for Unicode and std.regex supports Unicode by default. In general I agree - std.regex needs more love, a casual look at disasembly shows some degradation compared to a couple years back. Truth is, code like that needs constant tweaking. P.S. I lack time or energy to improve on regex esp. in std proper. I hope to get back to my experiments on rewind-regex though. JIT compilation is on the list, mostly to avoid reliance on compiler + being more aggressive on low-level tricks.  Apr 25 2019 Bastiaan Veelo <Bastiaan Veelo.net> writes: On Thursday, 25 April 2019 at 16:49:27 UTC, Dmitry Olshansky wrote: P.S. I lack time or energy to improve on regex esp. in std proper. I hope to get back to my experiments on rewind-regex though. JIT compilation is on the list, mostly to avoid reliance on compiler + being more aggressive on low-level tricks. Hey Dmitry, nice to see you again! Bastiaan.  Apr 25 2019 Dmitry Olshansky <dmitry.olsh gmail.com> writes: On Thursday, 25 April 2019 at 20:13:02 UTC, Bastiaan Veelo wrote: On Thursday, 25 April 2019 at 16:49:27 UTC, Dmitry Olshansky wrote: P.S. I lack time or energy to improve on regex esp. in std proper. I hope to get back to my experiments on rewind-regex though. JIT compilation is on the list, mostly to avoid reliance on compiler + being more aggressive on low-level tricks. Hey Dmitry, nice to see you again! Bastiaan. Thanks, I look forward to your DConf talk ;)  Apr 26 2019 "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes: On 4/17/19 4:10 PM, JN wrote: On Wednesday, 17 April 2019 at 19:15:40 UTC, Nick Sabalausky (Abscissa) wrote: On 4/17/19 1:53 PM, H. S. Teoh wrote: At the end of the day, it just boils down to (1) you have a bunch of input files, (2) there's some command that transforms these input files into output files, and (3) these output files reside somewhere in the filesystem. As long as you're told where the output files in (3) are, you know enough to use this package. What goes on inside (2) is irrelevant; said command can be arbitrarily complex -- it can be an entire build system, for example, that consists of many subcommmands -- but a package manager doesn't need to (and shouldn't) care about such details. +1, exactly correct In the ideal world, perhaps. No, there's no "ideal world" about this. What HSTeoh describes above are just plain the cold, hard facts of the matter. No ifs, ands or buts. The complications that we're accustomed to seeing and expecting are ones that come straight from DUB's failure to fully separate package management from building. I am just worried opening dub to other build systems will just open a can of worms. Packages will use different build systems, which will be a pain to set up. Right now, the advice is "to get started with D, install official DMD distribution and set up your project with dub". Afterwards, the advice may be "to get started with D, install official DMD distribution, set up your project with dub, and install git,svn,meson,scons,make,cmake so that you can build some packages. Oh and set these environmental variables for these tools. Oh and if you're on Windows you're screwed". With what I have in mind (and HS Teoh appears to have the same thing in mind), then no, those scenarios are absolutely, definitely not a realistic result. They're just fear-based speculation at best. In the long run, DUB may or may not remain the recommended buildsystem for newcomers, but that's all. And it least it won't actively PREVENT alternative buildsystems from coming along and proving superior enough to dethrone DUB. While dub may seem limiting to some, it offers standarization and unification. Well that's just the problem - It *doesn't* provide that, it completely fails to. It tries to standardize and unify through basically the same strong-arm "submit or die" approach as Nobunaga or Napoleon, but as I predicted it would from the start, it just would up fracturing the ecosystem instead[1]. And a fractured ecosystem is by definition NEITHER standardized NOR unified. [1] The no-so-well-known fact is, many projects are forced out of the DUB package ecosystem because of DUB buildsystem's limitations. Though that may not be easy to see, since such packages are forced under-the-radar because of their opting out of DUB as a buildsystem also forces them out of the visibility provided by code.dlang.org. And then the ones that do join the DUB side are too often forced into hardships that first of all, shouldn't even be necessary, and second of all, still fail to offer the tradeoff of unifying/standardizing the ecosystem anyway. The *only* way to standardize and unify is to do so in a way that allows individual packages to work the way they need to.  Apr 17 2019 "H. S. Teoh" <hsteoh quickfur.ath.cx> writes: On Wed, Apr 17, 2019 at 08:10:08PM +0000, JN via Digitalmars-d wrote: [...] In the ideal world, perhaps. I am just worried opening dub to other build systems will just open a can of worms. Packages will use different build systems, which will be a pain to set up. Right now, the advice is "to get started with D, install official DMD distribution and set up your project with dub". Afterwards, the advice may be "to get started with D, install official DMD distribution, set up your project with dub, and install git,svn,meson,scons,make,cmake so that you can build some packages. Oh and set these environmental variables for these tools. Oh and if you're on Windows you're screwed". Again, this concern is addressed by my proposal of build system wrapper dub packages. Basically, the act of declaring a dependency on, say, the 'make' wrapper package, should cause dub to download and install a working installation of make. This can be done by structuring the 'make' wrapper package such that its "build" command is to download and install a working version of make. In a nutshell, it would work like this: - Project P depends on project Q, so dub downloads project Q. - Project Q depends on 'make', so dub downloads the 'make' wrapper package. - The 'make' wrapper package's "build" command is actually an installer that downloads and installs make on the local system. For maximum non-intrusion, the installation location can be somewhere in a standard location in the dub cache, so it won't conflict with any other make installation you may have on your system. So dub invokes this "build" command, and now you have a working version of make. (Note that this includes setting up any environment variables that may be necessary.) - Now that project Q's dependencies are satisfied, it invokes Q's build command, which runs make to build everything. This now works, because the previous step ensures we have a working version of make installed. - Now that project P's dependencies are satisfied, we can build P as usual. This is one example of why abstracting away the "build" command of a package from the package manager can be highly beneficial: it allows you to pull in any arbitrary external package without needing dub to be aware of how said external package works. In the above scheme, you can substitute 'make' with any build system of your choice, and it will still work with no additional effort from the end user. This is ensured by the wrapper package abstracting away the dirty details of how to install a working version of${buildtool} so
that dub can perform the installation without needing to know the
details of how to do it.

The wrapper package also hides away the dirty details of how to setup a
build tool like make properly, including setting up environment
variables and what-not.  Basically, it does whatever it takes for the
whole process to be completely transparent to the user.

While dub may seem limiting to some, it offers standarization and
unification. I know these are forbidden words in D community, but
standards and forcing people to do things certain way are often
beneficial and boost the cohesion of the ecosystem.

Standards should be empowering, rather than arbitrarily limiting.  What
we're proposing here is a standard way for arbitrary software packages
to be used as dub dependencies.  Let me repeat that again: what we're
proposing here is a *standard* way for *arbitrary* software packages (of
any language, any build system, etc.) to be usable as dub dependencies.

The standard here is the standard "API" of how packages work:

1) A package can depend on one or more packages, with optional version
constraints. This is the way dub already works.

2) A package can consist of any arbitrary collection of input files.
What they are and how they are structured is irrelevant.

3) A package has a standard declaration of how it should be built.
Basically, this is any arbitrary command that transforms said input
files into output files. How this is carried out is irrelevant -- that's
a detail the package is responsible for, and is none of the package
manager's business.

4) A package has a standard declaration of where to find the built
products (import paths, library files, etc.).

5) A package that wraps around a build system has a standard declaration
of how to formulate commands to invoke it for various standard tasks
(build debug, build release, test, etc.). So dependent packages don't
actually have to know how to run make on Windows vs. Linux, it uses a
standard query to formulate the right command, and simply runs it. The
wrapper package ensures that this command will Just Work(tm).

Just like the range API, the user of the range (the package manager)
doesn't, and shouldn't, care about how each of these "methods" work
under the hood.  As long as they conform to the API (declare items (1)
to (4) in a standard way), that's all that's necessary for dub to do the
Right Thing(tm).

"I find this to be D's biggest current stumbling block, even moreso
than anything in the language or compiler" - really? The biggest D
stumbling block is the fact that it's not easy to set up a mixed C/C++
project with D?  How many projects of that kind are there even?

Given the amount of effort currently being put into C++ interop?  I'm
guessing it's a lot more common than you think.

How about issues such as crappy std.xml long overdue for replacement
(I could list more but I don't want to derail this thread)?

Jonathan's dxml is already a dub package, isn't it?  You could just use
that.

T

--
If it tastes good, it's probably bad for you.

Apr 17 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/17/19 4:10 PM, JN wrote:
"I find this to be D's biggest current stumbling block, even moreso than
anything in the language or compiler" - really? The biggest D stumbling
block is the fact that it's not easy to set up a mixed C/C++ project
with D?

Oh dear god that is not even *REMOTELY* in the ballpark of actual
reality. Lets keep the blatant strawmen far away from this, please.

Apr 17 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/17/19 12:03 PM, H. S. Teoh wrote:
On Wed, Apr 17, 2019 at 03:48:36AM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
On 4/16/19 1:39 PM, H. S. Teoh wrote:

[...]
I was thinking the build/test/etc. command can itself be defined by
a dependency.  For example, if your project uses make for builds,
there could be a make dub package that defines command-line
templates used for generating platform-specific build commands. Your
package would simply specify:

"build-system": "make"

and the "make" package would define the exact command(s) for
invoking make [...etc...]

Interesting idea. I definitely like it, though I would consider it not
a "basic fundamental" mechanism, but rather a really nice bonus
feature that provides a convenient shortcut for having to specifying
certain things manually...IF you happen to be using a buildsystem
known by the package manager...or (better yet, if possible) a
buildsystem that's able to tell the package manager enough about
itself to make this happen.

The idea behind this is that in order to avoid baking in support for a
multitude of build systems (make, scons, meson, gradle, whatever) into
dub, which will inevitably be unmaintained and out-of-date 'cos is too
much work to keep up with everything, we delegate the job to individual
packages.

I agree, letting build system packages tell the packmangr how to use
them is definitely preferable to baked-in support.

So for example, the 'make' package might contain a bunch of URLs for
downloading a 'make' executable and any associated libraries and data
files, and then export /path/to/dub/cache/bin/make as the command
template for invoking the build system.  There could be OS-dependent
variations, such as C:\path\to\dub\cache\make\gmake.exe if you're on
Windows, for example.  These command flavors are defined within the
'make' dub package, and then any project that uses make can just declare
a dependency on 'make', and dub will know how to do the Right Thing(tm).
Dub itself wouldn't have to know the specifics of how to work with a
make-based source tree; it will just execute the commands defined by the
'make' dub package.

Yes, in general, pre-built binary tool packages that originate from
outside (or inside) D-land must be supported. This has been one of my
bigger issues with DUB. And a little bit extra consideration for
buildsystems in particular is likely warranted.

Only change I would make is that declaring a dependency on a buildsystem
shouldn't *necessarily* mean that the package manager invokes the
buildsystem directly. In many cases it would, and could maybe be a good
default, but it also needs to support the case where a project uses its
own buildscript and the buildscript invokes something like make or such
(probably, in this situation, the buildscript would invoke the
buildsystem - or any other executable tools it relies on - through the
package manager...or at least through information provided by the
package manager via envvars, for example).

Another thing to keep in mind is that the interactions between package
manager and buildsystem will likely require more than just the
buildsystem telling the package manager "Here are the command(s) you use
to invoke me". For example, the package manager may need to relay things
like "Which configuration is being built". Although, now that I think of
it, since the package manager is banned by charter from being a
buildsystem, it might not need to pass nearly as much information to
buildscripts/systems/etc as DUB-as-a-pack-manager would need to.

It would help if somebody can provide a concrete example of a complex
dependency graph that requires anything more than a straightforward
topological walk.

Personally, in the absence of someone like Sonke who's already
experienced with real-world dependency resolution popping in to help out
on that, I'd really like to just simply start by adapting and re-using
some already-established dependency-resolving code, such as from DUB.
This is one thing I'd really want to avoid reinventing the wheel on if
at all possible.

--------------------------------

In response to suggestions from NaN and Russel, I'd like to move this to
a Github project.  No code yet, just because I don't have any yet, but
we can use the issues/PRs/wiki features to collaborate publicly in a
more structured, less transient, way:

https://github.com/Abscissa/DPak

H. S. Teoh: I forget your Github handle. Let me know and I'll add you
as a collaborator.

Apr 17 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 17, 2019 at 02:57:05PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
[...]
In response to suggestions from NaN and Russel, I'd like to move this
to a Github project.  No code yet, just because I don't have any yet,
but we can use the issues/PRs/wiki features to collaborate publicly in
a more structured, less transient, way:

https://github.com/Abscissa/DPak

Awesome, let's see if we can actually get this thing off the ground.

H. S. Teoh: I forget your Github handle. Let me know and I'll add you
as a collaborator.

quickfur.

T

--
Famous last words: I *think* this will work...

Apr 17 2019
Atila Neves <atila.neves gmail.com> writes:
On Monday, 15 April 2019 at 23:17:46 UTC, H. S. Teoh wrote:
On Mon, Apr 15, 2019 at 06:56:24PM -0400, Nick Sabalausky
(Abscissa) via Digitalmars-d wrote:
[...]
[...]

Y'know, the idea just occurred to me: why even bind the package
manager to the build system at all?

They really shouldn't be. If anything, dub is a good example of
why not.

Why not make the build system one of the dependencies of the
project?

That could work, as long as there's a sensible default one.

For backward compatibility, 'dub-build' would be assumed if no
build system is specified.

+1

Apr 16 2019
Dragos Carp <dragoscarp gmail.com> writes:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
Hi all,
...

I have a different proposal.

One of the goals of D is the interoperability with C/C++ and
interoperability in general. Because of that I think it really
make sens to not reinvent the wheel, but to have good support for
an already existing tool.
Since a couple of months I'm using Bazel, and I like it. Not
necessary the implementation (is big and Java), but the concepts
behind it. So I'm already working right now on improving the D
support [1]: adding more tests, supporting more compiler
versions, etc.

So my plan is:
- add tests: supporting d libraries, exectuables, tests and
C++/D hybrid applications
- add compiler version selection
- ldc and gcc support
- dub .json, .sdl -> bazel BUILD file converter (not foolproof).
If necessary, manual intervention is acceptable.
- add windows support
- write a D starlark [2] implementation
- rewrite bazel in D. Probably this will never happen, but maybe
the subset sufficent to cover the dub functionality will be
doable.

If somebody else is interested to join to this effort, please
give me a sign.

Dragos

[1] https://github.com/bazelbuild/rules_d
[2] https://docs.bazel.build/versions/master/skylark/language.html

Apr 16 2019
aliak <something something.com> writes:
On Tuesday, 16 April 2019 at 17:08:22 UTC, Dragos Carp wrote:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
[...]

I have a different proposal.

One of the goals of D is the interoperability with C/C++ and
interoperability in general. Because of that I think it really
make sens to not reinvent the wheel, but to have good support
for an already existing tool.
Since a couple of months I'm using Bazel, and I like it. Not
necessary the implementation (is big and Java), but the
concepts behind it. So I'm already working right now on
improving the D support [1]: adding more tests, supporting more
compiler versions, etc.

[...]

Super nice! I was just going to ask if anyone's been using bazel
and what they think of it. And here you are already hacking away
d rules :D

Apr 16 2019
9il <ilyayaroshenko gmail.com> writes:
On Tuesday, 16 April 2019 at 17:08:22 UTC, Dragos Carp wrote:
On Sunday, 14 April 2019 at 10:53:17 UTC, Seb wrote:
Hi all,
...

So my plan is:
- add tests: supporting d libraries, exectuables, tests and
C++/D hybrid applications
- add compiler version selection
- ldc and gcc support
- dub .json, .sdl -> bazel BUILD file converter (not
foolproof). If necessary, manual intervention is acceptable.
- add windows support
- write a D starlark [2] implementation
- rewrite bazel in D. Probably this will never happen, but
maybe the subset sufficent to cover the dub functionality will
be doable.

Very nice, I  can add support for mir packages. Do you have an
opensource working D+Bazel project and Travis config for it to
use it as an example? --Ilya

Apr 16 2019
Dragos Carp <dragoscarp gmail.com> writes:
On Wednesday, 17 April 2019 at 02:57:44 UTC, 9il wrote:
Very nice, I  can add support for mir packages. Do you have an
opensource working D+Bazel project and Travis config for it to
use it as an example? --Ilya

Not yet. I'll get back to you when this is set. Travis config is
part of first item on the plan. BTW the repo is
https://github.com/dcarp/rules_d.

Apr 16 2019
H. S. Teoh <hsteoh quickfur.ath.cx> writes:
Recently, Nick & some others (including myself) created a Github
project to discuss the possibility of a dub replacement that has
a better architecture. In the course of analysing our current
situation and where we'd like to be, Nick pointed out something
that immediately strikes me as low-hanging fruit that could
potentially improve dub's performance significantly.

According to Nick, every dub package carries its own description
file (sdl/json) that encodes, among other things, its version and
dependencies.  Apparently, this information resides per-package
on code.dlang.org, and so every time dub has to resolve
dependencies for a particular package, it has to download the
description files of *every version* of the package??  Can
somebody confirm whether or not this is really the case?

Because if this is true, that means dub is making N network
roundtrips *per dependency* just to obtain dependency information
from each package.  This is obviously extremely inefficient, and
I'll bet that it's a significant source of dub's perceived
slowness.

But this also immediately suggests a simple solution: upon
uploading a package to code.dlang.org, the server should collect
the description files of all versions of that package, and make
that available somewhere in a per-package global description
file.  It doesn't need to contain everything in dub.{sdl/json};
all it needs is version information and per-version dependency
lists.  This way, whenever dub needs to resolve dependencies on
that package, it can obtain all of the information it needs on
that package with a single HTTP roundtrip.  Of course, it will
still need to make more roundtrips in order to obtain information
for sub-dependencies and other dependencies, but if what Nick
says is true, this should significantly cut down on the total
number of roundtrips required (O(n) as opposed to O(n*v), where n
= #packages, v = #versions-per-package).

--T

Apr 26 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Apr 26, 2019 at 08:28:37PM +0000, H. S. Teoh via Digitalmars-d wrote:
[...]
According to Nick, every dub package carries its own description file
(sdl/json) that encodes, among other things, its version and
dependencies.  Apparently, this information resides per-package on
code.dlang.org, and so every time dub has to resolve dependencies for
a particular package, it has to download the description files of
*every version* of the package??  Can somebody confirm whether or not
this is really the case?

[...]

Actually, nevermind that.  Dub already combines the descriptions into a
single file.  So the bottleneck(s) must lie elsewhere...

T

--
2+2=4. 2*2=4. 2^2=4. Therefore, +, *, and ^ are the same operation.

Apr 26 2019
Andre Pany <andre s-e-a-p.de> writes:
On Friday, 26 April 2019 at 22:33:50 UTC, H. S. Teoh wrote:
On Fri, Apr 26, 2019 at 08:28:37PM +0000, H. S. Teoh via
Digitalmars-d wrote: [...]
According to Nick, every dub package carries its own
description file (sdl/json) that encodes, among other things,
its version and dependencies.  Apparently, this information
resides per-package on code.dlang.org, and so every time dub
has to resolve dependencies for a particular package, it has
to download the description files of *every version* of the
package??  Can somebody confirm whether or not this is really
the case?

[...]

Actually, nevermind that.  Dub already combines the
descriptions into a single file.  So the bottleneck(s) must lie
elsewhere...

T

Also with recent Dub version a major performance optimization was
done
https://dlang.org/changelog/2.086.0.html#single-api-requests

In addition Sebastian is also working on parallel dependencies
download.

Kind regards
Andre

Apr 26 2019
Seb <seb wilzba.ch> writes:
On Friday, 26 April 2019 at 22:56:53 UTC, Andre Pany wrote:
On Friday, 26 April 2019 at 22:33:50 UTC, H. S. Teoh wrote:
On Fri, Apr 26, 2019 at 08:28:37PM +0000, H. S. Teoh via
Digitalmars-d wrote: [...]
According to Nick, every dub package carries its own
description file (sdl/json) that encodes, among other things,
its version and dependencies.  Apparently, this information
resides per-package on code.dlang.org, and so every time dub
has to resolve dependencies for a particular package, it has
to download the description files of *every version* of the
package??  Can somebody confirm whether or not this is really
the case?

[...]

Actually, nevermind that.  Dub already combines the
descriptions into a single file.  So the bottleneck(s) must
lie elsewhere...

T

Also with recent Dub version a major performance optimization
was done
https://dlang.org/changelog/2.086.0.html#single-api-requests

In addition Sebastian is also working on parallel dependencies
download.

Kind regards
Andre

Unfortunately I'm not actively working on this. I have too many
other things on my plate. I just did a quick experiment and it
more than halfed the fresh build time for the tested projects
(the unzipping happens in parallel too). However, the quick
solution doesn't seem to work with older D versions.
In case anyone is interested:

https://github.com/dlang/dub/pull/1677

Apr 26 2019
Basile B. <b2.temp gmx.com> writes:
On Friday, 26 April 2019 at 22:33:50 UTC, H. S. Teoh wrote:
On Fri, Apr 26, 2019 at 08:28:37PM +0000, H. S. Teoh via
Digitalmars-d wrote: [...]
According to Nick, every dub package carries its own
description file (sdl/json) that encodes, among other things,
its version and dependencies.  Apparently, this information
resides per-package on code.dlang.org, and so every time dub
has to resolve dependencies for a particular package, it has
to download the description files of *every version* of the
package??  Can somebody confirm whether or not this is really
the case?

[...]

Actually, nevermind that.  Dub already combines the
descriptions into a single file.  So the bottleneck(s) must lie
elsewhere...

T

Speaking of bottlenecks... (and probably unrelated):

For dexed I have a project group that contains the 3 sdl for
dmd/druntime/phobos. It's very slow to load... the IDE launches
dub 3 times to convert the projects to JSON and it takes 8 to 10
seconds. But none of the sdl contains dependencies, nor configs,
build types or whatever...

I have the feeling that dub does something stupid with the files
since these 3 repos contains many.

Apr 26 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Apr 27, 2019 at 12:02:58AM +0000, Basile B. via Digitalmars-d wrote:
[...]
For dexed I have a project group that contains the 3 sdl for
dmd/druntime/phobos. It's very slow to load... the IDE launches dub 3
times to convert the projects to JSON and it takes 8 to 10 seconds.
But none of the sdl contains dependencies, nor configs, build types or
whatever...

I have the feeling that dub does something stupid with the files since
these 3 repos contains many.

Shouldn't we be able to just run a profiler on dub to figure out where
it's spending those 10 seconds? (or about 3 seconds per run.)  Seems
like something easy enough to identify, even if the problem itself may
not be so simple to fix.  We should find out whether it's CPU-bound,
I/O-bound, or network-bound, at the very least.

T

--
MS Windows: 64-bit rehash of 32-bit extensions and a graphical shell for a
16-bit patch to an 8-bit operating system originally coded for a 4-bit
microprocessor, written by a 2-bit company that can't stand 1-bit of
competition.

Apr 26 2019
Basile B. <b2.temp gmx.com> writes:
On Saturday, 27 April 2019 at 00:37:35 UTC, H. S. Teoh wrote:
On Sat, Apr 27, 2019 at 12:02:58AM +0000, Basile B. via
Digitalmars-d wrote: [...]
For dexed I have a project group that contains the 3 sdl for
dmd/druntime/phobos. It's very slow to load... the IDE
launches dub 3 times to convert the projects to JSON and it
takes 8 to 10 seconds. But none of the sdl contains
dependencies, nor configs, build types or whatever...

I have the feeling that dub does something stupid with the
files since these 3 repos contains many.

Shouldn't we be able to just run a profiler on dub to figure
out where it's spending those 10 seconds? (or about 3 seconds
per run.)  Seems like something easy enough to identify, even
if the problem itself may not be so simple to fix.  We should
find out whether it's CPU-bound, I/O-bound, or network-bound,
at the very least.

T

I've determined that the slowdown is caused by the auto-fetch
option of the IDE. When disabled the group loads instantly.
Actually there are dependencies in the sdl for DMD: dmd:lexer,
dmd:parser, dmd:frontend. Dexed fails to determine that they are
described directly in the description.

May 01 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Apr 26, 2019 at 05:37:35PM -0700, H. S. Teoh via Digitalmars-d wrote:
On Sat, Apr 27, 2019 at 12:02:58AM +0000, Basile B. via Digitalmars-d wrote:
[...]
For dexed I have a project group that contains the 3 sdl for
dmd/druntime/phobos. It's very slow to load... the IDE launches dub
3 times to convert the projects to JSON and it takes 8 to 10
seconds.  But none of the sdl contains dependencies, nor configs,
build types or whatever...

I have the feeling that dub does something stupid with the files
since these 3 repos contains many.

Shouldn't we be able to just run a profiler on dub to figure out where
it's spending those 10 seconds? (or about 3 seconds per run.)  Seems
like something easy enough to identify, even if the problem itself may
not be so simple to fix.  We should find out whether it's CPU-bound,
I/O-bound, or network-bound, at the very least.

[...]

OK, so I upgraded my dub to the latest git master, and did a quick and
dirty test.  Init a fresh new project with -t vibe.d, accept all
default values (name, license, etc.), then run time dub -v to make the
first build.

It took 2 minutes 45 seconds (!).  From a crude visual inspection, most
of the time was spent downloading and compiling the dependencies, so
that seems reasonable, even if unfortunate.

Subsequent runs (remove dub.selections, remove executable, rerun dub -v)
took about 11 seconds or so.  Still rather long, but it's a LOT faster
than the previous version of dub that I tested prior to upgrading to git
master.

The previous version of dub took about 2 minutes 15 seconds for the
first build of a fresh empty vibe.d project, but most of the time
appeared to be spent on the "searching for versions of xyz (1
suppliers)" phase, which I presume was the per-version downloading of
package descriptions.  (The fact that it took 30 seconds less than the
new dub is likely caused by some cached objects from previous vibe.d
builds, so that measurement shouldn't be trusted. I deleted the entire
dub cache before running the test on the new version of dub.)
Subsequent runs (delete dub.selections, rerun dub -v) took about 40
seconds, most of which was spent on the "searching for versions" phase.

So the new dub appeared to have eliminated a good chunk of network
turnaround time, which is good news.

10 seconds is still rather long for an empty project, but from casual
visual inspection of -v output, it appears to still be stuck on the
"searching for versions of vibe.d" phase, which took several solid
seconds.  I'm presuming this is where it's waiting on the network.
Re-running dub without deleting dub.selections eliminates this step, and
turnaround time drops to about 4-5 seconds.  Finally!! It's getting a
lot closer to a serviceable edit-compile-test turnaround time!

So the latest version of dub is looking a lot better than before. Now,
if we could lift some of its functional limitations, we might finally
have an acceptably performant and functional package/build tool.

In the meantime, it would seem that we need to look into why the
"searching for versions" phase takes so long.  Is it just a
network-dependent thing (my network has bogonously slow DNS resolution,
no thanks to my ISP), or is it something that can be fixed in dub
itself?

T

--
VI = Visual Irritation

Apr 26 2019
Seb <seb wilzba.ch> writes:
On Saturday, 27 April 2019 at 01:13:33 UTC, H. S. Teoh wrote:
On Fri, Apr 26, 2019 at 05:37:35PM -0700, H. S. Teoh via
Digitalmars-d wrote:
[...]

[...]

OK, so I upgraded my dub to the latest git master, and did a
quick and dirty test.  Init a fresh new project with -t
vibe.d, accept all default values (name, license, etc.), then
run time dub -v to make the first build.

[...]

10 seconds for a rebuild is still too much.

One quick solution is to upgrade to ld.gold - it will half your
build time.

Apr 26 2019
Seb <seb wilzba.ch> writes:
On Saturday, 27 April 2019 at 01:13:33 UTC, H. S. Teoh wrote:
In the meantime, it would seem that we need to look into why
the "searching for versions" phase takes so long.  Is it just a
network-dependent thing (my network has bogonously slow DNS
resolution, no thanks to my ISP), or is it something that can
be fixed in dub itself?

Thanks a lot for your interest in dub and investigating this!

First off you should never delete your dub.selections.json as it
locks your project dependencies (it won't be used if your project
is used as a library).

Anyhow, that being said there are still a ton of things that can
be done:

- The new single API request feature doesn't work for all 100%
with optional dependencies (see the respective GitHub PR that
introduced it for details)
- Dependencies could be checked in parallel
- The registry itself could be optimized more for caching (maybe
even with a CDN proxy)
...

There are more experimental things we could try like e.g. a fully
local JSON index that is only updated when needed and supports
partial updates (think apt), but I believe the bigger gains in
user-experience will be:

- initial fetch (important for fast CI turnaround times. First
point of attack: parallelizing the fetching process)
- build times with existing dependencies (important as the
default case. First points of attack: build independent
dependencies in parallel, warn if ld.gold isn't the default on
Linux, ...)

Apr 26 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/27/19 1:45 AM, Seb wrote:

- The new single API request feature doesn't work for all 100% with
optional dependencies (see the respective GitHub PR that introduced it
for details)

The API you're referring to, is it this one?

https://code.dlang.org/api/packages/[PACKAGE_NAME]/info

or something else?

Apr 27 2019
Seb <seb wilzba.ch> writes:
On Friday, 26 April 2019 at 20:28:37 UTC, H. S. Teoh wrote:
Recently, Nick & some others (including myself) created a
Github project to discuss the possibility of a dub replacement
that has a better architecture. In the course of analysing our
current situation and where we'd like to be, Nick pointed out
something that immediately strikes me as low-hanging fruit that
could potentially improve dub's performance significantly.

[...]

https://dlang.org/changelog/2.086.0.html#single-api-requests

Though tere are still a ton of other easy low-hanging fruits
which is why I started this thread.

Apr 26 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/26/19 4:28 PM, H. S. Teoh wrote:
Recently, Nick & some others (including myself) created a Github project
to discuss the possibility of a dub replacement that has a better
architecture.

To clarify, this project is only intended to replace the package
management side of dub.

It deliberately *doesn't* replace the buildsystem side of dub. It's
strictly buildsystem-agnostic (and even programming language agnostic),
so you can bring-your-own-buildsystem whether it be ninja, SCons, make,
manual buildscripts, or indeed, even dub itself.

Also, to assuage any fears, one of the key goals is to support the
existing DUB packages on code.dlang.org, with no work necessary on the
part of DUB package authors. (Naturally, such packages will still be
using dub as their buildsystem...or perhaps Atila's 'bud' once that's up
and running.)

For anyone interested, here's the project on Github:
https://github.com/Abscissa/DPak

Apr 27 2019
"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Apr 27, 2019 at 05:22:40AM +0000, Seb via Digitalmars-d wrote:
On Saturday, 27 April 2019 at 01:13:33 UTC, H. S. Teoh wrote:

[...]
OK, so I upgraded my dub to the latest git master, and did a quick
and dirty test.  Init a fresh new project with -t vibe.d, accept
all default values (name, license, etc.), then run time dub -v to
make the first build.
[...]

10 seconds for a rebuild is still too much.

One quick solution is to upgrade to ld.gold - it will half your build
time.

Just to be clear: the 10 seconds rebuild time is only if dub.selections
is deleted before the rebuild.  I only did that as a way of getting a
rough measure of how fast/slow the dependency resolution algorithm is,
when dependencies have already been downloaded into the dub cache.  It's
not something I do habitually, or recommend. :-D

Without deleting dub.selections, the turnaround time is about 4-5
seconds.  Which is not fast, but at least marginally acceptable.
Certainly, 10 seconds turnaround would be unacceptable edit-compile-test
turnaround time on an empty project.

On Sat, Apr 27, 2019 at 05:45:50AM +0000, Seb via Digitalmars-d wrote:
On Saturday, 27 April 2019 at 01:13:33 UTC, H. S. Teoh wrote:
In the meantime, it would seem that we need to look into why the
"searching for versions" phase takes so long.  Is it just a
network-dependent thing (my network has bogonously slow DNS
resolution, no thanks to my ISP), or is it something that can be
fixed in dub itself?

Thanks a lot for your interest in dub and investigating this!

First off you should never delete your dub.selections.json as it locks
your project dependencies (it won't be used if your project is used as
a library).

Yes, I only did that to get some idea of how efficient the dependency
resolution is.

Anyhow, that being said there are still a ton of things that can be
done:

- The new single API request feature doesn't work for all 100% with
optional dependencies (see the respective GitHub PR that introduced
it for details)
- Dependencies could be checked in parallel

Parallel checking is a must, IMO, since it doesn't make sense to
bottleneck on individual network requests when there's plenty of
bandwidth to run multiple queries in parallel.

- The registry itself could be optimized more for caching (maybe even
with a CDN proxy)

I'm not sure we need to use a CDN yet, unless code.dlang.org is really
getting that much traffic.

But what might help is if the registry allows more complex queries, like
"fetch me all candidate packages satisfying contraints P, Q, R... ."
Single network roundtrip for the entire query, rather than separate
network requests, once per package.  Of course, this puts more load on
the server, which may or may not be a good thing, I'm not sure.

[[...]
There are more experimental things we could try like e.g. a fully
local JSON index that is only updated when needed and supports partial
updates (think apt), but I believe the bigger gains in user-experience
will be:

- initial fetch (important for fast CI turnaround times. First point
of attack: parallelizing the fetching process)
- build times with existing dependencies (important as the default
case.  First points of attack: build independent dependencies in
parallel, warn if ld.gold isn't the default on Linux, ...)

Doesn't --parallel already do this??  If not, that certainly needs to be
fixed.

Sigh... what I wouldn't give for a generic topological walk framework
that allows maximal parallelization...

T

--
Customer support: the art of getting your clients to pay for your own
incompetence.

Apr 29 2019
"Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 4/29/19 3:11 PM, H. S. Teoh wrote:
But what might help is if the registry allows more complex queries, like
"fetch me all candidate packages satisfying contraints P, Q, R... ."
Single network roundtrip for the entire query, rather than separate
network requests, once per package.  Of course, this puts more load on
the server, which may or may not be a good thing, I'm not sure.

GraphQL: https://graphql.org/  (Forget where I came across it, but it
was just recently.)

And yes, you're pretty much right about the tradeoff between REST vs
GraphQL. Choosing between them is about balancing "minimizing server
trips" (GraphQL) vs "maximizing cacheability" (REST).

May 02 2019
John Colvin <john.loughran.colvin gmail.com> writes:
On Thursday, 2 May 2019 at 18:35:10 UTC, Nick Sabalausky
(Abscissa) wrote:
On 4/29/19 3:11 PM, H. S. Teoh wrote:
But what might help is if the registry allows more complex
queries, like
"fetch me all candidate packages satisfying contraints P, Q,
R... ."
Single network roundtrip for the entire query, rather than
separate
network requests, once per package.  Of course, this puts more
load on
the server, which may or may not be a good thing, I'm not sure.

GraphQL: https://graphql.org/  (Forget where I came across it,
but it was just recently.)

And yes, you're pretty much right about the tradeoff between
REST vs GraphQL. Choosing between them is about balancing
"minimizing server trips" (GraphQL) vs "maximizing
cacheability" (REST).

http://code.dlang.org/packages/graphqld

May 02 2019