www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - dud: A dub replacement

reply Robert Schadek <rschadek symmetryinvestments.com> writes:
So dub has some problems, and personally I find its code base 
very hard to get
into.
At Symmetry we are a very heavy user of dub, resulting in many 
wasted hours.

So I started to write dud [1]. I kept some boring/nice parts from 
dub, but most
code so far is a complete rewrite.

The goal of dud is mostly do what dub does, but more 
understandable.
dud will/does aim for a tasteful subset of dub's features.
Meaning, most dub files should be good with dud.
If they are not, you will/should get an error message telling you 
whats wrong.
The bigger goal, at least personally, is to have a code base of 
pure functions
that is "trivial" to understand and debug.
The rules of thumb is: "When your line gets longer than 80 
characters,
restructure your code!", "Branch statements are code smells."

So what can dud do right now.

$ dud convert

works.
IMO that is an important step, as this involves ingesting 
dub.(json|sdl) into
a common representation.

I use dubproxy [2] to get all ~1.6k packages from code.dlang.org 
and create a
git working tree for all the versions of all the packages.
Currently, that results in ~60k (package|dub).(json|sdl) files.
Then, I run dud on those and see what breaks.
Only a few percent are not ingestable by dud, and those are in 
IHMO not valid
anyway (to be fair, there is some strange 💩 out there).

Now that dud can parse dub files, the next step will be a 
semantic phase,
checking the input for errors.
After that, path expansion and dependency resolution.

PR's are always welcome.

Destroy!

[1] https://github.com/symmetryinvestments/dud
[2] https://github.com/symmetryinvestments/dubproxy
Nov 11 2019
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-11-11 14:44, Robert Schadek wrote:

 So dub has some problems, and personally I find its code base very hard 
 to get
 into.
 At Symmetry we are a very heavy user of dub, resulting in many wasted 
 hours.
 
 So I started to write dud [1]. I kept some boring/nice parts from dub, 
 but most
 code so far is a complete rewrite.


Hmm, that's unfortunate. It happens way too often in the D community, 
that a project is rewritten from scratch instead of improving the 
existing one. Especially since the D community is not that big.

Although I can understand the choice. Even if you were to contribute 
there would be no one to merge the pull requests :(.

-- 
/Jacob Carlborg
Nov 11 2019
parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 19:17:54 UTC, Jacob Carlborg wrote:

 [snip]

 Hmm, that's unfortunate. It happens way too often in the D 
 community, that a project is rewritten from scratch instead of 
 improving the existing one. Especially since the D community is 
 not that big.

 Although I can understand the choice. Even if you were to 
 contribute there would be no one to merge the pull requests :(.


Yeah...it was wondering if there was a way to have like a 
dub-core that both dub and dud can depend on.
Nov 11 2019
prev sibling next sibling parent mipri <mipri minimaltype.com> writes:
On Monday, 11 November 2019 at 13:44:28 UTC, Robert Schadek wrote:

 So I started to write dud [1]. I kept some boring/nice parts 
 from dub, but most
 code so far is a complete rewrite.


Someone's eventually going to say this, and they probably really
won't want to, so I'll spare them: this is a horrible name. Is it
even deliberate? Is it only accidentally a meaningful English
word?

https://en.wiktionary.org/wiki/dud#Noun

(informal) A device or machine that is useless because it does 
not work properly or has failed to work, such as a bomb, or 
explosive projectile.
(informal) A failure of any kind.
(informal) A loser; an unlucky person.
A lottery ticket that does not give a payout.

Maybe this is in the tradition of tool names that have bad
connotations but are actually intended to be meaningless, like
Fedora's dnf (Did Not Finish). Or even horrible, disgusting
connotations, with the hope that this will serve as an
attention-getter followed by the reader being unwilling to concede
that emotions can interfere in technical decisions.
( 
https://duckduckgo.com/the%20open-source,%20cloud%20native%20SQL%20database )
Nov 11 2019
prev sibling next sibling parent GoaLitiuM <goalitium dforums.mail.kapsi.fi> writes:
On Monday, 11 November 2019 at 13:44:28 UTC, Robert Schadek wrote:

 So dub has some problems, and personally I find its code base 
 very hard to get into.
 At Symmetry we are a very heavy user of dub, resulting in many 
 wasted hours.


I'm interested in which areas does you and/or Symmetry find dub 
not delivering expected results? Is it package management in 
general? Build process? What's the actual need to rewrite dub 
again?

I started working on a build tool to replace dub for my personal 
projects some time ago because I grew tired of long build times 
when only small portion of the project was touched. This last 
month I decided to rewrite it and work on it more actively, and 
I've already managed to get some huge libraries like vibe-d and 
serve-d building. I have no plans to rewrite the dub as a whole, 
but only focus on the build command, leaving the package 
management to dub which I have personally no problems with.
Nov 11 2019
prev sibling next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 11/11/19 8:44 AM, Robert Schadek wrote:

 
 The goal of dud is mostly do what dub does, but more understandable.
 dud will/does aim for a tasteful subset of dub's features.
 Meaning, most dub files should be good with dud.
 If they are not, you will/should get an error message telling you whats 
 wrong.
 The bigger goal, at least personally, is to have a code base of pure 
 functions
 that is "trivial" to understand and debug.
 The rules of thumb is: "When your line gets longer than 80 characters,
 restructure your code!", "Branch statements are code smells."
 


Sounds cool!


 Only a few percent are not ingestable by dud, and those are in IHMO not 
 valid
 anyway (to be fair, there is some strange 💩 out there).


 From what I've seen, dub tends to take the approach of "silently ignore 
anything in dub.sdl I don't recognize as valid". I'll admit that may be 
an important approach for a tool that has no concept of its own changing 
versions (which is somewhat strange for dub, considering half of dub is 
a package manager.) But the flipside is that this fosters and breeds 
errors and invalid files in the wild (see: 90's era web browsers).

I'm curious, are there any particular anti-patterns you've noticed as 
being common in the uningestible repos?
Nov 14 2019
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Thursday, 14 November 2019 at 23:33:06 UTC, Nick Sabalausky 
(Abscissa) wrote:


 From what I've seen, dub tends to take the approach of 
 "silently ignore anything in dub.sdl I don't recognize as 
 valid". I'll admit that may be an important approach for a tool 
 that has no concept of its own changing versions (which is 
 somewhat strange for dub, considering half of dub is a package 
 manager.) But the flipside is that this fosters and breeds 
 errors and invalid files in the wild (see: 90's era web 
 browsers).


Also, this, apparently, should lead to the fact that dud will 
have their own package description format. Almost inevitable. 
(This has already been discussed, perhaps.)
Nov 17 2019
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 17 November 2019 at 16:26:45 UTC, Denis Feklushkin 
wrote:

 On Thursday, 14 November 2019 at 23:33:06 UTC, Nick Sabalausky 
 (Abscissa) wrote:
 Also, this, apparently, should lead to the fact that dud will 
 have their own package description format. Almost inevitable. 
 (This has already been discussed, perhaps.)


SDL all the way please.
Nov 17 2019
next sibling parent Soulsbane <paul acheronsoft.com> writes:
On Sunday, 17 November 2019 at 19:10:05 UTC, Sebastiaan Koppe 
wrote:

 On Sunday, 17 November 2019 at 16:26:45 UTC, Denis Feklushkin 
 wrote:

 On Thursday, 14 November 2019 at 23:33:06 UTC, Nick Sabalausky 
 (Abscissa) wrote:
 Also, this, apparently, should lead to the fact that dud will 
 have their own package description format. Almost inevitable. 
 (This has already been discussed, perhaps.)


 SDL all the way please.


Yes please
Nov 17 2019
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Sunday, 17 November 2019 at 19:10:05 UTC, Sebastiaan Koppe 
wrote:


 SDL all the way please.


Perhaps this ship has already sail. But YAML would have been a 
better choice. It's a superset of JSON. All the existing JSON 
description files would have worked as is.

--
/Jacob Carlborg
Nov 18 2019
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 11/18/19 3:54 AM, Jacob Carlborg wrote:

 
 Perhaps this ship has already sail. But YAML would have been a better 
 choice. It's a superset of JSON. All the existing JSON description files 
 would have worked as is.


YAML's grammar is a convoluted read-only mess. Every time I have to 
write it I feel like I'm just guessing and hoping I'm magically writing 
it correctly. I've even had silent problems on travis-ci that turned out 
to be .travis.yml not being parsed the way I thought. That sort of thing 
shouldn't even be *possible* on a config file format.

Especially irritating to me is even just the basic nesting - the whole 
"sometimes you HAVE to use a hyphen and sometimes you have to OMIT the 
hyphen (or it'll silently be interpreted wrong)". And yea, yea, "Object 
vs array", whatever, but when you're using YAML for hand-written config 
files, the whole distinction between what should be objects vs arrays is 
just a big pile of arbitrary pointlessness (a flaw it inherits from JSON).

And the grammar being a superset of JSON doesn't really help much in 
practice, because real idiomatic YAML in the wild (including config file 
examples) doesn't use it.
Nov 18 2019
prev sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Sun, 2019-11-17 at 19:10 +0000, Sebastiaan Koppe via Digitalmars-d-annou=
nce=20
wrote:

 On Sunday, 17 November 2019 at 16:26:45 UTC, Denis Feklushkin=20
 wrote:

 On Thursday, 14 November 2019 at 23:33:06 UTC, Nick Sabalausky=20
 (Abscissa) wrote:
 Also, this, apparently, should lead to the fact that dud will=20
 have their own package description format. Almost inevitable.=20
 (This has already been discussed, perhaps.)


=20
 SDL all the way please.


Is SDL the right format? Cargo uses TOML to great effect.

And TOML has, I suspect greater traction more widely than SDL.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 18 2019
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 18 November 2019 at 08:57:58 UTC, Russel Winder wrote:

 Is SDL the right format? Cargo uses TOML to great effect.

 And TOML has, I suspect greater traction more widely than SDL.


I personally prefer SDL when it comes to nested data, but yeah, 
that would work as well.

The point I was making is to just have 1 format. With dud that 
should be possible.
Nov 18 2019
parent reply Russel Winder <russel winder.org.uk> writes:
On Mon, 2019-11-18 at 09:31 +0000, Sebastiaan Koppe via Digitalmars-d-annou=
nce=20
wrote:

 On Monday, 18 November 2019 at 08:57:58 UTC, Russel Winder wrote:

 Is SDL the right format? Cargo uses TOML to great effect.
=20
 And TOML has, I suspect greater traction more widely than SDL.


=20
 I personally prefer SDL when it comes to nested data, but yeah,=20
 that would work as well.
=20
 The point I was making is to just have 1 format. With dud that=20
 should be possible.


For me the argument is that SDL and TOML are intended for human's to write
configuration scripts, whilst XML and JSON are intended for computers to se=
nd
data to other computers.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 18 2019
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 18 November 2019 at 09:42:16 UTC, Russel Winder wrote:

 On Mon, 2019-11-18 at 09:31 +0000, Sebastiaan Koppe via 
 Digitalmars-d-announce wrote:

 On Monday, 18 November 2019 at 08:57:58 UTC, Russel Winder 
 wrote:

 Is SDL the right format? Cargo uses TOML to great effect.
 
 And TOML has, I suspect greater traction more widely than 
 SDL.


 
 I personally prefer SDL when it comes to nested data, but 
 yeah, that would work as well.
 
 The point I was making is to just have 1 format. With dud that 
 should be possible.


 For me the argument is that SDL and TOML are intended for 
 human's to write configuration scripts, whilst XML and JSON are 
 intended for computers to send data to other computers.


A win-win move would be to have dud emit the other formats 
automatically as part of the compilation procedure, so to have 
always all of them present and synced on the content.

It shouldn't be too much difficult, and maybe it's a cleaver way 
to move on from discussions about different formats.
Nov 18 2019
next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 18 November 2019 at 09:53:56 UTC, Paolo Invernizzi 
wrote:

 A win-win move would be to have dud emit the other formats 
 automatically as part of the compilation procedure, so to have 
 always all of them present and synced on the content.


I already regret starting about this. Instead of rooting for the 
SDL format, I should have just recommended to deprecate the json 
one.
Nov 18 2019
parent reply Russel Winder <russel winder.org.uk> writes:
On Mon, 2019-11-18 at 10:26 +0000, Sebastiaan Koppe via Digitalmars-d-annou=
nce=20
wrote:

 On Monday, 18 November 2019 at 09:53:56 UTC, Paolo Invernizzi=20
 wrote:

 A win-win move would be to have dud emit the other formats=20
 automatically as part of the compilation procedure, so to have=20
 always all of them present and synced on the content.


=20
 I already regret starting about this. Instead of rooting for the=20
 SDL format, I should have just recommended to deprecate the json=20
 one.


I do not think you should regret starting a discussion on this.

Personally I only ever use SDL with Dub. Even contemplating using JSON for
human written configuration files is, for me, totally the wrong thing to do=
.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 18 2019
parent reply JN <666total wp.pl> writes:
On Monday, 18 November 2019 at 10:40:28 UTC, Russel Winder wrote:

 On Mon, 2019-11-18 at 10:26 +0000, Sebastiaan Koppe via 
 Digitalmars-d-announce wrote:

 On Monday, 18 November 2019 at 09:53:56 UTC, Paolo Invernizzi 
 wrote:

 A win-win move would be to have dud emit the other formats 
 automatically as part of the compilation procedure, so to 
 have always all of them present and synced on the content.


 
 I already regret starting about this. Instead of rooting for 
 the SDL format, I should have just recommended to deprecate 
 the json one.


 I do not think you should regret starting a discussion on this.

 Personally I only ever use SDL with Dub. Even contemplating 
 using JSON for human written configuration files is, for me, 
 totally the wrong thing to do.


I only use the JSON format. JSON is widespread together with XML. 
SDL I heard first time of in the context of Dub and never seen it 
used elsewhere. TOML also I know only from Cargo. YAML at least I 
know from several different projects.

I guess the tool must be working very well though if the main 
argument is what kind of data format to use :)
Nov 18 2019
next sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Mon, 2019-11-18 at 16:54 +0000, JN via Digitalmars-d-announce wrote:

 [=E2=80=A6]
=20
 I only use the JSON format. JSON is widespread together with XML.=20
 SDL I heard first time of in the context of Dub and never seen it=20
 used elsewhere. TOML also I know only from Cargo. YAML at least I=20
 know from several different projects.


It is so easy to make spelling errors in keys using hand-written JSON.

SDL I have only ever used with Dub, but it works better for me than JSON.

TOML is really an INI evolution so known to many.

YAML is used by many but doesn't really allow for the ease of specifying li=
sts
as TOML and SDL have.


 I guess the tool must be working very well though if the main=20
 argument is what kind of data format to use :)


Probably yes. Though Cargo has taken many different decisions to Dub and
mostly I think Cargo took better decisions.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 18 2019
next sibling parent reply Tobias Pankrath <tobias pankrath.net> writes:
On Monday, 18 November 2019 at 19:54:38 UTC, Russel Winder wrote:


 Probably yes. Though Cargo has taken many different decisions 
 to Dub and mostly I think Cargo took better decisions.


Could you elaborate a bit, please? I am not familiar with Cargo 
though.
Nov 18 2019
next sibling parent Russel Winder <russel winder.org.uk> writes:
On Mon, 2019-11-18 at 20:19 +0000, Tobias Pankrath via Digitalmars-d-announ=
ce
wrote:

 On Monday, 18 November 2019 at 19:54:38 UTC, Russel Winder wrote:
=20

 Probably yes. Though Cargo has taken many different decisions=20
 to Dub and mostly I think Cargo took better decisions.


=20
 Could you elaborate a bit, please? I am not familiar with Cargo=20
 though.


The single biggest difference for me just now is management of compilation
products.

In an attempt to share compiled products between projects, Dub keeps all
downloaded package source and compilations products of that source in a
central location. This ends up with a very complicated, and more or less
unusable except by Dub, store of compilation products =E2=80=93 especially =
as the easy
to get to compilation product has unknown compilation details. Also it assu=
mes
you have infinite amounts of disc space.

Cargo keeps the downloaded package sources centrally, but leaves all
compilation products in the project build area. This makes it easy to manag=
e
and the compilation details of the compilation products are easy to
understand. OK you still have to manually manage the central repository but=
 as
it is source only, you get a much smaller growth to infinite disc space
requirement.

As for smaller issues:

I prefer TOML over SDL as project specification notation (JSON is not an
option for me).

The convention over configuration rules for projects with multiple compilat=
ion
products is nicer in Cargo compared to Dub.

Cargo is far more accepted in Rust circles than Dub is in D circles. The is=
sue
is not that this is true, but why is it true: is it just that D folk are st=
ill
obsessed with Make, SCons, CMake, Meson, etc. where Rust folk have never ev=
en
considered them.

D is still working with the ancient philosophy of the distributed standard
library provides everything, cf. Python and "batteries included". Python ha=
s
long ago had to give up on this and open its arms to PyPI based package
distribution.  Go and Rust never went with the "batteries included" but put
the infrastructure in place at the outset for supporting libraries from
repositories =E2=80=93 Rust/Cargo handles this far better than Go. The upsh=
ot is that
the D package repository is still not really central to D code development.
And using Git, Breezy, Mercurial, etc. repositories is just not as simple a=
nd
straightforward as it is with Go and Rust/Cargo.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 19 2019
prev sibling parent David Gileadi <gileadisNOSPM gmail.com> writes:
On 11/18/19 1:19 PM, Tobias Pankrath wrote:

 On Monday, 18 November 2019 at 19:54:38 UTC, Russel Winder wrote:
 

 Probably yes. Though Cargo has taken many different decisions to Dub 
 and mostly I think Cargo took better decisions.


 
 Could you elaborate a bit, please? I am not familiar with Cargo though.


See also Russel's previous post on this issue [1]:


 Go and Rust emphasised using Git, Mercurial, and Breezy repositories as
packages from the outset. Go chose not to add a central repository, Rust chose
to add one. Rust's choice was the correct one for supporting developers. In
hindsight, Go has had problems with packages from the outset based on the
initial workspace model. Slowly over the decade Go is finding ways forward.
Rust got it right from the beginning, once they had stripped down the standard
library and emphasised use of the central repository – if only Phobos could
be stripped right back to the absolute necessary and everything else provided
via the central repository. Obviously not all is good in Rust-land, just as
with Python and PyPI, the central repository is not curated, and this leads to
horrible messes. Ceylon got this more right, but it is a language few have
heard of and even fewer use.
 
 Dub does not allow for use of Git, Mercurial, or Breezy repositories only the
uncurated (and therefore potentially problematic) central repository. OK so you
can do Git checkouts as a separate activity and use local filestore references,
but this is not feasible for packages in the central repository.
 
 Dub builds packages to a location outside the project being worked on. Cargo
pulls sources to such a central non-project place and then compiles into a
project specific location. Dub tries to store all compiled version out of
project in the same area as the sources. Does this matter? Isn't Dub making
things easier to share? Sort of, sort of, and no. Dub stores all compilations,
but hides them and presents only the last compilation easily. This makes things
hard to work with for non-Dub tooling. Cargo makes it a lot easier, at the
expense of lack of sharing but always compiling everything into the project
area.
 
 Cargo uses TOML, Dub uses SDL (or if you are masochistic JSON). 'nuff said.
 QED.
 
 Dub seems to have become the de facto, and indeed de jure, standard for D
build, and yet somehow Cargo is just assumed by all Rust programmers whereas
Dub is a source of contention and ill-will in the D community.
 
 Dub's biggest problem is that there are many in the D community who will not
use it – and they are vocal about it. The solution is not censorship, the
solution is for Dub to evolve very rapidly so that these vocal people have the
rug pulled from under them, i.e. their complaints become invalid.


[1]: 
https://forum.dlang.org/post/mailman.543.1571300816.8294.digital
ars-d puremagic.com 
(scroll toward the end of the post)
Nov 19 2019
prev sibling parent reply Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Monday, 18 November 2019 at 19:54:38 UTC, Russel Winder wrote:

 It is so easy to make spelling errors in keys using 
 hand-written JSON.


I don't understand why this would apply to JSON specifically. 
Whatever the language is, the config files will be hand-written; 
spelling errors are pretty universal, and anything we write is 
prone to mistakes to some extent
Nov 18 2019
parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 18 November 2019 at 23:08:13 UTC, Laurent Tréguier 
wrote:


 I don't understand why this would apply to JSON specifically. 
 Whatever the language is, the config files will be 
 hand-written; spelling errors are pretty universal, and 
 anything we write is prone to mistakes to some extent


dud already tells you if you mistyped a key.

Adding new file formats is "trivial" with dud current code base.

This 
https://github.com/symmetryinvestments/dud/blob/master/pkgdescription/source/dud/pkgd
scription/package.d is the data structure used to mirror the file content.

This 
https://github.com/symmetryinvestments/dud/blob/master/pkgdescription/source/dud/p
gdescription/json.d is the json reader.

Currently, I have no plans to add any other file format.
But PR's are always welcome.
The decision on json and sdl has been made a long time ago, for 
better
or for worse.
Please don't turn this thread into bike-shedding.
Additionally, dub/dud is already on the way to add cli functions 
to manipulate
the config file.
A look at adding dependencies with "dub add" already shows that.
Nov 19 2019
next sibling parent Laurent =?UTF-8?B?VHLDqWd1aWVy?= <laurent.treguier.sink gmail.com> writes:
On Tuesday, 19 November 2019 at 08:15:20 UTC, Robert Schadek 
wrote:

 Currently, I have no plans to add any other file format.
 But PR's are always welcome.
 The decision on json and sdl has been made a long time ago, for 
 better
 or for worse.
 Please don't turn this thread into bike-shedding.


That wasn't my intention; sorry if I formulated my question 
badly. The reason I was asking is actually because I really don't 
see any one format having a huge advantage over the others; I'm 
fine with basically every format that has been listed in this 
thread so far.
Nov 19 2019
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/19/19 3:15 AM, Robert Schadek wrote:

 On Monday, 18 November 2019 at 23:08:13 UTC, Laurent Tréguier wrote:
 

 I don't understand why this would apply to JSON specifically. Whatever 
 the language is, the config files will be hand-written; spelling 
 errors are pretty universal, and anything we write is prone to 
 mistakes to some extent


 
 dud already tells you if you mistyped a key.
 
 Adding new file formats is "trivial" with dud current code base.
 
 This 
 https://github.com/symmetryinvestments/dud/blob/master/pkgdescription/source/dud/pkgd
scription/package.d 
 is the data structure used to mirror the file content.
 
 This 
 https://github.com/symmetryinvestments/dud/blob/master/pkgdescription/source/dud/p
gdescription/json.d 
 is the json reader.
 
 Currently, I have no plans to add any other file format.
 But PR's are always welcome.
 The decision on json and sdl has been made a long time ago, for better
 or for worse.
 Please don't turn this thread into bike-shedding.
 Additionally, dub/dud is already on the way to add cli functions to 
 manipulate
 the config file.
 A look at adding dependencies with "dub add" already shows that.


And I would complain that the fact json exists as a file format already 
screws up dub add -- using dub add removes ALL comments in an SDL file, 
and rewrites the file in the order it sees fit.

result: I don't use dub add any more.

-Steve
Nov 19 2019
next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 11/19/19 11:30 AM, Steven Schveighoffer wrote:

 
 And I would complain that the fact json exists as a file format already 
 screws up dub add -- using dub add removes ALL comments in an SDL file, 
 and rewrites the file in the order it sees fit.
 
 result: I don't use dub add any more.


Oops, that's probably actually my fault, not dub's. Those are current 
limitations of the sdl lib, SDLang-D.

1. The (pull) parser doesn't currently emit comments (and consequently 
the DOM doesn't support them either, not that it matters for dub which 
uses the pull parser directly). IIRC, the comments are currently being 
ignored in the lexer. Shouldn't be TOO terribly hard to fix, though. 
Modify the lexer to emit them as a new token type, and adjust the 
parser's grammar logic to accept them and to emit them as a new pull 
parser event. (PR's always welcome!)

2. IIRC, the lib's only method of generating sdl involves using the DOM, 
and TBH, I don't remember offhand just how friendly/unfriendly it is to 
maintaining a specific ordering. I'd have to look into that. Ideally, 
there should probably be a range-based reverse-pull-parser that just 
takes user-generated pull parser events and spits out an sdl doc.
Nov 19 2019
parent Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 19 November 2019 at 17:13:49 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 11/19/19 11:30 AM, Steven Schveighoffer wrote:

 
 And I would complain that the fact json exists as a file 
 format already screws up dub add -- using dub add removes ALL 
 comments in an SDL file, and rewrites the file in the order it 
 sees fit.
 
 result: I don't use dub add any more.


 Oops, that's probably actually my fault, not dub's. Those are 
 current limitations of the sdl lib, SDLang-D.


dud has its own sdlang parser, I wanted to have no dependencies 
other than
phobos and SDLang-D was not pure nor  safe, mostly because of
static Key[5] keywords; in Lexer.lexIdentKeyword.
Nov 20 2019
prev sibling parent Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 19 November 2019 at 16:30:26 UTC, Steven 
Schveighoffer wrote:

 And I would complain that the fact json exists as a file format 
 already screws up dub add -- using dub add removes ALL comments 
 in an SDL file, and rewrites the file in the order it sees fit.

 result: I don't use dub add any more.

 -Steve


I haven't implemented dud add yet, but it should be as easy as
parsing the file checking if the dep is already in.
If it is not the case add a new line to the end of the file.
dub.sdl files have no order after all.
Nov 20 2019
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 18 November 2019 at 16:54:21 UTC, JN wrote:

 Personally I only ever use SDL with Dub. Even contemplating 
 using JSON for human written configuration files is, for me, 
 totally the wrong thing to do.


 I only use the JSON format. JSON is widespread together with 
 XML. SDL I heard first time of in the context of Dub and never 
 seen it used elsewhere. TOML also I know only from Cargo. YAML 
 at least I know from several different projects.

 I guess the tool must be working very well though if the main 
 argument is what kind of data format to use :)


I also only ever use JSON. When SDL was introduced, it was 
promised "JSON will never be deprecated". And here people are 
wanting to deprecate JSON as if SDL parsers were all that 
numerous.

JSON is the answer, SDL is the failed experiment that just bring 
disagreement :)

We DO parse dub.json a lot. I suspect any D organization probably 
does because dub describe is slow.

I don't care how "inhuman" a 20-line file is.
Nov 22 2019
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/11/2019 4:37 AM, Guillaume Piolat wrote:

 On Monday, 18 November 2019 at 16:54:21 UTC, JN wrote:

 Personally I only ever use SDL with Dub. Even contemplating using 
 JSON for human written configuration files is, for me, totally the 
 wrong thing to do.


 I only use the JSON format. JSON is widespread together with XML. SDL 
 I heard first time of in the context of Dub and never seen it used 
 elsewhere. TOML also I know only from Cargo. YAML at least I know from 
 several different projects.

 I guess the tool must be working very well though if the main argument 
 is what kind of data format to use :)


 
 I also only ever use JSON. When SDL was introduced, it was promised 
 "JSON will never be deprecated". And here people are wanting to 
 deprecate JSON as if SDL parsers were all that numerous.
 
 JSON is the answer, SDL is the failed experiment that just bring 
 disagreement :)
 
 We DO parse dub.json a lot. I suspect any D organization probably does 
 because dub describe is slow.
 
 I don't care how "inhuman" a 20-line file is.


+1
Nov 22 2019
parent GoaLitiuM <goalitium dforums.mail.kapsi.fi> writes:
On Saturday, 23 November 2019 at 03:57:02 UTC, rikki cattermole 
wrote:

 On 23/11/2019 4:37 AM, Guillaume Piolat wrote:

 On Monday, 18 November 2019 at 16:54:21 UTC, JN wrote:

 Personally I only ever use SDL with Dub. Even contemplating 
 using JSON for human written configuration files is, for me, 
 totally the wrong thing to do.


 I only use the JSON format. JSON is widespread together with 
 XML. SDL I heard first time of in the context of Dub and 
 never seen it used elsewhere. TOML also I know only from 
 Cargo. YAML at least I know from several different projects.

 I guess the tool must be working very well though if the main 
 argument is what kind of data format to use :)


 
 I also only ever use JSON. When SDL was introduced, it was 
 promised "JSON will never be deprecated". And here people are 
 wanting to deprecate JSON as if SDL parsers were all that 
 numerous.
 
 JSON is the answer, SDL is the failed experiment that just 
 bring disagreement :)
 
 We DO parse dub.json a lot. I suspect any D organization 
 probably does because dub describe is slow.
 
 I don't care how "inhuman" a 20-line file is.


 +1


+1, and JSON as a main format would make sense too because 
querying information from DUB registry is already being done in 
JSON, so throwing in SDL would complicate things.

There's also plenty of JSON parsers out there, some of them being 
extremely performant (like fast.json and asdf), but only one 
(AFAIK) SDL parser with no ideas how fast it is. The performance 
should not be ignored as every build operation requires parsing 
all of the dependencies and their dependencies too which can 
quickly accumulate with larger complicated projects.

The easy of modifying SDL vs JSON is a non-issue for me, you 
don't spend that much time editing the configuration files while 
developing your project.
Nov 23 2019
prev sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Mon, 2019-11-18 at 09:53 +0000, Paolo Invernizzi via Digitalmars-d-annou=
nce=20
wrote:
[=E2=80=A6]

 A win-win move would be to have dud emit the other formats=20
 automatically as part of the compilation procedure, so to have=20
 always all of them present and synced on the content.


Why? In fact, why even think of doing this at all?

There should be one and only one human written configuration file for a bui=
ld
and it should be in a notation suitable for being written by humans.


 It shouldn't be too much difficult, and maybe it's a cleaver way=20
 to move on from discussions about different formats.


Again why? It seems like a pointless overhead that achieves nothing
constructive.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 18 2019
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 18 November 2019 at 10:44:18 UTC, Russel Winder wrote:

 On Mon, 2019-11-18 at 09:53 +0000, Paolo Invernizzi via 
 Digitalmars-d-announce
 wrote:
 […]

 A win-win move would be to have dud emit the other formats 
 automatically as part of the compilation procedure, so to have 
 always all of them present and synced on the content.


 Why? In fact, why even think of doing this at all?

 There should be one and only one human written configuration 
 file for a build and it should be in a notation suitable for 
 being written by humans.


 It shouldn't be too much difficult, and maybe it's a cleaver 
 way to move on from discussions about different formats.


 Again why? It seems like a pointless overhead that achieves 
 nothing constructive.


Closing this kind of discussions and letting anyone to choose 
"tabs or spaces" is a constructive solution, I think.

That's the whole point of a win-win solution, anyone has a gain, 
the JSON party and the SDL party.

But, hey, as someone has to implement that, feel free to simply 
ignore my opinion ...
Nov 18 2019
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 18 November 2019 at 13:19:12 UTC, Paolo Invernizzi 
wrote:

 Closing this kind of discussions and letting anyone to choose 
 "tabs or spaces" is a constructive solution, I think.


It is quite extraordinary how readily folks fall to arguing over 
what the config format should be, rather than what the app should 
actually be able to do. :-\
Nov 18 2019
next sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Mon, 2019-11-18 at 15:35 +0000, Joseph Rushton Wakeling via Digitalmars-=
d-
announce wrote:

=20


[=E2=80=A6]

 It is quite extraordinary how readily folks fall to arguing over=20
 what the config format should be, rather than what the app should=20
 actually be able to do. :-\


Perhaps because writing the configuration files is a critical part of the
usability of the tool.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 18 2019
next sibling parent reply bachmeier <no spam.net> writes:
On Monday, 18 November 2019 at 19:44:46 UTC, Russel Winder wrote:

 On Mon, 2019-11-18 at 15:35 +0000, Joseph Rushton Wakeling via 
 Digitalmars-d- announce wrote:

 


 […]

 It is quite extraordinary how readily folks fall to arguing 
 over what the config format should be, rather than what the 
 app should actually be able to do. :-\


 Perhaps because writing the configuration files is a critical 
 part of the usability of the tool.


IMO this is one of the most important parts of the first five 
minutes with the language. Someone has installed the compiler, 
and now they want to test it out. If they have a bad experience 
with Dub, they will not continue with the language. A package 
manager, including the choice of format, is something you have to 
get right. Rust understands this.
Nov 18 2019
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 18 November 2019 at 20:48:53 UTC, bachmeier wrote:

 IMO this is one of the most important parts of the first five 
 minutes with the language. Someone has installed the compiler, 
 and now they want to test it out. If they have a bad experience 
 with Dub, they will not continue with the language. A package 
 manager, including the choice of format, is something you have 
 to get right. Rust understands this.


Fair point.  But that isn't something that has to be decided at 
the _start_ of a rewrite: better to focus on wanted behaviour, 
and then derive the ideal config format from that.
Nov 19 2019
parent Russel Winder <russel winder.org.uk> writes:
On Tue, 2019-11-19 at 10:38 +0000, Joseph Rushton Wakeling via Digitalmars-=
d-
announce wrote:

 On Monday, 18 November 2019 at 20:48:53 UTC, bachmeier wrote:

 IMO this is one of the most important parts of the first five=20
 minutes with the language. Someone has installed the compiler,=20
 and now they want to test it out. If they have a bad experience=20
 with Dub, they will not continue with the language. A package=20
 manager, including the choice of format, is something you have=20
 to get right. Rust understands this.


=20
 Fair point.  But that isn't something that has to be decided at=20
 the _start_ of a rewrite: better to focus on wanted behaviour,=20
 and then derive the ideal config format from that.


I'd argue that from a socio-technical perspective it is an important factor
that should be an integral part of developing any user facing tool. The
functionality of the tool is important, but how that functionality is
presented to a user new to the tool and a user proficient with the tool sho=
uld
be equally important.

As an example of how not to do things, perhaps I can suggest earlier forms =
of
Git. It took nigh on a decade to simplify things to get to the overcomplica=
ted
CLI we have today with Git.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 19 2019
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 18 November 2019 at 19:44:46 UTC, Russel Winder wrote:

 On Mon, 2019-11-18 at 15:35 +0000, Joseph Rushton Wakeling via 
 Digitalmars-d- announce wrote:

 


 […]

 It is quite extraordinary how readily folks fall to arguing 
 over what the config format should be, rather than what the 
 app should actually be able to do. :-\


 Perhaps because writing the configuration files is a critical 
 part of the usability of the tool.


Fair.  My remark was maybe a little too intemperate :-)

I'm simply concerned that if we don't put enough scrutiny on the 
app features and behaviour, we run the risk of simply reproducing 
some of the problematic design decisions of the existing tool.

As an example -- try running `dub test --build=release`.  
Intuition suggests that this is testing a release build.  But 
actually it strips out your unittests, because `--build=release` 
determines not only the optimization flags, but the complete set 
of DFLAGS used -- and `-unittest` isn't among them.

I ran `dub test && dub test --build=release` as a matter of habit 
for some time before discovering the latter wasn't actually 
testing anything.  And while I now know that I _can_ define a 
custom build to get what I want, that's clumsy and hard(er) to 
discover, and annoying to have to repeat for every project I 
create.

That's the kind of usability and design concern we really ought 
to be revisiting in any rewrite.
Nov 19 2019
parent Russel Winder <russel winder.org.uk> writes:
On Tue, 2019-11-19 at 10:32 +0000, Joseph Rushton Wakeling via Digitalmars-=
d-
announce wrote:

 [=E2=80=A6]
=20
 I'm simply concerned that if we don't put enough scrutiny on the=20
 app features and behaviour, we run the risk of simply reproducing=20
 some of the problematic design decisions of the existing tool.
=20
 As an example -- try running `dub test --build=3Drelease`. =20
 Intuition suggests that this is testing a release build.  But=20
 actually it strips out your unittests, because `--build=3Drelease`=20
 determines not only the optimization flags, but the complete set=20
 of DFLAGS used -- and `-unittest` isn't among them.
=20
 I ran `dub test && dub test --build=3Drelease` as a matter of habit=20
 for some time before discovering the latter wasn't actually=20
 testing anything.  And while I now know that I _can_ define a=20
 custom build to get what I want, that's clumsy and hard(er) to=20
 discover, and annoying to have to repeat for every project I=20
 create.
=20
 That's the kind of usability and design concern we really ought=20
 to be revisiting in any rewrite.


How can I not agree with this being a critical bug in Dub!

:-)

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Nov 19 2019
prev sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 18 November 2019 at 15:35:12 UTC, Joseph Rushton 
Wakeling wrote:

 On Monday, 18 November 2019 at 13:19:12 UTC, Paolo Invernizzi 
 wrote:

 Closing this kind of discussions and letting anyone to choose 
 "tabs or spaces" is a constructive solution, I think.


 It is quite extraordinary how readily folks fall to arguing 
 over what the config format should be, rather than what the app 
 should actually be able to do. :-\


Exactly, that's why I love pragmatism: let's dud emit the ones 
that are outdated, automatically.

Everyone can choose, work and update or the format he prefers, 
without impacting other people choices, and the discussion can 
move forward to something more interesting ...

my 2 cents, at least ...
Nov 18 2019
prev sibling next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 11 November 2019 at 13:44:28 UTC, Robert Schadek wrote:

 Now that dud can parse dub files, the next step will be a 
 semantic phase,
 checking the input for errors.
 After that, path expansion and dependency resolution.

 PR's are always welcome.

 Destroy!

 [1] https://github.com/symmetryinvestments/dud
 [2] https://github.com/symmetryinvestments/dubproxy


Are you looking to replace dub as the reference build tool for D? 
(Please say yes...)

Any estimate what the schedule looks like until dud can be used 
in practice?
Nov 15 2019
parent Robert Schadek <rschadek symmetryinvestments.com> writes:
On Friday, 15 November 2019 at 10:29:07 UTC, FeepingCreature 
wrote:

 Are you looking to replace dub as the reference build tool for 
 D? (Please say yes...)


reference build tool, I don't know. We will see.


 Any estimate what the schedule looks like until dud can be used 
 in practice?


dub convert works already ;-)

No, PR's welcome
Nov 17 2019
prev sibling next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 11 November 2019 at 13:44:28 UTC, Robert Schadek wrote:

 So dub has some problems, and personally I find its code base 
 very hard to get into.


I understand what you mean. What I found most disruptive to my 
internal model of the code is the mutable state. It prevented me 
from a solid understanding of all the parts.


 The bigger goal, at least personally, is to have a code base of 
 pure functions
 that is "trivial" to understand and debug.
 The rules of thumb is: "When your line gets longer than 80 
 characters,
 restructure your code!", "Branch statements are code smells."


I certainly applaud your goal and I think many parts can be 
improved upon. However, not everything is accidental complexity; 
sometimes problems are hard and code is non trivial. Although in 
almost all cases it can be made nicer, it is just a matter of how 
much time you want to spend on it.
Nov 17 2019
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 11 November 2019 at 13:44:28 UTC, Robert Schadek wrote:

 So dub has some problems, and personally I find its code base 
 very hard to get
 into.
 At Symmetry we are a very heavy user of dub, resulting in many 
 wasted hours.

 So I started to write dud [1]. I kept some boring/nice parts 
 from dub, but most
 code so far is a complete rewrite.

 The goal of dud is mostly do what dub does, but more 
 understandable.


Cool :-)  Since I have also been experiencing a fair bit of 
production-use DUB pain in the last year, I really appreciate 
your taking action on this.

A few things that would be good to understand up front:

   * what are the particular issues with DUB that you want to 
address?

       - making the codebase cleaner and more functional is 
obviously
         nice but is at most a means to an end -- what's the real 
end
         you have in mind?

       - I would imagine getting dependency resolution really right
         would be top of the list -- it would be good to aim to fix
         issues like https://github.com/dlang/dub/issues/1732

       - I would personally really appreciate it if you would make 
it
         a design goal to better separate concerns in terms of what
         DFLAGS are used and why (for example, the fact that right 
now
         `dub test --build=release` will not actually run 
unittests,
         as `--build=release` drops the `-unittest` flag)

   * are there any particular known issues with DUB that this 
definitely
     will NOT address?

   * are there any key _breaking_ changes you have in mind?

   * where does this stand w.r.t. some of the proposals to break 
DUB apart
     into more cleanly separated components, e.g. determining 
compatible
     dependencies, downloading dependencies, building or running 
tests ... ?

Some concrete feedback on the project as it stands:

   * the tickboxes of compatible commands are nice, but it would 
be good to
     have a more contextualized roadmap, in particular outlining 
the design
     concerns for core data structures and functionality

       - this should probably be in issues rather than the README, 
but
         it needs to be somewhere, otherwise it's hard for anyone 
outside
         to identify things they can do

       - it might be nice to use a GitHub project to manage things 
so that
         us outside folks can identify better what's being worked 
on and
         what's blocked by what

   * I don't mind breaking changes in either the config format or 
the command
     line API if it gets us to a nicer tool, so please embrace the 
opportunity
     where appropriate

       - I can imagine this might be the reason why the aim is to 
support
         a "tasteful subset" of DUB's features: it means that 
others can
         be re-implemented in an incompatible but better way

       - auto-conversion mechanisms may be preferable to outright 
support
         for old formats and commands

   * I really recommend trying to start writing clean, clear 
commit messages
     from the very start -- think of this as another form of code 
documentation
     that communicates to all interested readers the intent and 
considerations
     behind any particular change to the codebase.  This makes it 
much easier
     for outsiders to get a clear understanding of the project and 
get into the
     habit of contribution

       - right now, pretty much all the commit messages read like 
spontaneous
         notes where even YOU won't remember the whys or 
wherefores in a few
         months' time

       - long term it saves a LOT of time when trying to look back 
and understand
         "Why did we do things that way?" -- particularly useful 
when trying to
         fix some subtle design bug

   * for the same reasons, really try to provide good 
documentation and comments
     for all code from the start -- this really makes it much 
easier for anyone
     interested to grasp the major design concerns and get 
contributing

   * these concerns are going to be strongest for the key data 
structures and
     algorithms -- please make sure these are REALLY documented 
well, from the
     very start

Hope all of that's helpful, and best wishes for taking this 
forward -- I will try to help as I can, but time right now is 
very constrained ... ;-)

Thanks & best wishes,

      -- Joe
Nov 18 2019
next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 11/18/19 7:59 AM, Joseph Rushton Wakeling wrote:

        - I would imagine getting dependency resolution really right
          would be top of the list -- it would be good to aim to fix
          issues like https://github.com/dlang/dub/issues/1732


As has been discussed elsewhere a few months ago, dependency resolution 
should be outsourced to an established SAT 
<https://en.wikipedia.org/wiki/Boolean_satisfiability_problem> solving 
lib, to avoid re-inventing a notoriously difficult wheel. This is what 
all the serious package managers have been moving towards, after 
invariably hitting problems (much like dub) trying to roll-their-own.
Nov 18 2019
next sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 18 November 2019 at 16:31:09 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 11/18/19 7:59 AM, Joseph Rushton Wakeling wrote:

        - I would imagine getting dependency resolution really 
 right
          would be top of the list -- it would be good to aim 
 to fix
          issues like https://github.com/dlang/dub/issues/1732


 As has been discussed elsewhere a few months ago, dependency 
 resolution should be outsourced to an established SAT 
 <https://en.wikipedia.org/wiki/Boolean_satisfiability_problem> 
 solving lib, to avoid re-inventing a notoriously difficult 
 wheel. This is what all the serious package managers have been 
 moving towards, after invariably hitting problems (much like 
 dub) trying to roll-their-own.


OT: By saying "all the __serious__" you effectively ended this 
part of the thread.
You basically say, that whatever I do, solve P=NP for instance, 
dud will
never be a __serious__ package manager because it does not use an
existing SAT solver.
That's just bad arguing.

The thing I want from dud, is to tell me what dependency chain 
let to conflicts
and this I find hard to extract from current SAT solvers.
Most I have worked with just told me: "This solution works" "NO"
Also the debug experience is not really great most of the time.
And I like a good debug experience.

I'm not saying I will or want to reinvent the wheel on dependency 
resolution,
but if there is a choice of Understandability + debugability vs. 
performance.
Performance will suffer.
Nov 19 2019
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 11/19/19 3:29 AM, Robert Schadek wrote:

 On Monday, 18 November 2019 at 16:31:09 UTC, Nick Sabalausky (Abscissa) 
 wrote:

 As has been discussed elsewhere a few months ago, dependency 
 resolution should be outsourced to an established SAT 
 <https://en.wikipedia.org/wiki/Boolean_satisfiability_problem> solving 
 lib, to avoid re-inventing a notoriously difficult wheel. This is what 
 all the serious package managers have been moving towards, after 
 invariably hitting problems (much like dub) trying to roll-their-own.


 
 OT: By saying "all the __serious__" you effectively ended this part of 
 the thread.
 You basically say, that whatever I do, solve P=NP for instance, dud will
 never be a __serious__ package manager because it does not use an
 existing SAT solver.
 That's just bad arguing.


Dude, it's just casual speech, not formal logic. A colloquialism, not a 
formal claim of "For all X, X always necessarily implies Y".


 The thing I want from dud, is to tell me what dependency chain let to 
 conflicts
 and this I find hard to extract from current SAT solvers.
 Most I have worked with just told me: "This solution works" "NO"
 Also the debug experience is not really great most of the time.


Ok, now that I wasn't aware of.
Nov 19 2019
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 18 November 2019 at 16:31:09 UTC, Nick Sabalausky 
(Abscissa) wrote:

 As has been discussed elsewhere a few months ago, dependency 
 resolution should be outsourced to an established SAT


I have no objections to that in principle, but I am not sure that 
will resolve the issue I posted: IIRC DUB's key problem is not so 
much how it resolves dependencies _per se_ but the fact that it 
includes dependency constraints that actually don't apply to a 
given build (e.g. constraints that derive from the testing 
requirements of dependencies).
Nov 19 2019
prev sibling parent drug <drug2004 bk.ru> writes:
On 11/18/19 7:31 PM, Nick Sabalausky (Abscissa) wrote:

 On 11/18/19 7:59 AM, Joseph Rushton Wakeling wrote:

        - I would imagine getting dependency resolution really right
          would be top of the list -- it would be good to aim to fix
          issues like https://github.com/dlang/dub/issues/1732


 
 As has been discussed elsewhere a few months ago, dependency resolution 
 should be outsourced to an established SAT 
 <https://en.wikipedia.org/wiki/Boolean_satisfiability_problem> solving 
 lib, to avoid re-inventing a notoriously difficult wheel. This is what 
 all the serious package managers have been moving towards, after 
 invariably hitting problems (much like dub) trying to roll-their-own.
 


does linear constraints solver fit here too?
Nov 19 2019
prev sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 18 November 2019 at 12:59:25 UTC, Joseph Rushton 
Wakeling wrote:

 Cool :-)  Since I have also been experiencing a fair bit of 
 production-use DUB pain in the last year, I really appreciate 
 your taking action on this.

 A few things that would be good to understand up front:

   * what are the particular issues with DUB that you want to 
 address?

       - making the codebase cleaner and more functional is 
 obviously
         nice but is at most a means to an end -- what's the 
 real end
         you have in mind?


My reason for making it cleaner is, because I assume this will 
give
me a build tool I can fix when broken. And hopefully it has less
bugs to begin with because its cleaner.


       - I would imagine getting dependency resolution really 
 right
         would be top of the list -- it would be good to aim to 
 fix
         issues like https://github.com/dlang/dub/issues/1732


That is one thing yes.


       - I would personally really appreciate it if you would 
 make it
         a design goal to better separate concerns in terms of 
 what
         DFLAGS are used and why (for example, the fact that 
 right now
         `dub test --build=release` will not actually run 
 unittests,
         as `--build=release` drops the `-unittest` flag)

   * are there any particular known issues with DUB that this 
 definitely
     will NOT address?


I'm not sure yet.


   * are there any key _breaking_ changes you have in mind?

   * where does this stand w.r.t. some of the proposals to break 
 DUB apart
     into more cleanly separated components, e.g. determining 
 compatible
     dependencies, downloading dependencies, building or running 
 tests ... ?


It is build as a library first. The CLI is just using the library
constructs.


 Some concrete feedback on the project as it stands:

   * the tickboxes of compatible commands are nice, but it would 
 be good to
     have a more contextualized roadmap, in particular outlining 
 the design
     concerns for core data structures and functionality

       - this should probably be in issues rather than the 
 README, but
         it needs to be somewhere, otherwise it's hard for 
 anyone outside
         to identify things they can do


True, I'll work on that.


       - it might be nice to use a GitHub project to manage 
 things so that
         us outside folks can identify better what's being 
 worked on and
         what's blocked by what


I already started that, somewhat.


   * I don't mind breaking changes in either the config format 
 or the command
     line API if it gets us to a nicer tool, so please embrace 
 the opportunity
     where appropriate

       - I can imagine this might be the reason why the aim is 
 to support
         a "tasteful subset" of DUB's features: it means that 
 others can
         be re-implemented in an incompatible but better way

       - auto-conversion mechanisms may be preferable to 
 outright support
         for old formats and commands

   * I really recommend trying to start writing clean, clear 
 commit messages
     from the very start -- think of this as another form of 
 code documentation
     that communicates to all interested readers the intent and 
 considerations
     behind any particular change to the codebase.  This makes 
 it much easier
     for outsiders to get a clear understanding of the project 
 and get into the
     habit of contribution

       - right now, pretty much all the commit messages read 
 like spontaneous
         notes where even YOU won't remember the whys or 
 wherefores in a few
         months' time


I know, I'll try to do better


       - long term it saves a LOT of time when trying to look 
 back and understand
         "Why did we do things that way?" -- particularly useful 
 when trying to
         fix some subtle design bug

   * for the same reasons, really try to provide good 
 documentation and comments
     for all code from the start -- this really makes it much 
 easier for anyone
     interested to grasp the major design concerns and get 
 contributing


Here is disagree, to a degree I consider comments a code smell.
If I have to write them, I failed to convey the information
needed to understand the code in the code.


   * these concerns are going to be strongest for the key data 
 structures and
     algorithms -- please make sure these are REALLY documented 
 well, from the
     very start

 Hope all of that's helpful, and best wishes for taking this 
 forward -- I will try to help as I can, but time right now is 
 very constrained ... ;-)

 Thanks & best wishes,

      -- Joe


Thank you
Nov 20 2019
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 20 November 2019 at 11:40:19 UTC, Robert Schadek 
wrote:


 Here is disagree, to a degree I consider comments a code smell.
 If I have to write them, I failed to convey the information
 needed to understand the code in the code.


You think this is a code smell: 
https://dlang.org/phobos/std_array.html ?

--
/Jacob Carlborg
Nov 20 2019
next sibling parent Robert Schadek <rschadek symmetryinvestments.com> writes:
I assume you don't mean the documentation for std.array 
specifically,
but the act of having documentation of the module.

Then, yes I do think documentation should not be needed.

I think it would be far better if I only needed the signatures of
the functions and the members of the structs to use them.
That I need to look at an example usage of a function to grasp its
meaning or read some text about it, is not a good sign IMHO.

Please keep in mind, I'm striving for an ideal, not reality.
dud will be documented, no question about it.
But I think, the longer documentation is not needed to understand
dud, the better.

p.s. I find it hard to explain this in writing.
It makes perfect sense in my head. Maybe I can convince you in 
person
at the next dconf.
Nov 20 2019
prev sibling parent reply Rumbu <rumbu rumbu.ro> writes:
On Wednesday, 20 November 2019 at 13:37:39 UTC, Jacob Carlborg 
wrote:

 On Wednesday, 20 November 2019 at 11:40:19 UTC, Robert Schadek 
 wrote:


 Here is disagree, to a degree I consider comments a code smell.
 If I have to write them, I failed to convey the information
 needed to understand the code in the code.


 You think this is a code smell: 
 https://dlang.org/phobos/std_array.html ?

 --
 /Jacob Carlborg


When a function signature looks like this

ElementEncodingType!(ElementType!RoR)[] join(RoR, R)(RoR ror, 
scope R sep)
if (isInputRange!RoR && isInputRange!(Unqual!(ElementType!RoR)) 
&& isInputRange!R && is(Unqual!(ElementType!(ElementType!RoR)) == 
Unqual!(ElementType!R)))

or like this:

E[] replaceFirst(E, R1, R2)(E[] subject, R1 from, R2 to)
if (isDynamicArray!(E[]) && isForwardRange!R1 && 
is(typeof(appender!(E[])().put(from[0..1]))) && isForwardRange!R2 
&& is(typeof(appender!(E[])().put(to[0..1]))));

it's understandable why documentation is mandatory.
Nov 20 2019
parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Wednesday, 20 November 2019 at 16:29:20 UTC, Rumbu wrote:

 When a function signature looks like this

 ElementEncodingType!(ElementType!RoR)[] join(RoR, R)(RoR ror, 
 scope R sep)
 if (isInputRange!RoR && isInputRange!(Unqual!(ElementType!RoR)) 
 && isInputRange!R && is(Unqual!(ElementType!(ElementType!RoR)) 
 == Unqual!(ElementType!R)))

 or like this:

 E[] replaceFirst(E, R1, R2)(E[] subject, R1 from, R2 to)
 if (isDynamicArray!(E[]) && isForwardRange!R1 && 
 is(typeof(appender!(E[])().put(from[0..1]))) && 
 isForwardRange!R2 && 
 is(typeof(appender!(E[])().put(to[0..1]))));

 it's understandable why documentation is mandatory.


That's true, Rumbu!

And despite that, it's always marvel me the fact that I can 
simply read the above and actually "understand it"!

It's some kind of magic, but maybe it's simply why I'm forced to 
read too much C++ recently...  :-P

PS ... the most difficult part for a beginner maybe is the 
historical "is(typeof( ... bla bla ...)"
Nov 21 2019
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 20 November 2019 at 11:40:19 UTC, Robert Schadek 
wrote:

 Here is disagree, to a degree I consider comments a code smell.
 If I have to write them, I failed to convey the information
 needed to understand the code in the code.


That depends on what you're using documentation and comments for.

It's obviously great to try to write code that is as clean and 
clear and comprehensible as possible.  But in general even the 
cleanest and clearest code rarely communicates the WHY behind 
design decisions.  "This design was chosen because ..."  "These 
are the assumptions made ..." etc.

That WHY is what you want to clearly document, because that's 
what saves the most time for anyone looking to understand and 
modify the codebase.
Nov 21 2019
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/20/19 6:40 AM, Robert Schadek wrote:

 * for the same reasons, really try to provide good documentation and 
 comments
     for all code from the start -- this really makes it much easier 
 for anyone
     interested to grasp the major design concerns and get contributing


 
 Here is disagree, to a degree I consider comments a code smell.
 If I have to write them, I failed to convey the information
 needed to understand the code in the code.


Assuming the code you wrote does what you wanted it to do...

Often times, comments convey what you're thinking, and it's much easier 
to understand a description than mentally compiling and running the code 
to figure out what you were thinking.

-Steve
Nov 21 2019
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Thursday, 21 November 2019 at 14:50:59 UTC, Steven 
Schveighoffer wrote:

 Assuming the code you wrote does what you wanted it to do...

 Often times, comments convey what you're thinking, and it's 
 much easier to understand a description than mentally compiling 
 and running the code to figure out what you were thinking.


Exactly.  And of course this isn't just for the benefit of others 
-- future you often needs the exact same help to understand what 
past you was thinking ...
Nov 21 2019
prev sibling next sibling parent Robert Schadek <rschadek symmetryinvestments.com> writes:
Regarding dependency resolution:

Did anybody here had a look at what the Dart people are
doing with pubgrab?

https://github.com/dart-lang/pub/blob/master/doc/solver.md
https://medium.com/ nex3/pubgrub-2fb6470504f
https://www.youtube.com/watch?v=Fifni75xYeQ

Especially the error reporting looks promising to me.
Nov 25 2019
prev sibling next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Monday, 11 November 2019 at 13:44:28 UTC, Robert Schadek wrote:

 So dub has some problems, and personally I find its code base 
 very hard to get
 into.
 At Symmetry we are a very heavy user of dub, resulting in many 
 wasted hours.

 So I started to write dud [1]. I kept some boring/nice parts 
 from dub, but most
 code so far is a complete rewrite.

 The goal of dud is mostly do what dub does, but more 
 understandable.
 dud will/does aim for a tasteful subset of dub's features.
 Meaning, most dub files should be good with dud.
 If they are not, you will/should get an error message telling 
 you whats wrong.
 The bigger goal, at least personally, is to have a code base of 
 pure functions
 that is "trivial" to understand and debug.
 The rules of thumb is: "When your line gets longer than 80 
 characters,
 restructure your code!", "Branch statements are code smells."

 So what can dud do right now.

 $ dud convert

 works.
 IMO that is an important step, as this involves ingesting 
 dub.(json|sdl) into
 a common representation.

 I use dubproxy [2] to get all ~1.6k packages from 
 code.dlang.org and create a
 git working tree for all the versions of all the packages.
 Currently, that results in ~60k (package|dub).(json|sdl) files.
 Then, I run dud on those and see what breaks.
 Only a few percent are not ingestable by dud, and those are in 
 IHMO not valid
 anyway (to be fair, there is some strange 💩 out there).

 Now that dud can parse dub files, the next step will be a 
 semantic phase,
 checking the input for errors.
 After that, path expansion and dependency resolution.

 PR's are always welcome.

 Destroy!

 [1] https://github.com/symmetryinvestments/dud
 [2] https://github.com/symmetryinvestments/dubproxy


Is there any chance you can be convinced to join our force to 
improve Dub?
A lot of developers invested their time to either improve Dub in 
general
or to get their needed scenarios running.

My gut feeling is, it would take years to rebuild the same set of 
functionality
starting from the green field.

The code base of Dub isn't that bad. I also invested time and 
there are some
parts you can easily get into. Of course there are also some 
parts which are quite
complex but I would say this is caused by their nature 
(Dependency resolution
is a quite complex topic).
But really the other parts are straightforward.

The biggest issue with Dub is to find out whether some specific 
parts a bugs
or intended feature. But this issue you will also face if you 
rebuild the
functionality of Dub. You never know whether it was intended or 
it is a bug,
or you should it work.

Every effort which is put into Dub is immediately available for 
the whole
D community, free developers and also companies from small size 
to the biggest size.

(In no way I want to say you should not work on Dud, it is just 
my hope we can
win you to help working on Dub).

Kind regards
André
Nov 25 2019
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 25 November 2019 at 11:59:11 UTC, Andre Pany wrote:

 Is there any chance you can be convinced to join our force to 
 improve Dub?
 A lot of developers invested their time to either improve Dub 
 in general
 or to get their needed scenarios running.

 My gut feeling is, it would take years to rebuild the same set 
 of functionality
 starting from the green field.


The distinction seems academic at this stage, because nothing 
stops anyone blessing dud as dub 2.0 when it's in a good enough 
state to be used in production, and nothing stops dub devs and 
contributors from offering feedback and code to help out with 
dud.  And even if Robert were to work inside the dub repo, the 
basic tasks -- going bottom-up to rewrite core data structures 
and algorithms, etc. -- would still be mostly the same.

That's why I strongly recommend that at this stage, the focus 
should be on the behaviours we want from our build system, and 
how they should interact and overlap.

What's currently broken or impossible in DUB?  What parts of that 
can be fixed without changing the config or CLI?  And what 
improvements are most efficiently made via breaking changes?

Please, let's bring our focus on that.
Nov 25 2019
next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 25 November 2019 at 12:15:42 UTC, Joseph Rushton 
Wakeling wrote:

 What's currently broken or impossible in DUB?  What parts of 
 that can be fixed without changing the config or CLI?  And what 
 improvements are most efficiently made via breaking changes?

 Please, let's bring our focus on that.


The biggest thing for me would be incremental compilation. As 
well as a dub build and test 'watch' mode to avoid scanning the 
dependencies every time.
Nov 25 2019
next sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 25 November 2019 at 13:14:09 UTC, Sebastiaan Koppe 
wrote:

 The biggest thing for me would be incremental compilation. As 
 well as a dub build and test 'watch' mode to avoid scanning the 
 dependencies every time.


I think there are two levels to incremental compilation (IC).

1. File level IC. Meaning, if you have one file change, you only 
recompile
all files that depend on that directly or transitively. Finally, 
you relink.

2. Symbol level IC. Meaning, the compiler becomes a daemon, and 
you track
dependency on a symbol basis.

For 1. my goal with dud is to do that. My first target is to emit 
ninja files.
So IC is going to be the default, at least at first.

For 2. dud is build as a library first, meaning should the 
compiler
become a daemon at some point, libdud can be used to pull in 
packages from
code.dlang.org and resolve dub packages dependencies.
Nov 25 2019
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 25 November 2019 at 15:27:10 UTC, Robert Schadek wrote:

 On Monday, 25 November 2019 at 13:14:09 UTC, Sebastiaan Koppe 
 wrote:

 The biggest thing for me would be incremental compilation. As 
 well as a dub build and test 'watch' mode to avoid scanning 
 the dependencies every time.


 I think there are two levels to incremental compilation (IC).

 1. File level IC. Meaning, if you have one file change, you 
 only recompile
 all files that depend on that directly or transitively. 
 Finally, you relink.



This is the done already by reggae. Unfortunately, since every D 
module is effectively a header, the number of files that need to 
be recompiled is usually large, despite the fact that for most 
changes the recompilation isn't actually necessary.



 For 1. my goal with dud is to do that. My first target is to 
 emit ninja files.


That isn't going to work:

https://issues.dlang.org/show_bug.cgi?id=14188

reggae wraps dmd to act in a way that ninja can use. I'd also 
suggest that writing more ninja emitting code when reggae already 
does the job might be unnecessary.

The reason I stopped working on my own dub replacement is because 
I needed to attach to "a real build system", and since I want to 
rewrite reggae from scratch by building on the "Build Systems à 
la carte" paper...
Nov 28 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 28 November 2019 at 13:10:44 UTC, Atila Neves wrote:

 This is the done already by reggae. Unfortunately, since every 
 D module is effectively a header, the number of files that need 
 to be recompiled is usually large, despite the fact that for 
 most changes the recompilation isn't actually necessary.


Do you think it might work if it did dmd -H and make the 
auto-generated .di file and then did a content-based change 
detection on them for recompiling dependencies?

It'd prolly still do more work than it has to, but seeing .d 
changed, rebuild .di, if .di changed, rebuild other stuff might 
just avoid full rebuilds upon just simple function body changes.
Nov 28 2019
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 28 November 2019 at 14:06:39 UTC, Adam D. Ruppe 
wrote:

 On Thursday, 28 November 2019 at 13:10:44 UTC, Atila Neves 
 wrote:

 This is the done already by reggae. Unfortunately, since every 
 D module is effectively a header, the number of files that 
 need to be recompiled is usually large, despite the fact that 
 for most changes the recompilation isn't actually necessary.


 Do you think it might work if it did dmd -H and make the 
 auto-generated .di file and then did a content-based change 
 detection on them for recompiling dependencies?


Probably. In fact, I had a plan to do exactly that and measure to 
see what the difference was. I just haven't gotten around to 
doing it yet.


 It'd prolly still do more work than it has to, but seeing .d 
 changed, rebuild .di, if .di changed, rebuild other stuff might 
 just avoid full rebuilds upon just simple function body changes.


That was my idea as well.
Nov 29 2019
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Monday, 25 November 2019 at 13:14:09 UTC, Sebastiaan Koppe 
wrote:

 On Monday, 25 November 2019 at 12:15:42 UTC, Joseph Rushton 
 Wakeling wrote:

 What's currently broken or impossible in DUB?  What parts of 
 that can be fixed without changing the config or CLI?  And 
 what improvements are most efficiently made via breaking 
 changes?

 Please, let's bring our focus on that.


 The biggest thing for me would be incremental compilation.


https://github.com/atilaneves/reggae

Works right now with any (*) dub project.

* If it doesn't, please file a bug
Nov 28 2019
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Nov 25, 2019 at 12:15:42PM +0000, Joseph Rushton Wakeling via
Digitalmars-d-announce wrote:
[...]

 What's currently broken or impossible in DUB?


I'm probably not the intended audience here, but just so it's out there,
here's a list of dub showstoppers for me:

- lack of support for multi-language projects.  E.g., a mixed C++/D
  codebase, or a Java/D codebase.  (I didn't check, but supposedly this
  can be done as a pre/post action, but it's hacky and not
  well-integrated into the dependency resolution system).

- lack of support for build-time code generation (i.e., build a subset
  of files into an executable, run the executable to generate .d files,
  compile output .d files plus other existing .d files into final
  product).

- lack of support for cross-compilation (e.g., cross-compile to Android
  from a Linux x86 host, or cross-compile from Linux host to Windows
  executable via wine / cygwin).

- lack of support for multiple targets (i.e., from given subsets of
  source files (with overlapping core files), build one executable for
  Linux/X11, build another executable in APK form for Android target).

- unfriendly towards integration with other existing build systems
  (e.g., build a sub-project with cmake, then integrate products into
  current D project with dub).

Probably some (all?) of these will require breaking changes because of
the way dub is designed, so I'm not expecting this list will be met
anytime in the near future. Though I'd love to be pleasantly surprised!


--T
Nov 25 2019
next sibling parent reply GreatSam4sure <greatsam4sure gmail.com> writes:
On Monday, 25 November 2019 at 18:28:55 UTC, H. S. Teoh wrote:

 On Mon, Nov 25, 2019 at 12:15:42PM +0000, Joseph Rushton 
 Wakeling via Digitalmars-d-announce wrote: [...]

 [...]


 I'm probably not the intended audience here, but just so it's 
 out there, here's a list of dub showstoppers for me:

 [...]



I am interested in D/java project. Can you help me with material 
or link on that. I Will be happy to use javafx as front end and D 
as back end for a desktop App.

I have looking on GraalVm but no luck yet. Just need a good 
tutorial for a start-up
Nov 25 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Nov 25, 2019 at 07:48:46PM +0000, GreatSam4sure via
Digitalmars-d-announce wrote:
[...]

 I am interested in D/java project. Can you help me with material or
 link on that. I Will be happy to use javafx as front end and D as back
 end for a desktop App.
 
 I have looking on GraalVm but no luck yet. Just need a good tutorial
 for a start-up


So far, my only experience with interfacing Java with D is my Android
project, that involves Java code talking to Android's Java GUI APIs,
communicating with D code via JNI.

If you wish to go this route, I'd recommend reading up on JNI. The docs
are mostly geared for interfacing with C/C++ code, but it should be
relatively easy to interface with D code the same way.  You just need to
declare JNI methods in a D module so that D code can call it. Also, need
to be aware of JNI quirks to avoid runtime crashes.  With D's
compile-time introspection capabilities, I was able to come up with a
quite-nice wrapper for the JNI API, that saves a lot of typing (JNI is
notoriously verbose to setup per call across the JNI boundary).

In a nutshell, you declare native methods in your Java class(es), then
write a D module with appropriately-named functions according to JNI
naming convention, and compile that into an .so (or .lib), then use
System.loadLibrary() to load the methods at runtime, then call away.

This assumes the main program is started in Java, of course.  I haven't
tried doing it the other way (having main() in D, and calling Java from
there), but I imagine it'd be pretty similar: link with a JVM, and in
D's main() call whatever methods it takes to create the VM, then use JNI
to call Java methods in that VM instance.


Compiling a mixed-language project is a separate issue.  Dub is, sad to
say, of no help here. You're better off using something like CMake, or
something of that sort. I use SCons, but it's not as well-known and does
require a bit of an initial learning curve.


T

-- 
If the comments and the code disagree, it's likely that *both* are wrong. --
Christopher
Nov 25 2019
prev sibling parent =?UTF-8?B?UsOpbXkgTW91w6t6YQ==?= <remy.moueza gmail.com> writes:
On Monday, 25 November 2019 at 19:48:46 UTC, GreatSam4sure wrote:

 On Monday, 25 November 2019 at 18:28:55 UTC, H. S. Teoh wrote:

 On Mon, Nov 25, 2019 at 12:15:42PM +0000, Joseph Rushton 
 Wakeling via Digitalmars-d-announce wrote: [...]

 [...]


 I'm probably not the intended audience here, but just so it's 
 out there, here's a list of dub showstoppers for me:

 [...]



 I am interested in D/java project. Can you help me with 
 material or link on that. I Will be happy to use javafx as 
 front end and D as back end for a desktop App.

 I have looking on GraalVm but no luck yet. Just need a good 
 tutorial for a start-up


I once got a php/D bridge proof of concept, that should have 
worked with Java as well:
- create some D utility code,
- make an extern (C) API,
- write the C headers of the API,
- use swig (http://www.swig.org/) to wrap the C API to the 
language of choice.
- Once loaded, the module has to initialize the D runtime, by a 
call to core.runtime.rt_init() or 
core.runtime.Runtime.initialize() 
(https://dlang.org/phobos/core_runtime.html).

Swig generates shared objects. This worked well on Linux. I am 
not sure how the dll support is on Windows though: it could get 
more difficult.

Back then, the extern (C++) wasn't as extensive as it is now, so 
today it might be more pertinent to create an extern (C++) API to 
conveniently wrap extern (C++) D classes/interfaces to Java 
classes/interfaces.

These pages of the documentation detail how to interface with C 
and C++:
https://dlang.org/spec/interfaceToC.html
https://dlang.org/spec/cpp_interface.html

Maybe the extern (C/C++) API approach can be used with GraalVM, I 
haven't researched that yet. When I tried GraalVM (for ahead of 
time native compilation) I got into difficulties with libraries 
using the java reflection API. One need to get the "pro" GraalVM 
distribution if I recall correctly. It seemed too much of a 
hurdle for what I was evaluating. I did eventually did fine 
without GraalVM.

The C/C++ API from D process is quite involving. If I were to get 
Java and D together I would explore a D server communicating with 
a Java client over a TCP socket or HTTP, maybe using json:
- Adam D. Ruppe wrote a D socket tutorial not long ago: 
https://forum.dlang.org/post/ciiiskgfeyhqfkfuxdai forum.dlang.org
- I used asdf (http://code.dlang.org/packages/asdf) for JSON 
serialization and deserialization with success. There is also 
std.json already available in phobos.
- Maybe a small http server with vibe.d could be started with 
even less code.

For a desktop app, I would rather use dlangui 
(http://code.dlang.org/packages/dlangui). There are other 
maintained GUI libraries for D.

Ron Tarrant has written many tutorials an GtkD on his blog page:
- https://gtkdcoding.com/
- http://code.dlang.org/packages/gtk-d

There is also DWT, a D port of the SWT Java GUI library: 
http://code.dlang.org/packages/dwt

TKD is a binding over the Tcl/Tk toolkit:
- http://code.dlang.org/packages/tkd
Nov 25 2019
prev sibling next sibling parent reply kinke <noone nowhere.com> writes:
On Monday, 25 November 2019 at 18:28:55 UTC, H. S. Teoh wrote:

 - lack of support for cross-compilation (e.g., cross-compile to 
 Android
   from a Linux x86 host, or cross-compile from Linux host to 
 Windows
   executable via wine / cygwin).


You're not up to speed, this has been solved with latest dub and 
LDC. No need for any Wine/Cygwin/MinGW/... either.
Nov 25 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Nov 25, 2019 at 09:51:31PM +0000, kinke via Digitalmars-d-announce
wrote:

 On Monday, 25 November 2019 at 18:28:55 UTC, H. S. Teoh wrote:

 - lack of support for cross-compilation (e.g., cross-compile to
   Android from a Linux x86 host, or cross-compile from Linux host to
   Windows executable via wine / cygwin).


 
 You're not up to speed, this has been solved with latest dub and LDC.
 No need for any Wine/Cygwin/MinGW/... either.


Oooh very nice!!  That's wonderful to hear.  So you're saying LDC
out-of-the-box can cross-compile from Linux to Windows directly?  How to
do this?  I'm *very* interested!


T

-- 
What did the alien say to Schubert? "Take me to your lieder."
Nov 25 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 25 November 2019 at 23:46:31 UTC, H. S. Teoh wrote:

 Oooh very nice!!  That's wonderful to hear.  So you're saying 
 LDC out-of-the-box can cross-compile from Linux to Windows 
 directly?  How to do this?  I'm *very* interested!


ldc2 -mtriple=x86_64-pc-windows-msvc

though you will prolly have to grab the druntime libs from the 
windows build.

this same basic thing works for android and mac os too.
Nov 25 2019
next sibling parent tchaloupka <chalucha gmail.com> writes:
On Tuesday, 26 November 2019 at 00:32:54 UTC, Adam D. Ruppe wrote:

 On Monday, 25 November 2019 at 23:46:31 UTC, H. S. Teoh wrote:

 Oooh very nice!!  That's wonderful to hear.  So you're saying 
 LDC out-of-the-box can cross-compile from Linux to Windows 
 directly?  How to do this?  I'm *very* interested!


 ldc2 -mtriple=x86_64-pc-windows-msvc

 though you will prolly have to grab the druntime libs from the 
 windows build.

 this same basic thing works for android and mac os too.


Or with dub:

dub build --arch=x86_64-pc-windows-msvc

You only need to provide windows libs from windows ldc package to 
your linux installation and tweak the config.
See https://wiki.dlang.org/Cross-compiling_with_LDC for details.

Using it every day now and it works like a charm.
Nov 25 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 26, 2019 at 12:32:54AM +0000, Adam D. Ruppe via
Digitalmars-d-announce wrote:

 On Monday, 25 November 2019 at 23:46:31 UTC, H. S. Teoh wrote:

 Oooh very nice!!  That's wonderful to hear.  So you're saying LDC
 out-of-the-box can cross-compile from Linux to Windows directly?
 How to do this?  I'm *very* interested!


 
 ldc2 -mtriple=x86_64-pc-windows-msvc
 
 though you will prolly have to grab the druntime libs from the windows
 build.
 
 this same basic thing works for android and mac os too.


Hmm. I tried this (test.d is a hello world program):

	ldc2 -mtriple=x86_64-pc-windows-msvc test.d

and got this:

	Error: failed to locate link.exe

Where am I supposed to get link.exe?


T

-- 
That's not a bug; that's a feature!
Nov 26 2019
parent reply kinke <noone nowhere.com> writes:
On Tuesday, 26 November 2019 at 21:19:58 UTC, H. S. Teoh wrote:

 	Error: failed to locate link.exe

 Where am I supposed to get link.exe?


You're supposed to consult the dedicated Wiki page linked by 
tchaloupka above (also linked in DMD 2.089 and LDC 1.18 release 
notes) in such a case, where you can read up on the used 
cross-linker, the -link-internally switch, and that it's 
supported in *official* prebuilt packages (i.e., the GitHub 
downloads, we don't officially support any distros). If you don't 
want to or cannot use the official package, then you can also 
install an external lld and use it via -linker=lld-link.

On Monday, 25 November 2019 at 23:46:31 UTC, H. S. Teoh wrote:

 Oooh very nice!!  That's wonderful to hear.  So you're saying 
 LDC out-of-the-box can cross-compile from Linux to Windows 
 directly?


Almost out of the box as mentioned; Windows targets are 
particularly simple (no need for a cross-gcc/clang toolchain). 
This has been possible for over 2 years now (LDC v1.3); the MS 
libs aren't required anymore starting with LDC v1.13. Glancing 
over the release notes does pay off sometimes. ;)
Nov 26 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 27, 2019 at 12:24:11AM +0000, kinke via Digitalmars-d-announce
wrote:

 On Tuesday, 26 November 2019 at 21:19:58 UTC, H. S. Teoh wrote:

 	Error: failed to locate link.exe
 
 Where am I supposed to get link.exe?


 
 You're supposed to consult the dedicated Wiki page linked by
 tchaloupka above (also linked in DMD 2.089 and LDC 1.18 release notes)
 in such a case,


[...]

I did consult the LDC cross-compiling page, and did modify ldc2.conf
appropriately. But somewhere along the line I must have done something
wrong, because it's still not cross-compiling properly. It's using the
built-in LLD now, so the link.exe error is gone, but now it's coming
back with a bunch of undefined druntime symbols even though I did
specify the path to the prebuilt Windows druntime-ldc.lib in the
ldc2.conf.

I gave up and resorted to installing the Windows build of ldc2 under
Wine, and now I have a build script that uses wineconsole to compile
Windows executables. :-P  So far, it appears to work.


T

-- 
Don't throw out the baby with the bathwater. Use your hands...
Nov 26 2019
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 26, 2019 at 05:10:41PM -0800, H. S. Teoh via Digitalmars-d-announce
wrote:
[...]

 I did consult the LDC cross-compiling page, and did modify ldc2.conf
 appropriately. But somewhere along the line I must have done something
 wrong, because it's still not cross-compiling properly. It's using the
 built-in LLD now, so the link.exe error is gone, but now it's coming
 back with a bunch of undefined druntime symbols even though I did
 specify the path to the prebuilt Windows druntime-ldc.lib in the
 ldc2.conf.


[...]

Just for the record, I finally figured out that it was a mistake on my
part -- I overlooked some source files when testing the
cross-compilation so that's why it had link errors.  After copying over
the Windows libs and setting up ldc2.conf properly, cross-compilation to
Windows does actually work flawlessly.

Absolutely awesome. Kudos to the LDC team for making this happen!!


T

-- 
A bend in the road is not the end of the road unless you fail to make the turn.
-- Brian White
Nov 27 2019
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 26, 2019 at 01:19:58PM -0800, H. S. Teoh via Digitalmars-d-announce
wrote:

 On Tue, Nov 26, 2019 at 12:32:54AM +0000, Adam D. Ruppe via
Digitalmars-d-announce wrote:


[...]

 ldc2 -mtriple=x86_64-pc-windows-msvc
 
 though you will prolly have to grab the druntime libs from the windows
 build.




[...]

Tried installing the libs from the LDC windows build and editing
ldc2.conf to setup the paths, etc., but still no go. It finally does
find the linker and the Windows .lib's, but then it spews a whole bunch
of undefined symbol errors.

Decided instead to just run the native Windows version of ldc2 under
Wine (via wineconsole); works like a charm.  For now, at least for
building Windows executables, I'm not gonna bother with cross-compiling.
Wine appears to do the Right Thing(tm).


T

-- 
Those who've learned LaTeX swear by it. Those who are learning LaTeX swear at
it. -- Pete Bleackley
Nov 26 2019
prev sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Monday, 25 November 2019 at 18:28:55 UTC, H. S. Teoh wrote:

 - lack of support for build-time code generation (i.e., build a 
 subset
   of files into an executable, run the executable to generate 
 .d files,
   compile output .d files plus other existing .d files into 
 final
   product).


I am doing exactly that. It can be done through 
preGenerateCommands, calling either a secondary build script or 
nested dub project, optionally using excludedSourceFiles. The 
only reason it doesn’t work well is because of 
https://github.com/dlang/dub/issues/1474. But there are 
workarounds, and a poc fix.

Bastiaan.
Dec 01 2019
prev sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
dud needs your help.

I'm starting work on the dependency resolution and for that I had
to implement proper handling for Semantic Versions, Version 
Ranges,
and Version Unions(VersionUnion is basically a VersionRange[]).
The dependency resolution algorithm I want to implement (based on
the algorithm used by Dart) needs a few checks and operations on
those types.

```D
SemVer parseSemVer(string input);

Nullable!VersionRange parseVersionRange(string input);

alias Types = AliasSeq!((SemVer,VersionRange,VersionUnion);
static foreach(T, Types) {
     auto inv = invert(T);
     static foreach(S, Types) {
         bool allowsAll(T, T);
         bool allowsAll(T, S);
         bool allowsAll(S, T);

         bool allowsAny(T, T);
         bool allowsAny(T, S);
         bool allowsAny(S, T);

         auto unionOf(T, T);
         auto unionOf(T, S);
         auto unionOf(S, T);

         auto intersectionOf(T, T);
         auto intersectionOf(T, S);
         auto intersectionOf(S, T);

         auto differenceOf(T, T);
         auto differenceOf(T, S);
         auto differenceOf(S, T);
     }
}
```

I think I did okay work and the tests cover all/most cases.
But that's properly being a bit overconfident.

Therefore, it would be awesome if you could try to break
these functions and create PRs that break the functions.

The code can be found in the git here 
https://github.com/symmetryinvestments/dud
the relevant folder is semver.
The tests are located in the files:

allowsAny: semver/source/dud/semver/checkstest.d
allowsAll: semver/source/dud/semver/checkstest1.d
allowsAll: semver/source/dud/semver/setoperationtest.d
intersectionOf: semver/source/dud/semver/setoperationtest1.d
invert, differenceOf: semver/source/dud/semver/setoperationtest2.d
semver/source/dud/semver/versionrangetest.d

Building dud, and semver should be as easy as cloning and typing 
dud, dub test.

```sh
git clone https://github.com/symmetryinvestments/dud.git
cd dud/semver
dub test
```
should get you going.

DESTROY!
Jan 23 2020
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 23 January 2020 at 17:05:10 UTC, Robert Schadek 
wrote:

 DESTROY!


Tried:

---
unittest {
   immutable SemVer v1 = SemVer(1,1,1);
   immutable SemVer v2 = SemVer(2,2,2);
   immutable SemVer v3 = SemVer(3,3,3);
   immutable SemVer v4 = SemVer(4,4,4);
   immutable VersionRange a = VersionRange(v1, Inclusive.yes, v2, 
Inclusive.yes);
   immutable VersionRange b = VersionRange(v3, Inclusive.yes, v4, 
Inclusive.yes);
   immutable VersionRange c = VersionRange(v2, Inclusive.yes, v3, 
Inclusive.yes);

   assert(intersectionOf(a, invert(a)).ranges.length == 0);
   assert(intersectionOf(a, a) == a);
   assert(intersectionOf(a, b) != a);
   assert(intersectionOf(a, b) != b);
   assert(intersectionOf(b, a) != a);
   assert(intersectionOf(b, a) != b);
   assert(intersectionOf(invert(a), invert(a)) == invert(a));
   assert(invert(intersectionOf(a, invert(b))) == 
invert(intersectionOf(a,a)));

   auto add(T, P)(T a, P b) {
	return invert(intersectionOf(invert(a), invert(b)));
   }
   auto remove(T, P)(T a, P b) {
	return intersectionOf(a, invert(b));
   }

   assert(add(a,b) == add(b,a));

   assert(invert(invert(a)) == add(intersectionOf(a,c), 
remove(a,c)));
   assert(differenceOf(a, a).ranges.length == 0);
   assert(invert(intersectionOf(invert(a), invert(a))) == 
invert(invert(a)));
   assert(invert(invert(differenceOf(a, invert(a)))) == 
invert(invert(a)));
   assert(invert(invert(intersectionOf(a, c))) == 
invert(invert(remove(a, remove(a, c)))));
}
---

Everything green. Nice work.

Haven't tried Inclusive.no yet. I'll leave that to someone else.
Jan 23 2020
next sibling parent Robert Schadek <rschadek symmetryinvestments.com> writes:
On Thursday, 23 January 2020 at 20:50:09 UTC, Sebastiaan Koppe 
wrote:

 Haven't tried Inclusive.no yet. I'll leave that to someone else.


Indirectly you already did, invert turns Inclusive.yes bound into 
Inclusive.no bounds.
Jan 24 2020
prev sibling parent Robert Schadek <rschadek symmetryinvestments.com> writes:
Thank you, very nice test
Jan 24 2020