www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dub: JSON, SDL, YAML, TOML, what color should we paint it ?

reply Mathias LANG <geod24 gmail.com> writes:
Currently, Dub uses two formats: JSON and SDL.

JSON was there from the beginning, I assume due to Dub's npm 
inspiration.
In fact, we still very much depend on JSON internally: until 
recently, JSON object were passed around in quite a few places. 
When we download a package, regardless of the format the package 
recipe is in, we store it as `dub.json`.

But JSON is a terrible format to write configurations in, given 
how verbose it is, and it lacking support for comments.
So around 2014, we added support for SDL (no, not [this 
one](https://www.libsdl.org/), or [this 
one](https://web.fe.up.pt/~mricardo/05_06/amsr/artigos/sdlTutorial.pdf), but
[this one](https://sdlang.org/), and it took me 10 minutes to find the link
because Wikipedia doesn't even know about it).

At some point last year, I did some heavy refactoring on dub 
along with quite a few improvements. In the process of doing so, 
I took advantage of the fact that YAML is a subset of JSON to add 
[a YAML parser](https://github.com/dlang-community/D-YAML/), on 
which I have built a 
[library](https://github.com/dlang-community/configy) to do 
automatic configuration parsing, because let's be honest, it's 
2023 and we're using the language with the best compile-time 
reflection capabilities, so we really don't want to manually 
parse a standard file format to fill a struct.

This parser is used to parse all JSON files:
- `dub.json` (but not `dub.sdl`);
- `dub.selections.json`;
- `dub.settings.json`;

Now, I think that having `dub.settings.json` being written in 
JSON doesn't make sense, and I would like to [change 
this](https://github.com/dlang/dub/pull/2546). In fact, I have 
been wanting this for more than 3 years, and the people that 
[took part in the 
discussion](https://github.com/dlang/dub/issues/1832) broadly 
agreed.

Here are the options that have been floated around:
1) Use SDL everywhere;
2) Add support for YAML;
3) Add support for another format (E.g. TOML);

I would like to go with YAML, not because it is the greatest 
format on earth, but because SDLang is not well known (it means 
newcomer have to learn yet another format, and while not a big 
deal, it adds friction), while YAML is everywhere. In the [thread 
were SDL was 
discussed](https://forum.dlang.org/thread/gnfctbuhiemidetngr
i forum.dlang.org), people even thought we invented it!

In practice, this would mean adding support for the file format 
with a YAML extension, giving it priority over the old format, 
then after a number of releases (most likely 10, probably more 
for `dub.json`), deprecating the JSON / SDL equivalent.

Obviously such a change would not happen overnight, and would 
need broad support from the community. Opinions ?
Feb 28 2023
next sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
I don't enjoy JSON so i gave SDL a try few months ago, i quickly 
gave up, can't add new lines, can't add 'platform' at the 
beginning, nice for hello worlds, horrible for real world 
projects or simply when you need to manage multiple configuration

JSON doesn't support comments, it's noisy, and inflate the number 
of indentation for no reason

YAML is nice, a simple INI even better, LUA even more better 
(premake/xmake anyone?)

LUA would make managing multiple configurations easier, i'd 
experiment with adding it, but DUB takes an eternity to compile 
for me to do any work with it unfortunately
Feb 28 2023
prev sibling next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Obviously such a change would not happen overnight, and would 
 need broad support from the community. Opinions ?


Here's a wild idea: use the D language. Most the things dub.json 
defines are already available in D anyway.
Feb 28 2023
next sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
On Tuesday, 28 February 2023 at 15:15:33 UTC, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG 
 wrote:

 Obviously such a change would not happen overnight, and would 
 need broad support from the community. Opinions ?


 Here's a wild idea: use the D language. Most the things 
 dub.json defines are already available in D anyway.


This is the proper solution indeed, i didn't mention it because 
i'm working on a build.d project haha
Feb 28 2023
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Tue, Feb 28, 2023 at 03:15:33PM +0000, Adam D Ruppe via Digitalmars-d wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Obviously such a change would not happen overnight, and would need
 broad support from the community. Opinions ?


 
 Here's a wild idea: use the D language. Most the things dub.json
 defines are already available in D anyway.


Yes!!  This is the way to go.

At least, use a reasonable subset of D if being Turing-complete is not
necessarily a good thing. :-P  Why make the user learn a different
syntax?  We're already writing D.  Just reuse D syntax.


T

-- 
"Maybe" is a strange word.  When mom or dad says it it means "yes", but when my
big brothers say it it means "no"! -- PJ jr.
Feb 28 2023
parent reply Mathias LANG <geod24 gmail.com> writes:
On Tuesday, 28 February 2023 at 16:04:42 UTC, H. S. Teoh wrote:

 On Tue, Feb 28, 2023 at 03:15:33PM +0000, Adam D Ruppe via 
 Digitalmars-d wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG 
 wrote:

 Obviously such a change would not happen overnight, and 
 would need broad support from the community. Opinions ?


 
 Here's a wild idea: use the D language. Most the things 
 dub.json defines are already available in D anyway.


 Yes!!  This is the way to go.

 At least, use a reasonable subset of D if being Turing-complete 
 is not necessarily a good thing. :-P  Why make the user learn a 
 different syntax?  We're already writing D.  Just reuse D 
 syntax.


 T


There's a whole world between "imperative" recipes (à la 
Makefile) and declarative ones. I didn't come up with the 
declarative decision for dub, but I don't think it makes sense to 
pivot at this point. And such a pivot would be a much more 
complex discussion than just the format.

Unless you decide to go with D-style, declarative only DSL, but 
is there really a point to that ? Also it means writting yet 
another custom parser, and building something that is inherently 
not easy to interface with. We want more tooling, not less!
Feb 28 2023
parent Kagamin <spam here.lot> writes:
On Tuesday, 28 February 2023 at 16:23:57 UTC, Mathias LANG wrote:

 There's a whole world between "imperative" recipes (à la 
 Makefile) and declarative ones.


Makefile is declarative though. And here's average yaml: 
https://github.com/dlang/dmd/blob/master/.cirrus.yml
Mar 02 2023
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/28/23 10:15 AM, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Obviously such a change would not happen overnight, and would need 
 broad support from the community. Opinions ?


 
 Here's a wild idea: use the D language. Most the things dub.json defines 
 are already available in D anyway.


Is there a way to do this and prevent mischief? Like maybe:

```d
module buildfile;

import safe.modules.only;

static immutable buildcode = import("build.d");

shared static this() {
    import dparser;
    parseEnsuringNoImportsOrMixins(buildcode);
}

void build() {
    mixin(buildcode);
}
```

-Steve
Feb 28 2023
next sibling parent reply Bradley Chatha <sealabjaster gmail.com> writes:
On Tuesday, 28 February 2023 at 16:10:19 UTC, Steven 
Schveighoffer wrote:

 Is there a way to do this and prevent mischief? Like maybe:


Unlikely: In your example they can still use imports by doing 
something like `mixin("impor"~"t std.file;");`

Besides there's also inline ASM which someone _very_ motivated 
could use to do stuff with.

The previous suggestion of a subset/sibling version of D 
specifically for configuration sounds interesting though.
Feb 28 2023
parent Bradley Chatha <sealabjaster gmail.com> writes:
On Tuesday, 28 February 2023 at 16:15:44 UTC, Bradley Chatha 
wrote:

 Unlikely: In your example they can still use imports by doing 
 something like `mixin("impor"~"t std.file;");`


I'm a numpty, I didn't read your post carefully enough :D
Feb 28 2023
prev sibling next sibling parent reply Commander Zot <no no.no> writes:
On Tuesday, 28 February 2023 at 16:10:19 UTC, Steven 
Schveighoffer wrote:

 On 2/28/23 10:15 AM, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG 
 wrote:

 Obviously such a change would not happen overnight, and would 
 need broad support from the community. Opinions ?


 
 Here's a wild idea: use the D language. Most the things 
 dub.json defines are already available in D anyway.


 Is there a way to do this and prevent mischief? Like maybe:

 ```d
 module buildfile;

 import safe.modules.only;

 static immutable buildcode = import("build.d");

 shared static this() {
    import dparser;
    parseEnsuringNoImportsOrMixins(buildcode);
 }

 void build() {
    mixin(buildcode);
 }
 ```

 -Steve


whats preventing some buildfile from including some malicious cmd 
line in dub?
if you want to restrict buildfiles, they become pretty useless 
for a lot of cases
Feb 28 2023
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/28/23 11:18 AM, Commander Zot wrote:


 whats preventing some buildfile from including some malicious cmd line 
 in dub?
 if you want to restrict buildfiles, they become pretty useless for a lot 
 of cases


You need to at least restrict them to things they should be doing. 
Opening up the entire world of compiled programs to building means you 
can make extremely brittle build files.

Note also that with such a mechanism, we can have more control over 
building than current dub has. For instance, you could configure a 
global "security" import that validates the build file, so it can e.g. 
only run approved external programs.

I would also like to draw attention to the dmd build.d setup:

https://github.com/dlang/dmd/blob/master/compiler/src/build.d

It's a complex mess that's nigh unreadable/unmaintainable.

Keeping build files *simple* should be a goal.

That being said, there are plenty of languages (I know swift is one of 
them) that use the language to configure/direct builds and package 
management. I think it's a reasonable and promising idea. We should also 
be looking at the precedents and see how they tackled it.

-Steve
Feb 28 2023
prev sibling next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Tuesday, 28 February 2023 at 16:10:19 UTC, Steven 
Schveighoffer wrote:

 module buildfile;


This isn't what I'm talking about at all.

I don't want to run D to build, I want to *introspect* D for 
build info. You can read module names, dependencies (including 
platform-specific and configuration-specific ones) right out of D 
code. Author, description, license, etc., have standard ddoc 
sections in D code.

Other things dub.json define like flags could easily be a UDA on 
the module declaration and be pulled out pretty easily by 
compilers or even relatively basic parsers.

Even subconfigurations can... kinda be pulled out of D code by 
looking for version blocks though that's probably another thing 
I'd put on a UDA.
Feb 28 2023
next sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Tue, Feb 28, 2023 at 04:32:42PM +0000, Adam D Ruppe via Digitalmars-d wrote:

 On Tuesday, 28 February 2023 at 16:10:19 UTC, Steven Schveighoffer wrote:

 module buildfile;


 
 This isn't what I'm talking about at all.
 
 I don't want to run D to build, I want to *introspect* D for build
 info. You can read module names, dependencies (including
 platform-specific and configuration-specific ones) right out of D
 code. Author, description, license, etc., have standard ddoc sections
 in D code.
 
 Other things dub.json define like flags could easily be a UDA on the
 module declaration and be pulled out pretty easily by compilers or
 even relatively basic parsers.
 
 Even subconfigurations can... kinda be pulled out of D code by looking
 for version blocks though that's probably another thing I'd put on a
 UDA.


Hmm.  Now THIS is an awesome idea.  Instead of having a separate build
description apart from the code, inspect the code itself to figure out
how to build it.  With the help of UDAs and other such annotations to
help things along.  This is definitely a direction worth exploring!


T

-- 
When you breathe, you inspire. When you don't, you expire. -- The Weekly Reader
Feb 28 2023
prev sibling next sibling parent reply Bradley Chatha <sealabjaster gmail.com> writes:
On Tuesday, 28 February 2023 at 16:32:42 UTC, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 16:10:19 UTC, Steven 
 Schveighoffer wrote:

 module buildfile;


 This isn't what I'm talking about at all.

 I don't want to run D to build, I want to *introspect* D for 
 build info. You can read module names, dependencies (including 
 platform-specific and configuration-specific ones) right out of 
 D code. Author, description, license, etc., have standard ddoc 
 sections in D code.

 Other things dub.json define like flags could easily be a UDA 
 on the module declaration and be pulled out pretty easily by 
 compilers or even relatively basic parsers.

 Even subconfigurations can... kinda be pulled out of D code by 
 looking for version blocks though that's probably another thing 
 I'd put on a UDA.


Oh wow, that's definitely an interesting idea.

Have you written down a more refined version of these thoughts?
Feb 28 2023
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Tuesday, 28 February 2023 at 17:46:25 UTC, Bradley Chatha 
wrote:

 Have you written down a more refined version of these thoughts?


http://dpldocs.info/this-week-in-d/Blog.Posted_2022_03_28.html
http://dpldocs.info/this-week-in-d/Blog.Posted_2022_04_25.html


Even though more often than not, I just delete people's dub.jsons 
or makefiles or reggaefiles and what not and use a plain `dmd -i 
main.d` command (possibly adding more flags for configs) and it 
works reliably and like 2x faster than the other options.

I think people underestimate how well dmd -i works.
Mar 01 2023
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 1 March 2023 at 14:06:29 UTC, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 17:46:25 UTC, Bradley Chatha 
 wrote:

 Have you written down a more refined version of these thoughts?


 http://dpldocs.info/this-week-in-d/Blog.Posted_2022_03_28.html
 http://dpldocs.info/this-week-in-d/Blog.Posted_2022_04_25.html


 Even though more often than not, I just delete people's 
 dub.jsons or makefiles or reggaefiles and what not and use a 
 plain `dmd -i main.d` command (possibly adding more flags for 
 configs) and it works reliably and like 2x faster than the 
 other options.

 I think people underestimate how well dmd -i works.


For most simple dub packages, I'd be surprised if it didn't. Any 
that actually need a build system, well...
Mar 01 2023
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Wednesday, 1 March 2023 at 14:12:11 UTC, Atila Neves wrote:

 Any that actually need a build system, well...


This is the minority of packages... and dub's current thing tends 
to be inadequate for them anyway especially since they generally 
need interop with third party things.
Mar 01 2023
parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 1 March 2023 at 14:26:51 UTC, Adam D Ruppe wrote:

 On Wednesday, 1 March 2023 at 14:12:11 UTC, Atila Neves wrote:

 Any that actually need a build system, well...


 This is the minority of packages...


Yes. The *vast* majority are "the code is in source, have at it".


 and dub's current thing tends to be inadequate for them anyway


Yes.


 especially since they generally need interop with third party 
 things.


Yes.

As a build system, dub is a decent package manager (tongue firmly 
in cheek).
Mar 02 2023
prev sibling parent bachmeier <no spam.net> writes:
On Wednesday, 1 March 2023 at 14:06:29 UTC, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 17:46:25 UTC, Bradley Chatha 
 wrote:

 Have you written down a more refined version of these thoughts?


 http://dpldocs.info/this-week-in-d/Blog.Posted_2022_03_28.html
 http://dpldocs.info/this-week-in-d/Blog.Posted_2022_04_25.html


 Even though more often than not, I just delete people's 
 dub.jsons or makefiles or reggaefiles and what not and use a 
 plain `dmd -i main.d` command (possibly adding more flags for 
 configs) and it works reliably and like 2x faster than the 
 other options.

 I think people underestimate how well dmd -i works.


What works well if you are using the same dependencies in 
multiple projects is `-i` plus symbolic links of all dependencies 
into the appropriate subdirectories. Changes to those libraries 
are reflected immediately everywhere, and updated with a git 
pull. You can add the symbolic link creation to the Makefile if 
you're working on multiple machines.
Mar 02 2023
prev sibling next sibling parent reply tastyminerals <tastyminerals gmail.com> writes:
On Tuesday, 28 February 2023 at 16:32:42 UTC, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 16:10:19 UTC, Steven 
 Schveighoffer wrote:

 module buildfile;


 This isn't what I'm talking about at all.

 I don't want to run D to build, I want to *introspect* D for 
 build info. You can read module names, dependencies (including 
 platform-specific and configuration-specific ones) right out of 
 D code. Author, description, license, etc., have standard ddoc 
 sections in D code.

 Other things dub.json define like flags could easily be a UDA 
 on the module declaration and be pulled out pretty easily by 
 compilers or even relatively basic parsers.

 Even subconfigurations can... kinda be pulled out of D code by 
 looking for version blocks though that's probably another thing 
 I'd put on a UDA.


Yeah, this sounds like magic if it works and has a big "wow" 
effect potential for any new D user.
Feb 28 2023
parent Guillaume Piolat <first.last spam.org> writes:
On Wednesday, 1 March 2023 at 07:45:26 UTC, tastyminerals wrote:

 Yeah, this sounds like magic if it works and has a big "wow" 
 effect potential for any new D user.


For that to work you would still need version information for 
imports.

 dependency("~>2.0") import audioformats;

It also wouldn't work with configurations:


     mixin myTemplateThatDependUponAVersion!(stuff);


if myTemplateThatDependUponAVersion is declared in another 
dependency, that then depends upon a version, then the magic 
parser has no way to know that, or that this program would need a 
configuration, or that a configuration needs two different 
version.

Then you would need:

      configuration(bubbleUp, "myConf") 
version("myVersionIdentifier")
     {
         // blah
     }


Now you need to parse all possible dependencies to know the 
configurations? The build description is still encoded in your 
program but distributed.
Not sure what to think of that.
Mar 01 2023
prev sibling next sibling parent Mathias LANG <geod24 gmail.com> writes:
On Tuesday, 28 February 2023 at 16:32:42 UTC, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 16:10:19 UTC, Steven 
 Schveighoffer wrote:

 module buildfile;


 This isn't what I'm talking about at all.

 I don't want to run D to build, I want to *introspect* D for 
 build info. You can read module names, dependencies (including 
 platform-specific and configuration-specific ones) right out of 
 D code. Author, description, license, etc., have standard ddoc 
 sections in D code.

 Other things dub.json define like flags could easily be a UDA 
 on the module declaration and be pulled out pretty easily by 
 compilers or even relatively basic parsers.

 Even subconfigurations can... kinda be pulled out of D code by 
 looking for version blocks though that's probably another thing 
 I'd put on a UDA.


On paper it sounds quite good, but I am not sure how well it 
works to derive the build instructions from the thing you are 
trying to build. Most of the time, when I build something, it 
fails. It's quite obvious why: Build something new, it will fail 
for a reason or another. Fix it, rebuild, and it either pass, or 
you discover another error. Things do succeed on the first try on 
some occasions, but the number of time they don't is much higher.

How do you ensure your build system is resilient in the presence 
of wrong modules ? How do you ensure that build system errors are 
not replaced by linker / compilation errors ?
Mar 01 2023
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/28/2023 8:32 AM, Adam D Ruppe wrote:

 I don't want to run D to build, I want to *introspect* D for build info. You
can 
 read module names, dependencies (including platform-specific and 
 configuration-specific ones) right out of D code. Author, description,
license, 
 etc., have standard ddoc sections in D code.
 
 Other things dub.json define like flags could easily be a UDA on the module 
 declaration and be pulled out pretty easily by compilers or even relatively 
 basic parsers.


This is a fine idea.
Mar 06 2023
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/28/2023 8:10 AM, Steven Schveighoffer wrote:

     import dparser;


I've been working on making the lexer/parser more amenable to being standalone. 
So far, the lexer is. The parser is close.
Mar 06 2023
prev sibling parent Ki Rill <rill.ki yahoo.com> writes:
On Tuesday, 28 February 2023 at 15:15:33 UTC, Adam D Ruppe wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG 
 wrote:

 Obviously such a change would not happen overnight, and would 
 need broad support from the community. Opinions ?


 Here's a wild idea: use the D language. Most the things 
 dub.json defines are already available in D anyway.


Using D to build D would be great. This reminds me of 
[Ziglang](https://ziglang.org/). We can learn from it. I really 
enjoyed the fact that Zig builds Zig.

Here is a simple 
[template](https://github.com/gamercade-io/zig-template/blob/main/build.zig).

I also like the fact that Zig understands and can seamlessly 
import C.

---
On the other hand, I'd like the SDL format. It's simple. I was 
able to understand it and modify the file on the second read.

I don't think adding a new config format is the way to go. It 
will only add more ambience complexity and confuse users what to 
choose.

There should be one way to do something and it should good and 
exhaustive.
Feb 28 2023
prev sibling next sibling parent reply Bradley Chatha <sealabjaster gmail.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Obviously such a change would not happen overnight, and would 
 need broad support from the community. Opinions ?


IMO JSON just needs to go. It has no purpose being used in dub 
for anything user-facing.

I do like SDL however it's a bit of a "dead end" language, so is 
definitely worth trying to move away from going forward.

TOML makes my eyes bleed and I don't understand at all how anyone 
likes it.

YAML is decent to mentally parse, but writing it can be a bit 
frustrating, especially if you don't have a schema setup in your 
code editor.

Considering dub is also a build system and not just a package 
manager, I wonder if we could take a note out of other build 
managers and use something stronger than basic config language. 
e.g. Meson has their own language; Bazel uses Starlark, CMake has 
abomination. If not instead embracing dub as a package manager, 
and another existing tool as a build system.

My personal pipe dream is that we'd use CUE: https://cuelang.org/ 
- however disregarding the relative complexity of a native D 
implementation, the language itself is still unstable (no v1 
release). I can dream though ;D

Another example are the likes of XMake that use Lua, though I am 
always skeptical about using a full language for build systems. 
(Side node: I've had ideas in the past that you could use 
something like Lua to define jobs/stages that could then be used 
from a declarative/config language to then describe the build.)

Considering the likes of rdmd existing, you could technically 
even just use D itself...

So ultimately is the discussion more "Which language should we 
use" or instead "What exactly do we want dub to do and be capable 
of?"
Feb 28 2023
parent reply Bradley Chatha <sealabjaster gmail.com> writes:
On Tuesday, 28 February 2023 at 15:22:41 UTC, Bradley Chatha 
wrote:

 ...


Adding onto my reply: Adopting HCL could be a potentially strange 
but interesting choice since it has a slightly more extensive 
(though still limited) feature set for a config language, namely: 
interpolation; ability to call but not define functions, and 
support for certain kinds of expressions such as list 
comprehensions (at least I think that's native to the language 
and not a Terraform addon).

Of course there's still the lack of an implementation available 
to D though.
Feb 28 2023
parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 28 February 2023 at 15:34:45 UTC, Bradley Chatha 
wrote:

 On Tuesday, 28 February 2023 at 15:22:41 UTC, Bradley Chatha 
 wrote:

 ...


 Adding onto my reply: Adopting HCL could be a potentially 
 strange but interesting choice since it has a slightly more 
 extensive (though still limited) feature set for a config 
 language, namely: interpolation; ability to call but not define 
 functions, and support for certain kinds of expressions such as 
 list comprehensions (at least I think that's native to the 
 language and not a Terraform addon).

 Of course there's still the lack of an implementation available 
 to D though.


At this point, we may as well choose to use [Nix][0] given that 
its [expression language][1] is surprisingly expressive compared 
to how [simple it is][2] for a Turing-complete language (or its 
likely future successor - [Nickel][3]). Looking solely at its 
expression language, I'd say it's much better thought out 
compared to HCL, while if you look at the combination of Nix + 
Nixpkgs, it's undeniably a better fit for build systems.
The benefit of using Nix is that this would allow seamless 
integration with its huge collection of packages, which 
[dwarfs][4] those of most other system package managers. Of 
primary interest being the libraries which D can interface with 
via `extern (C)` / `extern (C++)` - e.g. for projects containing 
components written in mutiple programming languages. Another 
bonus point is how [active][5] its community is.

[0]: https://nixos.org/
[1]: https://nixos.org/manual/nix/stable/language/index.html
[2]: https://learnxinyminutes.com/docs/nix/
[3]: https://github.com/tweag/nickel/blob/master/RATIONALE.md
[4]: https://repology.org/repositories/graphs
[5]: https://github.com/NixOS/nixpkgs/pulse/weekly
Mar 01 2023
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/28/23 9:29 AM, Mathias LANG wrote:
re are the options that have been floated around:

 1) Use SDL everywhere;
 2) Add support for YAML;
 3) Add support for another format (E.g. TOML);


Json5. It's json but with comments and allows key names without quotes.

https://json5.org/

"JSON5 was started in 2012, and as of 2022, now gets >65M 
downloads/week, ranks in the top 0.1% of the most depended-upon packages 
on npm, and has been adopted by major projects like Chromium, Next.js, 
Babel, Retool, WebStorm, and more. It's also natively supported on Apple 
platforms like MacOS and iOS."


 I would like to go with YAML, not because it is the greatest format on 
 earth, but because SDLang is not well known (it means newcomer have to 
 learn yet another format, and while not a big deal, it adds friction), 
 while YAML is everywhere. In the [thread were SDL was 
 discussed](https://forum.dlang.org/thread/gnfctbuhiemidetngr
i forum.dlang.org), people even thought we invented it!


I have to say, I hate YAML. With a passion. The "backwards compatibility 
with JSON" is pretty must just bolted on. And having multiple ways to 
write configurations might be nice for people writing configs, it's not 
nice for people reading them. Also, significant whitespace = bad.

I would be OK with TOML. But then we have 3 formats. Changing to a 
json-compatible format means we still only have to maintain 2 parsers.

-Steve
Feb 28 2023
parent reply Mathias LANG <geod24 gmail.com> writes:
On Tuesday, 28 February 2023 at 15:35:58 UTC, Steven 
Schveighoffer wrote:

 Json5. It's json but with comments and allows key names without 
 quotes.

 https://json5.org/


I didn't knew it supported quoteless keys, that's pretty nice!
If it's a superset of JSON, we could potentially do it in place 
too.


 I have to say, I hate YAML. With a passion. The "backwards 
 compatibility with JSON" is pretty must just bolted on. And 
 having multiple ways to write configurations might be nice for 
 people writing configs, it's not nice for people reading them. 
 Also, significant whitespace = bad.


I think the significant whitespace boat has sailed. I don't like 
it anymore than you do, but nowadays, editors align everything, 
and if they don't, the linter will complain, or the reviewer 
will. As much as I dislike it, I think significant whitespace 
just hasn't proved to be the barrier it was envisioned to be, as 
demonstrated by Python and YAML.


 I would be OK with TOML. But then we have 3 formats. Changing 
 to a json-compatible format means we still only have to 
 maintain 2 parsers.


Note that going with YAML allows us to deprecate the SDL parser. 
Going with JSON5 includes writing a new parser, and a library to 
do what Configy currently does, on top.
Mar 01 2023
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/1/23 4:21 AM, Mathias LANG wrote:

 On Tuesday, 28 February 2023 at 15:35:58 UTC, Steven Schveighoffer wrote:

 Json5. It's json but with comments and allows key names without quotes.

 https://json5.org/


 
 I didn't knew it supported quoteless keys, that's pretty nice!
 If it's a superset of JSON, we could potentially do it in place too.


It also allows trailing commas.


 

 I have to say, I hate YAML. With a passion. The "backwards 
 compatibility with JSON" is pretty must just bolted on. And having 
 multiple ways to write configurations might be nice for people writing 
 configs, it's not nice for people reading them. Also, significant 
 whitespace = bad.


 
 I think the significant whitespace boat has sailed. I don't like it 
 anymore than you do, but nowadays, editors align everything, and if they 
 don't, the linter will complain, or the reviewer will. As much as I 
 dislike it, I think significant whitespace just hasn't proved to be the 
 barrier it was envisioned to be, as demonstrated by Python and YAML.


It hasn't sailed for dub. Neither json nor sdl require significant white 
space.


 I would be OK with TOML. But then we have 3 formats. Changing to a 
 json-compatible format means we still only have to maintain 2 parsers.


 
 Note that going with YAML allows us to deprecate the SDL parser. Going 
 with JSON5 includes writing a new parser, and a library to do what 
 Configy currently does, on top.


We will never be rid of SDL. Remember that dub has to not only parse the 
current config, but all tagged configs. Updating all old projects, and 
all previous tags of old projects, is not going to happen.

JSON5 requires *updating* the existing JSON parser. JSON is valid JSON5, 
so I think we would be OK just parsing `dub.json` and `dub.json5` 
equivalently. Even if not, the parser can have a flag whether to reject 
JSON5-specific features.

Also, what is configy? And why wouldn't yaml also require changes?

-Steve
Mar 01 2023
next sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Wednesday, 1 March 2023 at 14:21:12 UTC, Steven Schveighoffer 
wrote:

 It hasn't sailed for dub. Neither json nor sdl require 
 significant white space.



I meant that the argument, in general, has been debunked. Also, 
SDL has significant whitespace in how it treats newlines.


 We will never be rid of SDL. Remember that dub has to not only 
 parse the current config, but all tagged configs. Updating all 
 old projects, and all previous tags of old projects, is not 
 going to happen.


That's a problem we can work around, but having code to convert 
from SDL to YAML/JSON. We're already doing this (accidentaly) 
when downloading a package.
Mar 01 2023
next sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
On Wednesday, 1 March 2023 at 21:02:05 UTC, Mathias LANG wrote:

 On Wednesday, 1 March 2023 at 14:21:12 UTC, Steven 
 Schveighoffer wrote:

 It hasn't sailed for dub. Neither json nor sdl require 
 significant white space.



 I meant that the argument, in general, has been debunked. Also, 
 SDL has significant whitespace in how it treats newlines.


 We will never be rid of SDL. Remember that dub has to not only 
 parse the current config, but all tagged configs. Updating all 
 old projects, and all previous tags of old projects, is not 
 going to happen.


 That's a problem we can work around, but having code to convert 
 from SDL to YAML/JSON. We're already doing this (accidentaly) 
 when downloading a package.


Couldn't it convert the file on the fly when it downloads 
dependency?

It's just data at the end of the day
Mar 01 2023
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/1/23 4:02 PM, Mathias LANG wrote:

 On Wednesday, 1 March 2023 at 14:21:12 UTC, Steven Schveighoffer wrote:

 It hasn't sailed for dub. Neither json nor sdl require significant 
 white space.


 
 
 I meant that the argument, in general, has been debunked.


I'm thinking you are using "debunked" in the wrong way here. YAML (or 
anything else) using significant whitespace is not proof that 
significant whitespace is a good policy.


 Also, SDL has 
 significant whitespace in how it treats newlines.


That isn't what I am talking about. Anyone who has used YAML knows this 
isn't the problem that happens when you edit a YAML file.


 We will never be rid of SDL. Remember that dub has to not only parse 
 the current config, but all tagged configs. Updating all old projects, 
 and all previous tags of old projects, is not going to happen.


 
 That's a problem we can work around, but having code to convert from SDL 
 to YAML/JSON. We're already doing this (accidentaly) when downloading a 
 package.


Right, and dub is the one doing this. Dub fetches the package from the 
git hosting server (e.g. github), not code.dlang.org.

We still need an SDL parser in dub.

-Steve
Mar 01 2023
prev sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Wednesday, 1 March 2023 at 14:21:12 UTC, Steven Schveighoffer 
wrote:

 Also, what is configy? And why wouldn't yaml also require 
 changes?


Forgot to answer this. YAML doesn't require any change (well, 
almost), because as of v1.30, dub parses dub.json using a YAML 
parser. As for configy, I provided a link: 
https://github.com/dlang-community/configy
Mar 01 2023
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/1/23 4:04 PM, Mathias LANG wrote:

 On Wednesday, 1 March 2023 at 14:21:12 UTC, Steven Schveighoffer wrote:

 Also, what is configy? And why wouldn't yaml also require changes?


 
 Forgot to answer this. YAML doesn't require any change (well, almost), 
 because as of v1.30, dub parses dub.json using a YAML parser. As for 
 configy, I provided a link: https://github.com/dlang-community/configy


So the benefit of YAML is that you already infected dub with it?

Another benefit of json5 then becomes, we can get rid of the YAML parser!

-Steve
Mar 01 2023
parent 0xEAB <desisma heidel.beer> writes:
On Wednesday, 1 March 2023 at 21:20:15 UTC, Steven Schveighoffer 
wrote:

 On 3/1/23 4:04 PM, Mathias LANG wrote:

 Forgot to answer this. YAML doesn't require any change (well, 
 almost), because as of v1.30, dub parses dub.json using a YAML 
 parser. As for configy, I provided a link: 
 https://github.com/dlang-community/configy


 So the benefit of YAML is that you already infected dub with it?

 Another benefit of json5 then becomes, we can get rid of the 
 YAML parser!


I wouldn’t phrase it this way, but “we already have a YAML parser 
in place” is not exactly a convincing argument to me either.

Like, JSON + SDL support is already in place as well (and even 
live, so zero effort needed!).
Mar 03 2023
prev sibling next sibling parent reply CM <celestialmachinist example.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 [...]


Is obscurity SDL's biggest downside? When I was a D and DUB 
initiate, I had no issues acclimating to its usage, especially 
compared to SBT's DSL. Instead, the source of the friction was 
due to DUB's documentation, and there's already an effort to 
improve this.
Feb 28 2023
parent reply Mathias LANG <geod24 gmail.com> writes:
On Tuesday, 28 February 2023 at 17:27:24 UTC, CM wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG 
 wrote:

 [...]


 Is obscurity SDL's biggest downside? When I was a D and DUB 
 initiate, I had no issues acclimating to its usage, especially 
 compared to SBT's DSL. Instead, the source of the friction was 
 due to DUB's documentation, and there's already an effort to 
 improve this.


The syntax will always be an outlier. Someone building a project 
nowadays need not only a language and a build system, but a 
wealth of tools around it. CI, editor, deployment configs... Many 
of which have adopted YAML, none of which use SDL.
Mar 01 2023
parent CM <celestialmachinist example.com> writes:
On Wednesday, 1 March 2023 at 09:10:36 UTC, Mathias LANG wrote:

 On Tuesday, 28 February 2023 at 17:27:24 UTC, CM wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG 
 wrote:

 [...]


 Is obscurity SDL's biggest downside? When I was a D and DUB 
 initiate, I had no issues acclimating to its usage, especially 
 compared to SBT's DSL. Instead, the source of the friction was 
 due to DUB's documentation, and there's already an effort to 
 improve this.


 The syntax will always be an outlier. Someone building a 
 project nowadays need not only a language and a build system, 
 but a wealth of tools around it. CI, editor, deployment 
 configs... Many of which have adopted YAML, none of which use 
 SDL.


I'm not a professional programmer, and I'm ignorant as to how 
this pertains to SDL's current usage. From man's perspective, 
there's already syntax highlighting, auto-completion, and 
on-demand parsing of a recipe file. To a machine, perhaps SDL 
requires a better parsing library that's more appropriate for D 
and its ecosystem instead of Java, from which it came, but that 
seems easier than an upheaval.
Mar 01 2023
prev sibling next sibling parent reply tastyminerals <tastyminerals gmail.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Currently, Dub uses two formats: JSON and SDL.

 JSON was there from the beginning, I assume due to Dub's npm 
 inspiration.
 In fact, we still very much depend on JSON internally: until 
 recently, JSON object were passed around in quite a few places. 
 When we download a package, regardless of the format the 
 package recipe is in, we store it as `dub.json`.

 [...]


It is safer to just go with whatever is used and accepted as a 
reliable format for configs. I like the idea to use plain D tbo.

JSON lacks comments, but it's fine for minimal default projects. 
We try to use TOML everywhere in a company, although I cannot say 
how well it works as a config for building projects. YAML is the 
most risky, since it has a high chance to alienate a sizeable 
amount of potential users.
Feb 28 2023
next sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Wednesday, 1 March 2023 at 07:41:02 UTC, tastyminerals wrote:

 It is safer to just go with whatever is used and accepted as a 
 reliable format for configs. I like the idea to use plain D tbo.

 JSON lacks comments, but it's fine for minimal default 
 projects. We try to use TOML everywhere in a company, although 
 I cannot say how well it works as a config for building 
 projects. YAML is the most risky, since it has a high chance to 
 alienate a sizeable amount of potential users.


Why would YAML be the most risky choice ? For me it's quite the 
opposite: It's a well known and accepted format, AND the 
implementation is already there. So I clearly don't see how it 
would alienate users, except if those users are also alienated by 
Github Actions, Circle CI, Kubernetes, Ansible, Docker...
Mar 01 2023
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/1/23 4:14 AM, Mathias LANG wrote:

 On Wednesday, 1 March 2023 at 07:41:02 UTC, tastyminerals wrote:

 It is safer to just go with whatever is used and accepted as a 
 reliable format for configs. I like the idea to use plain D tbo.

 JSON lacks comments, but it's fine for minimal default projects. We 
 try to use TOML everywhere in a company, although I cannot say how 
 well it works as a config for building projects. YAML is the most 
 risky, since it has a high chance to alienate a sizeable amount of 
 potential users.


 
 Why would YAML be the most risky choice ? For me it's quite the 
 opposite: It's a well known and accepted format, AND the implementation 
 is already there. So I clearly don't see how it would alienate users, 
 except if those users are also alienated by Github Actions, Circle CI, 
 Kubernetes, Ansible, Docker...


Because YAML is a huge, complex format. It not only contains a lot of 
specification, but has annoying rules (whitespace requirements) and has 
multiple ways of writing configuration -- within the same file.

JSON is much simpler. If YAML is chosen, I will stick with JSON. And so 
for many users, we add lots of complication for no benefit.

Even SDL has multiple ways of specifying things. It gets in the way. 
When there is one way to do something, you do that, and then you're done.

I'm not looking for artistry, cleverness or beauty in build files. I'm 
looking for structure and consistency.

-Steve
Mar 01 2023
prev sibling parent tastyminerals <tastyminerals gmail.com> writes:
On Wednesday, 1 March 2023 at 09:14:17 UTC, Mathias LANG wrote:

 On Wednesday, 1 March 2023 at 07:41:02 UTC, tastyminerals wrote:

 It is safer to just go with whatever is used and accepted as a 
 reliable format for configs. I like the idea to use plain D 
 tbo.

 JSON lacks comments, but it's fine for minimal default 
 projects. We try to use TOML everywhere in a company, although 
 I cannot say how well it works as a config for building 
 projects. YAML is the most risky, since it has a high chance 
 to alienate a sizeable amount of potential users.


 Why would YAML be the most risky choice ? For me it's quite the 
 opposite: It's a well known and accepted format, AND the 
 implementation is already there. So I clearly don't see how it 
 would alienate users, except if those users are also alienated 
 by Github Actions, Circle CI, Kubernetes, Ansible, Docker...


You are right, YAML is used in many places these days and good 
developers should not be having much trouble adjusting.

When I talk about YAML, I am referring to hundreds of frustration 
filled experiences that people eagerly share on hacker news or 
face-to-face when it comes to YAML. Starting from different IDE 
using different formatting and messing people's PRs, ending with 
scary stories about accidentally broken kustomizations causing 
prod outages. Because people make mistakes :/
It takes a few seconds to find a bunch of blog articles with some 
critique towards YAML and why you should not use it. Imagine how 
many beginners are getting inspired by such information and 
forming their own opinions.

Valid or not, there is a lot of hate towards YAML among 
developers. I am sure that adopting it has its benefits, it is 
used successfully in our company as well. What I am not sure of 
is whether the main building D tool needs this notoriety and 
mental "baggage". Once adopted, next thing you will see on this 
forum are questions why YAML for dub?
Mar 01 2023
prev sibling parent 0xEAB <desisma heidel.beer> writes:
On Wednesday, 1 March 2023 at 07:41:02 UTC, tastyminerals wrote:

 YAML is the most risky, since it has a high chance to alienate 
 a sizeable amount of potential users.


In my humble opinion it’s an awful format to write for users that 
only come in contact with it occasionally.

It’s easy enough to understand when reading it, but writing it is 
another story.

I don’t think switching to YAML will make DUB more accessible.
It’s too hard to get right without experience and DUB isn’t worth 
learning YAML when you have no other uses for it.

DUB recipes should be fun to write or they’ll needlessly end up 
low quality, i.e. people not bothering to create configurations, 
sub-packages in cases where they would make sense. If I have to 
lookup YAML specifics first to deploy advanced features, I will 
probably just not use them so I can get away (with not looking 
deeper into YAML than absolutely necessary).

Furthermore, the acceptance of DUB in the community is already 
noticeably less than it should be. Will the introduction of YAML 
improve this? Or will it cause a further hit to the acceptance?
Mar 03 2023
prev sibling next sibling parent reply Arjan <arjan ask.me.to> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Currently, Dub uses two formats: JSON and SDL.


...

 Here are the options that have been floated around:
 1) Use SDL everywhere;
 2) Add support for YAML;
 3) Add support for another format (E.g. TOML);

 I would like to go with YAML, not because it is the greatest 
 format on earth, but because SDLang is not well known (it means 
 newcomer have to learn yet another format, and while not a big 
 deal, it adds friction), while YAML is everywhere. In the 
 [thread were SDL was 
 discussed](https://forum.dlang.org/thread/gnfctbuhiemidetngr
i forum.dlang.org), people even thought we invented it!


what about this:
https://news.ycombinator.com/item?id=34351503
do we risk having a future effort and discussion again?

I don't like SDL probably because I don't know it and have never 
seen it used besides D. With JSON esp. JSON5 I can live, besides 
its massively used and well understood.

I do not see a real benefit in yet another file format for 
configuration.
Mar 01 2023
next sibling parent Mathias LANG <geod24 gmail.com> writes:
On Wednesday, 1 March 2023 at 09:53:08 UTC, Arjan wrote:

 what about this:
 https://news.ycombinator.com/item?id=34351503
 do we risk having a future effort and discussion again?

 I don't like SDL probably because I don't know it and have 
 never seen it used besides D. With JSON esp. JSON5 I can live, 
 besides its massively used and well understood.

 I do not see a real benefit in yet another file format for 
 configuration.


90% of this can be solved by simply quoting strings in value.
Also, once again, the plan is not "Yet another format", but "One 
format".
Mar 01 2023
prev sibling parent Guillaume Piolat <first.last spam.org> writes:
On Wednesday, 1 March 2023 at 09:53:08 UTC, Arjan wrote:

 I do not see a real benefit in yet another file format for 
 configuration.


+1

And going SDL was called a "strategic mistake"

I'd be ok with removing one of SDL or JSON, but beyond that there 
is really nothing to win.
JSON5 idea sounds like the less worse.

My fav scenarii:
1. SDL is removed, JSON comments reinstated
2. JSON is removed
3. JSON5
Mar 01 2023
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Currently, Dub uses two formats: JSON and SDL.

 JSON was there from the beginning, I assume due to Dub's npm 
 inspiration.
 In fact, we still very much depend on JSON internally: until 
 recently, JSON object were passed around in quite a few places. 
 When we download a package, regardless of the format the 
 package recipe is in, we store it as `dub.json`.

 But JSON is a terrible format to write configurations in, given 
 how verbose it is, and it lacking support for comments.
 So around 2014, we added support for SDL (no, not [this 
 one](https://www.libsdl.org/), or [this 
 one](https://web.fe.up.pt/~mricardo/05_06/amsr/artigos/sdlTutorial.pdf), but
[this one](https://sdlang.org/), and it took me 10 minutes to find the link
because Wikipedia doesn't even know about it).

 At some point last year, I did some heavy refactoring on dub 
 along with quite a few improvements. In the process of doing 
 so, I took advantage of the fact that YAML is a subset of JSON 
 to add [a YAML 
 parser](https://github.com/dlang-community/D-YAML/), on which I 
 have built a 
 [library](https://github.com/dlang-community/configy) to do 
 automatic configuration parsing, because let's be honest, it's 
 2023 and we're using the language with the best compile-time 
 reflection capabilities, so we really don't want to manually 
 parse a standard file format to fill a struct.

 This parser is used to parse all JSON files:
 - `dub.json` (but not `dub.sdl`);
 - `dub.selections.json`;
 - `dub.settings.json`;

 Now, I think that having `dub.settings.json` being written in 
 JSON doesn't make sense, and I would like to [change 
 this](https://github.com/dlang/dub/pull/2546). In fact, I have 
 been wanting this for more than 3 years, and the people that 
 [took part in the 
 discussion](https://github.com/dlang/dub/issues/1832) broadly 
 agreed.

 Here are the options that have been floated around:
 1) Use SDL everywhere;
 2) Add support for YAML;
 3) Add support for another format (E.g. TOML);

 I would like to go with YAML, not because it is the greatest 
 format on earth, but because SDLang is not well known (it means 
 newcomer have to learn yet another format, and while not a big 
 deal, it adds friction), while YAML is everywhere. In the 
 [thread were SDL was 
 discussed](https://forum.dlang.org/thread/gnfctbuhiemidetngr
i forum.dlang.org), people even thought we invented it!

 In practice, this would mean adding support for the file format 
 with a YAML extension, giving it priority over the old format, 
 then after a number of releases (most likely 10, probably more 
 for `dub.json`), deprecating the JSON / SDL equivalent.

 Obviously such a change would not happen overnight, and would 
 need broad support from the community. Opinions ?


I have several thoughts on this post and the replies to it.

At the time SDL was introduced, I didn't think it was a good 
idea. I'd still rather it hadn't happened, but the truth is that 
once one gets used to it, it's a lot nicer to write and read than 
JSON. But: it's obscure, and now we have both, and two different 
pages to look at for dub documentation. I don't think it's a 
great place to be.

Having said that, having *3* formats is even worse. I don't think 
we can talk about changing the format without having a plan for 
migrating every single dub.json/dub.sdl out there. I don't know 
how that'd work with issuing PRs to all of them, especially if 
the maintainers ignore them.

To all who doubt that using a turing-complete language is a bad 
idea: it's not. Any complicated project ends up needing it anyway 
and getting around the limitation by generating the declarative 
description or something like that. I've written enough CMake to 
know. I've worked around dub's limitations more times than I can 
count.

Using D sounds great, until one realises that reflecting can take 
a while, and unless there's a generation step followed by an 
actual simple build system, is too annoying to use. I *might* 
know what I'm talking about here: 
https://github.com/atilaneves/reggae/.
It's gotten to the point that I'm thinking of transitioning the 
build descriptions to Python at work to get around 2-3s of 
compile-time. The main reason I haven't done it yet is our builds 
are so complicated and dub takes so long to actually give me the 
information I need that it frequently dwarfs those seconds I'd 
save anyway.

I wish I had a conclusion but I don't.
Mar 01 2023
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/1/23 7:49 AM, Atila Neves wrote:

 Having said that, having *3* formats is even worse.


Having a very similar format to JSON can lessen this impact. e.g. JSON5, 
which looks very similar to JSON and is backwards compatible.


 I don't think we can 
 talk about changing the format without having a plan for migrating every 
 single dub.json/dub.sdl out there. I don't know how that'd work with 
 issuing PRs to all of them, especially if the maintainers ignore them.


This isn't feasible. dub must parse and deal with not only the current 
configuration, but all prior configurations (in case you want to depend 
on prior versions).

We are stuck forever with both JSON and SDL as valid config formats, 
even if they are deprecated/undocumented.

-Steve
Mar 01 2023
next sibling parent reply Guillaume Piolat <first.last spam.org> writes:
On Wednesday, 1 March 2023 at 14:50:16 UTC, Steven Schveighoffer 
wrote:

 We are stuck forever with both JSON and SDL as valid config 
 formats, even if they are deprecated/undocumented.


Unmaintained packages don't have _that_ much value for the 
ecosystem.
Mar 01 2023
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/1/23 10:05 AM, Guillaume Piolat wrote:

 On Wednesday, 1 March 2023 at 14:50:16 UTC, Steven Schveighoffer wrote:

 We are stuck forever with both JSON and SDL as valid config formats, 
 even if they are deprecated/undocumented.


 
 Unmaintained packages don't have _that_ much value for the ecosystem.


These aren't unmaintained projects.

e.g. vibe-d has 207 versions. If you depend on version 0.9.5 (released 
on July 2022), that version used a dub.sdl, now what?

-Steve
Mar 01 2023
parent Guillaume Piolat <first.last spam.org> writes:
On Wednesday, 1 March 2023 at 15:09:20 UTC, Steven Schveighoffer 
wrote:

 On 3/1/23 10:05 AM, Guillaume Piolat wrote:

 On Wednesday, 1 March 2023 at 14:50:16 UTC, Steven 
 Schveighoffer wrote:

 We are stuck forever with both JSON and SDL as valid config 
 formats, even if they are deprecated/undocumented.


 
 Unmaintained packages don't have _that_ much value for the 
 ecosystem.


 These aren't unmaintained projects.

 e.g. vibe-d has 207 versions. If you depend on version 0.9.5 
 (released on July 2022), that version used a dub.sdl, now what?

 -Steve


Dplug users don't use major version 1, they use major version 11+
Mar 01 2023
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/1/2023 6:50 AM, Steven Schveighoffer wrote:

 We are stuck forever with both JSON and SDL as valid config formats, even if 
 they are deprecated/undocumented.


Since the code to support them is already written, would we need to do more
than 
just not delete it?
Mar 04 2023
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 05/03/2023 9:43 AM, Walter Bright wrote:

 Since the code to support them is already written, would we need to do 
 more than just not delete it?


The entire configuration system was replaced with configy which in turns 
uses some of the same libraries, but swaps others out like dyaml instead 
of std.json.

This has increased build times by a noticeable amount. I did push back 
originally to keep the old system in place, but ultimately it is gone 
now. All in the hope of improving error messages, successfully I might add.
Mar 04 2023
parent reply Mathias LANG <geod24 gmail.com> writes:
On Saturday, 4 March 2023 at 21:01:45 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 This has increased build times by a noticeable amount.


Are you talking about the time it takes to build dub, or the time 
dub takes to build a project ?
Mar 04 2023
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 05/03/2023 11:12 AM, Mathias LANG wrote:

 On Saturday, 4 March 2023 at 21:01:45 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 This has increased build times by a noticeable amount.


 
 Are you talking about the time it takes to build dub, or the time dub 
 takes to build a project ?


The time it takes to build dub.

The other is also a problem of course, but with the former being slow it 
doesn't make the latter easier to fix.
Mar 04 2023
prev sibling next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
I found an argument against YAML

https://www.baeldung.com/yaml-json-differeneces

30x slower than json
Mar 01 2023
next sibling parent max haughton <maxhaton gmail.com> writes:
On Wednesday, 1 March 2023 at 15:43:26 UTC, ryuukk_ wrote:

 I found an argument against YAML

 https://www.baeldung.com/yaml-json-differeneces

 30x slower than json


30x slower than almost zero is still almost zero (a long *long* 
dub.json is going to be 1k lines at most)
Mar 01 2023
prev sibling parent Monkyyy <crazymonkyyy gmail.com> writes:
On Wednesday, 1 March 2023 at 15:43:26 UTC, ryuukk_ wrote:

 I found an argument against YAML

 https://www.baeldung.com/yaml-json-differeneces

 30x slower than json


"Json can be better cause you can delete endlines"

I feel like this person is confusing configuration and binary data
Mar 01 2023
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 while YAML is everywhere


Which is nothing to be proud of, it's the worst markup language.
Mar 01 2023
prev sibling next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
I've been working on trying to get dub to build faster.

Dyaml appears to be a big ol' pile of slow code to build.

So far I've shaved off 2s from its build, down to 22s with dmd.

Either way, we have a problem... and it'll affect this particular decision.
Mar 01 2023
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 1 March 2023 at 18:13:35 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 I've been working on trying to get dub to build faster.


The simplest way is to get it to build per D package, in threads, 
and track dependencies. Which reggae does already, but it'd be 
nice if everyone else could benefit.

dub also takes quite a while to do mostly nothing, which reggae 
sidesteps by only asking dub for info when it generates the build 
instead of every time.


 Dyaml appears to be a big ol' pile of slow code to build.

 So far I've shaved off 2s from its build, down to 22s with dmd.


This would be considered fast at work. Much to my chagrin.
Mar 02 2023
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 03/03/2023 12:40 AM, Atila Neves wrote:

 On Wednesday, 1 March 2023 at 18:13:35 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 I've been working on trying to get dub to build faster.


 
 The simplest way is to get it to build per D package, in threads, and 
 track dependencies. Which reggae does already, but it'd be nice if 
 everyone else could benefit.
 
 dub also takes quite a while to do mostly nothing, which reggae 
 sidesteps by only asking dub for info when it generates the build 
 instead of every time.


Its significantly faster once you turn off networking.

There needs to be a local cache of packages rather than asking the 
dub-registry for everything all the time.


 Dyaml appears to be a big ol' pile of slow code to build.

 So far I've shaved off 2s from its build, down to 22s with dmd.


 
 This would be considered fast at work. Much to my chagrin.


Oh no.
Mar 02 2023
next sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Thursday, 2 March 2023 at 13:56:40 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 Its significantly faster once you turn off networking.

 There needs to be a local cache of packages rather than asking 
 the dub-registry for everything all the time.


There is already. If you have evidence of the contrary, please 
raise an issue (or better, a PR).
Mar 02 2023
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 03/03/2023 5:24 AM, Mathias LANG wrote:

 On Thursday, 2 March 2023 at 13:56:40 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 Its significantly faster once you turn off networking.

 There needs to be a local cache of packages rather than asking the 
 dub-registry for everything all the time.


 
 There is already. If you have evidence of the contrary, please raise an 
 issue (or better, a PR).


No, we do not currently store the entire dub-registry's package database 
locally. Which is what I'm talking about and yes it has been discussed 
on the issue tracker.
Mar 02 2023
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 2 March 2023 at 13:56:40 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 On 03/03/2023 12:40 AM, Atila Neves wrote:

 On Wednesday, 1 March 2023 at 18:13:35 UTC, Richard (Rikki) 
 Andrew Cattermole wrote:

 I've been working on trying to get dub to build faster.


 
 The simplest way is to get it to build per D package, in 
 threads, and track dependencies. Which reggae does already, 
 but it'd be nice if everyone else could benefit.
 
 dub also takes quite a while to do mostly nothing, which 
 reggae sidesteps by only asking dub for info when it generates 
 the build instead of every time.


 Its significantly faster once you turn off networking.


I'm aware, I've had `db` aliased to `clear && dub build -q 
--nodeps --skip-registry=all` for many years now. It still 
does... something.


 There needs to be a local cache of packages rather than asking 
 the dub-registry for everything all the time.


There is, it just doesn't look there. There's a reason why and 
Matthias is working on it.
Mar 06 2023
prev sibling next sibling parent reply 0xEAB <desisma heidel.beer> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 1) Use SDL everywhere;


I’d prefer to ditch SDL.

It’s a lost bet from today’s point of view; nobody is using it 
elsewhere essentially.

Compared to other formats that are widely used (JSON, YAML, …), 
SDL can be perceived as “the bizarre format that one will only 
learn/use to write DUB recipes”.

I’ve been using DUB for years, and still can’t remember SDL’s 
syntax.
When I decide to use it, it’s only because it allows me to put 
"subPackage" + "dependency" next to each other, unlike the less 
flexible dub.json format. I don’t think this advantage is 
actually worth it, given this approach could get quite messy 
(esp. for other uses).

Also, I’m curious:
Does anyone here in the community use it for anything else but 
DUB recipes?
Mar 03 2023
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 3 March 2023 at 17:14:45 UTC, 0xEAB wrote:

 [..]
 I’ve been using DUB for years, and still can’t remember SDL’s 
 syntax.


The same applies to Dub's json format. I doubt many people can 
write a complicated dub.json file (e.g. with various 
configurations, platform & compiler specific D or LD flags, 
optional dependencies, git depedencies, and so on) without 
looking up the docs, or more likely browsing other D projects for 
inspiration.


 [..]

 Also, I’m curious:
 Does anyone here in the community use it for anything else but 
 DUB recipes?


Back in 2014 I was writing a toy ray tracer for a university 
project. Initially, I started using JSON for describing scenes, 
since it seemed like the obvious choice (and `std.json` among 
another things). Not long after I started the project I quickly 
got annoyed with JSON's lack of expressiveness(especially coming 
using [XAML][1] at work), and its lack of comments. Not long 
after I had started the project I learned that Dub was adding SDL 
format and I read [DEP1][2] which convinced me to try and 
experiment with SDL. Then I gave it a go and extended my scene 
loader so it could load SDL (in addition to JSON) files. In the 
end, I liked the result, so I stuck with it. As I said, that was 
a toy project, so I don't have anything particularly interesting 
to show, but still, you may be curious to see an example of 
another D project using SDL as a human-readable data format:
https://github.com/PetarKirov/Chess2RT/blob/master/data/lecture5.sdl

[1]: 
https://learn.microsoft.com/en-us/dotnet/desktop/wpf/xaml/?view=netdesktop-7.0
[2]: https://github.com/dlang/dub/wiki/DEP1
Mar 03 2023
parent reply 0xEAB <desisma heidel.beer> writes:
On Friday, 3 March 2023 at 18:44:54 UTC, Petar Kirov [ZombineDev] 
wrote:

 On Friday, 3 March 2023 at 17:14:45 UTC, 0xEAB wrote:

 [..]
 I’ve been using DUB for years, and still can’t remember SDL’s 
 syntax.


 The same applies to Dub's json format.


I’d say there a difference between syntax and mapping data to a 
format.
If you aren’t familiar with DUB’s JSON format, you can’t really 
claim that you were familiar with DUB’s SDL format either.

If I’m already familiar with JSON in general, DUB’s manual will 
be all I have to worry about.
If I’m not familiar with neither SDL nor DUB (which is more 
likely than not being familiar with JSON), I’ll end up looking at 
two manuals.

Your IDE is probably also better at auto completing DUB recipe 
property names than general file syntax.
Mar 03 2023
next sibling parent reply Bradley Chatha <sealabjaster gmail.com> writes:
On Friday, 3 March 2023 at 22:12:46 UTC, 0xEAB wrote:

 Your IDE is probably also better at auto completing DUB recipe 
 property names than general file syntax.


It'd be better if we had a JSON schema file for dub.json as well. 
If we were to get it into https://www.schemastore.org/json/ then 
VSCode at the very least would be able to make use of it 
automatically. Slightly less painful than having to tab between 
your editor and the dub.json documentation.
Mar 03 2023
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 04/03/2023 11:56 AM, Bradley Chatha wrote:

 On Friday, 3 March 2023 at 22:12:46 UTC, 0xEAB wrote:

 Your IDE is probably also better at auto completing DUB recipe 
 property names than general file syntax.


 
 It'd be better if we had a JSON schema file for dub.json as well. If we 
 were to get it into https://www.schemastore.org/json/ then VSCode at the 
 very least would be able to make use of it automatically. Slightly less 
 painful than having to tab between your editor and the dub.json 
 documentation.


We do have one.

I don't think it is maintained on dub's end, but one does exist.

https://github.com/intellij-dlanguage/intellij-dlanguage/blob/develop/dub/src/main/resources/jsonSchema/dub.json-schema.json
Mar 03 2023
prev sibling parent reply 0xEAB <desisma heidel.beer> writes:
On Friday, 3 March 2023 at 22:56:57 UTC, Bradley Chatha wrote:

 It'd be better if we had a JSON schema file for dub.json as 
 well.


You’re welcome:
https://github.com/Pure-D/code-d/blob/master/json-validation/dub.schema.json

Part of the code-d extension btw.
Mar 03 2023
parent reply Guillaume Piolat <first.last spam.org> writes:
On Friday, 3 March 2023 at 23:27:01 UTC, 0xEAB wrote:

 On Friday, 3 March 2023 at 22:56:57 UTC, Bradley Chatha wrote:

 It'd be better if we had a JSON schema file for dub.json as 
 well.


 You’re welcome:
 https://github.com/Pure-D/code-d/blob/master/json-validation/dub.schema.json

 Part of the code-d extension btw.


And newest dub auto-checks dub.json so you don't have to use it 
anymore.
Mar 04 2023
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/4/23 6:25 AM, Guillaume Piolat wrote:

 On Friday, 3 March 2023 at 23:27:01 UTC, 0xEAB wrote:

 On Friday, 3 March 2023 at 22:56:57 UTC, Bradley Chatha wrote:

 It'd be better if we had a JSON schema file for dub.json as well.


 You’re welcome:
 https://github.com/Pure-D/code-d/blob/master/json-validation/dub.schema.json

 Part of the code-d extension btw.


 
 And newest dub auto-checks dub.json so you don't have to use it anymore.


The schema helps with autocomplete though.

-Steve
Mar 04 2023
prev sibling parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 03.03.2023 um 23:12 schrieb 0xEAB:

 On Friday, 3 March 2023 at 18:44:54 UTC, Petar Kirov [ZombineDev] wrote:
 ...
 
 If I’m already familiar with JSON in general, DUB’s manual will be all I 
 have to worry about.
 If I’m not familiar with neither SDL nor DUB (which is more likely than 
 not being familiar with JSON), I’ll end up looking at two manuals.


Literally the only thing that I think is worth looking up in the SDLang 
manual is line continuations. Everything else that is needed is already 
there in the DUB documentation by simply listing the directive itself. 
The value and comment syntax just matches D.

See https://sdlang.org - the examples there contain everything there is 
to know, and I think only the basic tag and nesting syntax is what 
really distinguishes it from a simple D subset (date/time/binary values, 
too, but they are not used by DUB).
Mar 07 2023
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/3/23 12:14 PM, 0xEAB wrote:


 Also, I’m curious:
 Does anyone here in the community use it for anything else but DUB recipes?


I use it to configure my vibe-d project.

However, the existing SDL parser (sdlang-d) serializes to a DOM, which 
sucks for configuration. I want it to serialize to a struct I specify.

I'm presently working on adding json5 to iopipejson, and then I'm going 
to switch to that.

-Steve
Mar 03 2023
prev sibling parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 03.03.2023 um 18:14 schrieb 0xEAB:

 Also, I’m curious:
 Does anyone here in the community use it for anything else but DUB recipes?


I'm using it for a few little projects as configuration and document 
files, nothing big, but a few of them are in daily use by clients. I've 
written a little serialization adapter for those cases (vibe-sdlang).

One reason why I chose to adopt it was that it incidentally almost 
exactly matches a bunch of custom data formats that I defined for a game 
engine, just in a generic manner.

Those formats in turn were inspired by the approach that the Quake/Doom 
engines used, which was to define very concise and readable text based 
data formats based on a common tokenizer. Working with them was a true 
revelation compared to the typical XML (or JSON) based alternatives.
Mar 06 2023
prev sibling next sibling parent reply 0xEAB <desisma heidel.beer> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 JSON was there from the beginning, I assume due to Dub's npm 
 inspiration.


Composer uses it, too.
(It’s not like NPM and DUB were the only ones to use it for 
package recipes.)

It’s a simple and commonly understood format, widely used (mostly 
for other use cases, of course) and feels similar to program code.

I assume most users are probably already familiar with it. 
Chances that someone has to “get into” this format just to use 
DUB are pretty low.

What do other languages use for their “package” descriptions:

- Windows: Chocolatey: XML (+ JSON)
- Go.mod: custom(?)
- Java: Maven: XML
- JavaScript: Bower: JSON
- JavaScript: NPM: JSON
- .NET: NuGet: XML
- Nim: nimble: custom(?)
- PHP: Composer: JSON
- Rust: crates: TOML
- Zig: zigmod: YAML
Mar 03 2023
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Friday, 3 March 2023 at 17:42:03 UTC, 0xEAB wrote:

 On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG 
 wrote:

 JSON was there from the beginning, I assume due to Dub's npm 
 inspiration.


 Composer uses it, too.
 (It’s not like NPM and DUB were the only ones to use it for 
 package recipes.)

 It’s a simple and commonly understood format, widely used 
 (mostly for other use cases, of course) and feels similar to 
 program code.

 I assume most users are probably already familiar with it. 
 Chances that someone has to “get into” this format just to use 
 DUB are pretty low.

 What do other languages use for their “package” descriptions:

 - Windows: Chocolatey: XML (+ JSON)
 - Go.mod: custom(?)
 - Java: Maven: XML
 - JavaScript: Bower: JSON
 - JavaScript: NPM: JSON
 - .NET: NuGet: XML
 - Nim: nimble: custom(?)
 - PHP: Composer: JSON
 - Rust: crates: TOML
 - Zig: zigmod: YAML


Zig doesn't use zigmod, it's a one of the community project, they 
have their own thing now, zig build, you use zig, and 'zon', wich 
is basically a zig file that returns a struct, to describe your 
packages/dependencies

Java doesn't use Maven, it's a dead format, most popular one is 
Gradle and it's Groovy or  maybe Kotlin, i don't remember

I think trying to mimic what other language DID in the past isn't 
really helpful

What problem trying to solve? i think adam has the best idea, no 
need anything, stick to D, that's something worth experimenting 
with

And i wouldn't be surprised if the web people move towards that 
in the future aswel

https://deno.com/blog/you-dont-need-a-build-step
Mar 03 2023
parent 0xEAB <desisma heidel.beer> writes:
On Friday, 3 March 2023 at 18:04:06 UTC, ryuukk_ wrote:

 Zig doesn't use zigmod, it's a one of the community project, 
 they have their own thing now, zig build, you use zig, and 
 'zon', wich is basically a zig file that returns a struct, to 
 describe your packages/dependencies

 Java doesn't use Maven, it's a dead format, most popular one is 
 Gradle and it's Groovy or  maybe Kotlin, i don't remember


“…and Windows isn’t a programming language!!”
Anyway, that wasn’t the idea here. I was just quickly looking up 
stuff and pasted the first  few things came up. Whatever…

Thanks for sharing your findings about Zig.
Mar 03 2023
prev sibling next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Opinions ?


No fan of yaml, eventhough it is used a lot.

What about Amazon's ion?
Mar 03 2023
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/28/2023 6:29 AM, Mathias LANG wrote:

 Opinions ?


I don't know much technically about the various formats, but on a non-technical 
basis:

1. We win if we go with a well-known, popular format. We are not in the
business 
of promoting an obscure format or inventing our own. We picked github and git 
long ago simply because it was the most popular, and that decision paid off 
handsomely for us. As you wrote, it minimizes user friction.

2. We need a quality import/export mechanism for it written in D with an easy 
license (preferably Boost).

I'm on board with whatever you choose.
Mar 04 2023
prev sibling next sibling parent Chris Piker <chris hoopjump.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Currently, Dub uses two formats: JSON and SDL.


...

 But JSON is a terrible format to write configurations in, given 
 how verbose it is, and it lacking support for comments.


...

 Obviously such a change would not happen overnight, and would 
 need broad support from the community. Opinions ?


Anything than supports comments would we welcomed by me.  
Adopting Json5 seems like the least painful.

Also, putting the build information into the source is what I do 
most of the time anyway, then calling `dub build --single` from a 
makefile.
Mar 04 2023
prev sibling next sibling parent reply Ahmet Sait <nightmarex1337 hotmail.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 ...


I mostly agree with Steve and others in this thread that we 
should go with [JSON5](https://json5.org). We can make the JSON 
parser accept JSON5 with a fully backward compatible manner (just 
like you did with YAML) and hopefully deprecate SDL so there is 
one canonical format everyone use (even though we silently keep 
supporting SDL).

What I would like to mention is that DUB has this `add` command 
for automatically adding a dependency (e.g. `dub add vibe-d`), 
and it is extremely broken to the core: It completely overwrites 
your file discarding all the formatting and even comments in a 
`dub.sdl` while also reordering json fields in `dub.json` 
alphabetically. It should have never existed in it's current form 
and should be removed.
Mar 06 2023
parent Guillaume Piolat <first.last spam.org> writes:
On Monday, 6 March 2023 at 08:22:05 UTC, Ahmet Sait wrote:

 It should have never existed in it's current form and should be 
 removed.


Yes please, I would never recommend anyone to use "dub add" 
rather than just add a line.
Mar 06 2023
prev sibling next sibling parent Witold <witold.baryluk+dlang gmail.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 Obviously such a change would not happen overnight, and would 
 need broad support from the community. Opinions ?


Pure JSON. And only it.

Reason? It is rather simple, and super compatible. Easy to parse 
from anywhere, JavaScript, Python, Ruby, PHP, C++, C, syntax 
highliters on website, web frameworks, editors, formatters, 
command line tools (jq), etc. Do not assume dub files will only 
be consumed by dub.

YAML is horrible in my opinion. I use it a lot, and from a 
distance it looks nicer than JSON (comments, less verbosity, less 
quoting, etc), but it is not good in the long run. 1) YAML Parser 
are super complex. 2) backreferences are complicated. 3) multiple 
ways of doing same thing (strings, arrays, dicts), so you cannot 
easily read and write it back programatically, without likely 
messing diffs, 4) too many damn ways to write strings, 5) no 
quoting on values, causes issues when that value accidentally is 
integer/float-like or boolean-like (including word `no`). I hate 
this, and I just quote absolutely everything because of this, 
which defeats a big part of yaml. 6) multi-document feature is an 
anti-feature. 7) slow. 8) slow. 9) There is a lot of 
implementations, and in reality they all differ a bit in minor 
details. 10) slow. 11) Some extra features are just broken (like 
date parsing).

I use YAML a lot, in Ansible, Prometheus, Kubernetes, Github, 
Docker, and few other frequently used project. I never use it for 
personally build projects, because I do not like its complexity.

Only thing that would be nice in JSON to have: comments, trailing 
commas. JSON5 you say. I say no. Why? Compatibility. One can live 
without commas. Comments can be emulated using object keys, i.e. 
starting with underscore, and ignore them during processing. 
Multi-document can be easily done by just having JSON after JSON 
in one file (most parsers will just parse one at the time, and 
allow you to parse next object). Not that dub needs this feature 
anyway.

I do not like JSON either, but I do not like YAML, JSON5 and TOML 
even more. SDL is too XML-like (with attributes), but do not map 
nicely to processing in most programming languages (i.e. it is 
not just a dict / aa), and often require akward XML-like / 
DOM-like parsing, which is also more complex than it needs to be.

I would say YAML is okish, if the files are not too big, and you 
edit them literally every day. But the ones in dub, you edit only 
few very few times. So its human friendliness isn't really a good 
selling point to me.

But YAML spec is so big, has (or had) so many bugs, and issues, 
that I consider it horrible language.

Every few weeks I have some YAML issues, be it in Python, Go, 
Ansible. We even had few production outages caused by YAML 
idiotic parsing rules.

Simple, fast, and universal, is better than complex, slow and 
niche.


 But JSON is a terrible format to write configurations in, given 
 how verbose it is, and it lacking support for comments.


I do not agree with this statement.

You want comments. Just add them as underscore-prefixed keys. Or 
use `//`, which is rather easy to strip before passing to other 
tools.

If you want to comment some part of the config temporarily, then 
just remove it. Most people use version control. It will be in 
their history.

I would not consider JSON really a configuration language. It is 
more of a storage and data transfer format. Configuration 
languages are different things, there few decent ones out there, 
like jsonnet, Hashicorp's HCL, Dhall, and few more. The 
interoperability issue of them is not an issue, as: 1) there are 
actually few implementations, 2) they are not used directly by 
any system, rather they are passed through processing, and simple 
(flat and dump) format is used as output (usually JSON) to be 
consumed by programs. I like proper configuration languages like 
jsonnet, because otherwise you end up in some horrible templating 
like jinja in Ansible, or craziness of Helm Charts, K8s 
Kustomize, which are all just horrible hacks with poor usability. 
But, for dub using proper configuration language, would be an 
overkill in my opinion.

Personally I would use text-encoded protocol buffers. Schema 
based validation and typing out of the box. I use protocol 
buffers and text-encoded ones for configs in most of my personnel 
projects (Go, D, C++, Python), but it does come with some other 
tradeoffs (proto buffer definition files, extra compilation step, 
which is easy to automate, but a barrier for some).

But adding anything new is just not a good idea long term. You 
will need to support all the formats for years.

I think way more important than format of config is better 
documentation and tooling. Nodejs and TypeScript / JavaScript 
people use huge JSON files as config to build system, and it all 
just works fine.


Cheers.
Mar 07 2023
prev sibling parent Dmytro Katyukha <firemage.dima gmail.com> writes:
On Tuesday, 28 February 2023 at 14:29:28 UTC, Mathias LANG wrote:

 ...


Hi!


I like the idea of YAML - it is readable! it is everywhere, we 
already have libraries to parse it, it has *aliases and anchors* 
that allows to simplify complex configs.
The slowness of parsing yaml is not an argument, from my point of 
view, because it will not be parsed too frequently.
But, may be introducing new config format, especially if there a 
lot of people that dislike it for some reason, is not a good idea.

In case of config files, i prefer readability. And in this case, 
SDLang fits nice - it is readable, and it is easy to work with 
it. So i think it is good to keep it.

About JSON (even JSON5) - i think it is not good for 
configuration. It is good for serialization, but not for manually 
adited configuration files: editing json manualy - is a pain. 
Using some hacks to add comments and enable trailing comma is not 
good idea.
I think comments are required feature for configs (there are 
cases when it is needed to explain why this options is set in 
this way and why it must not be changed or removed, or to 
describe options in config templates).
If we silently enable JSON5 syntax support for our json files, it 
also could lead issues with parsing json files by other tools 
(not dub itself). Using json5 as third format? May be we could 
find some better config format for third variant of 
configuration? Because deprecation of old formats could take 
years (there are some packages that had no new releases for few 
years, but seems to be working and useful, and i think we must 
not break them for no good reason).

Also, i think that it is not good idea to mix package 
configuration (description), that have to be read by other tools, 
with build scripts (d, lua, etc), that impretatively describes 
how to build the app/lib. Thus dub config must not be defined in 
any imperative language, instead it must be kept declarative.

So, in my opinion, we have to use YAML or some other 
readable/editable config format, or just keep SDLang with JSON 
and make SDLang default format for dub config.
Mar 10 2023