www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Phobos 3 Discussion Notes - 02-01-2024

reply Adam Wilson <flyboynw gmail.com> writes:
Walter and I had a productive conversation yesterday about Phobos 
3 and we felt it would be appropriate to share some notes on our 
discussion.

The conversation started out with me fumbling around trying to 
understand how Editions are going to effect Phobos. The 
discussion settled around two topics.

The first was how to construct the package structure for Phobos 
3. Walter has proposed using `std2.` as the root namespace. His 
primary reason was that he wanted to avoid the potential for 
importing anything from `std.` that might trigger auto-decoding. 
I pointed out two major flaws with this design.

First, it would be very easy to accidentally type `std.` in an 
`std2.` module. We realized that this would be a problem no 
matter what root name was used and that the appropriate answer 
would be to specify a DStyle rule specific to Phobos that only 
modules in `std.` can import `std.` modules.

Second, this design implies that the '2' in `std2.` is a version 
specifier that would be incremented with each Phobos release. 
This was not the intention and we agreed that it would be 
confusing. I proposed using `sys.` as the root name for Phobos 3 
and Walter found that acceptable. We briefly discussed splitting 
up Phobos into multiple roots and no firm agreement was reached.

The other major topic of discussion was what I've been calling 
the "Crippled by Default" design of editions, where the oldest 
edition (technically the last pre-edition release) is the default 
edition if no edition is specified. This poses a few challenges 
from an end-user standpoint, but the argument that ended up 
resonating was the idea that in engineering we always want to 
make the "right" way the default or easiest way to do something, 
and then provide escape hatches where necessary.

Therefore, the compiler should default to use the latest version 
and then provide the ability via a switch to set the edition, or 
lack thereof, of the modules an import path. This solves the 
problem of abandon-ware packages being accessible without 
presenting the new user with an ever more decayed version of the 
compiler. We want to put out best foot forward and presenting the 
last pre-editions release, which is constantly getting old as 
time passes, does not do that. No specific agreement was reached 
but Walter agreed to consider it.

When then moved on to a conversation about how Walter envisions 
editions actually working. Since none of have seen the document 
that Atila is working on, Walter shared his opinions on how it 
should work. Essentially, Walter would like to see a "hybrid" 
approach having edition attributes for specific experimental 
features, and then having a yearly "roll-up" edition that 
includes all the promoted features from the prior year. So if 
DIP1000 gets promoted to Edition 2025, then DIP1000 would be 
active by default in that edition and all subsequent editions 
without having to specifically enable it.

I did bring up that this was likely to cause another "function 
attribute soup" problem, but in general I wholeheartedly agree 

C++ both do something similar so it would be conceptually 
comfortable to users coming from those languages. Atila, if 
you're reading this, this is what Walter was thinking/hoping 
would appear, but since we don't have your document we were 
mostly left to speculate. We await your document drop with 
anticipation!

After that we had a discussion about how to distribute Phobos. 
This mostly centered on what release cadence to use. I argued for 
linking the Phobos version to the edition release schedule. I 
think this is sensible and makes it easier for people to reason 
about which compiler/library pairing they are using. Walter was 
fine with that, but he does not want to use the "Edition" 
language to describe Phobos releases. I think this makes sense as 
Phobos doesn't really have editions, but following SemVer, it 
would have yearly major releases. This would mean that new 
features are released once per year, and bugfixes would be 
released on whatever the fast-cadence schedule is for DMD in 
between Yearly Edition roll-ups.

Finally, we touched briefly on the major changes we would like to 
see in Phobos 3 and these are the major changes we are committing 
to for Phobos 3 so far:
- Removal of Autodecoding.
- Promoting allocators out of experimental.
- Range interface redesign (see JMD's thread 
[here](https://forum.dlang.org/thread/mailman.588.1705813271.3719.digitalmars-d puremagic.com)).
- Fix std.traits.

The above list is not exhaustive and we are open to further 
suggestions. For myself, I would love to see Cryptography and 
Stream API's make it into Phobos, but I am sure that the list of 
what the community wants to add is legion so those could end up 
being lower priority.

One thing I would like to arrange if possible is planning 
sessions with the people involved with that specific change 
streamed to YouTube so that the community can participate via the 
Discord. If a person wishes to join the stream for a face-to-face 
discussion we would make the link available via DM. I found this 
discussion with Walter to be a much more efficient use of time 
and far more agreeable than writing long posts on the NG (the 
irony given the length of this post... I know).

If you would like to participate in the design discussion on 
Phobos 3, I am currently hosting a repo on my GitHub with GH 
Discussions for unresolved/undesigned topics, and PR's for 
editing the actual design document. Link is here: 
https://github.com/LightBender/PhobosV3-Design
Feb 02
next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about 
 Phobos 3 and we felt it would be appropriate to share some 
 notes on our discussion.

 [...]


What about the missing standard library pieces  that are core to 
everyday software dev? Stuff you get out of the box with 
java/python. Is there such considerations beyond fixing the 
mistakes of v2?

* Json
* XML
* Streams
* Networking (http, socket, ...)
....



If I understand correctly, v3 is about . How about new module 
additions to v3 such as the above?
Feb 02
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 03/02/2024 3:46 AM, aberba wrote:

 On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about Phobos 3 
 and we felt it would be appropriate to share some notes on our 
 discussion.

 [...]


 
 What about the missing standard library pieces  that are core to 
 everyday software dev? Stuff you get out of the box with java/python. Is 
 there such considerations beyond fixing the mistakes of v2?
 
 * Json
 * XML
 * Streams
 * Networking (http, socket, ...)
 ....
 
 
 
 If I understand correctly, v3 is about . How about new module additions 
 to v3 such as the above?


That is not part of the current work, although Adam does have a wish 
list and thoughts about it.

Please note that Phobos is getting close enough to the 64k symbol limit 
on Windows that we cannot be expanding its scope. So the consideration 
of how to architect a new standard library must occur prior to work on it.

Starting with something that isn't just a wish list is right. It isn't 
some library that we can chop and change on, its going to live for 20+ 
years so its going to have to be designed right.
Feb 02
parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 2 February 2024 at 15:08:37 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 On 03/02/2024 3:46 AM, aberba wrote:

 [...]


 That is not part of the current work, although Adam does have a 
 wish list and thoughts about it.

 Please note that Phobos is getting close enough to the 64k 
 symbol limit on Windows that we cannot be expanding its scope. 
 So the consideration of how to architect a new standard library 
 must occur prior to work on it.

 Starting with something that isn't just a wish list is right. 
 It isn't some library that we can chop and change on, its going 
 to live for 20+ years so its going to have to be designed right.


I understand what you're saying however it doesn't necessarily 
mean "adding" to existing code. I think a lot of the old legacy 
stuff should be removed from new std.
Feb 02
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 03/02/2024 6:48 AM, aberba wrote:

 On Friday, 2 February 2024 at 15:08:37 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 On 03/02/2024 3:46 AM, aberba wrote:

 [...]


 That is not part of the current work, although Adam does have a wish 
 list and thoughts about it.

 Please note that Phobos is getting close enough to the 64k symbol 
 limit on Windows that we cannot be expanding its scope. So the 
 consideration of how to architect a new standard library must occur 
 prior to work on it.

 Starting with something that isn't just a wish list is right. It isn't 
 some library that we can chop and change on, its going to live for 20+ 
 years so its going to have to be designed right.


 
 I understand what you're saying however it doesn't necessarily mean 
 "adding" to existing code. I think a lot of the old legacy stuff should 
 be removed from new std.


Oh it will need reviewing line by line prior to copying.

So yeah that should be our default.
Feb 02
prev sibling parent Adam Wilson <flyboynw gmail.com> writes:
On Friday, 2 February 2024 at 17:48:30 UTC, aberba wrote:

 On Friday, 2 February 2024 at 15:08:37 UTC, Richard (Rikki) 
 Andrew Cattermole wrote:

 On 03/02/2024 3:46 AM, aberba wrote:

 [...]


 That is not part of the current work, although Adam does have 
 a wish list and thoughts about it.

 Please note that Phobos is getting close enough to the 64k 
 symbol limit on Windows that we cannot be expanding its scope. 
 So the consideration of how to architect a new standard 
 library must occur prior to work on it.

 Starting with something that isn't just a wish list is right. 
 It isn't some library that we can chop and change on, its 
 going to live for 20+ years so its going to have to be 
 designed right.


 I understand what you're saying however it doesn't necessarily 
 mean "adding" to existing code. I think a lot of the old legacy 
 stuff should be removed from new std.


If you have a list of stuff that you think should be removed from 
Phobos, I am all ears. If  you could start a discussion over in 
the repo that would be most helpful.
https://github.com/LightBender/PhobosV3-Design/discussions
Feb 02
prev sibling next sibling parent Adam Wilson <flyboynw gmail.com> writes:
On Friday, 2 February 2024 at 14:46:45 UTC, aberba wrote:

 What about the missing standard library pieces  that are core 
 to everyday software dev? Stuff you get out of the box with 
 java/python. Is there such considerations beyond fixing the 
 mistakes of v2?

 * Json
 * XML
 * Streams
 * Networking (http, socket, ...)
 ....



 If I understand correctly, v3 is about . How about new module 
 additions to v3 such as the above?


The above list are all things that I would love to see make it in 
to Phobos 3. The reason that I haven't put them down as 
commitments is because, as Rikki noted, the present design does 
not yet allow us to make those commitments yet. The multi-root 
design I've proposed would allow us to deliver them without 
breaking Windows, but my campaign to get it accepted hasn't been 
approved yet.

And you did hit on an important point though, the things that we 
are looking at adding are cross-cutting concerns that are 
applicable to a wide variety of software and that's the guideline 
we're going to follow moving forward.
Feb 02
prev sibling parent reply Lance Bachmeier <no spam.net> writes:
On Friday, 2 February 2024 at 14:46:45 UTC, aberba wrote:

 On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about 
 Phobos 3 and we felt it would be appropriate to share some 
 notes on our discussion.

 [...]


 What about the missing standard library pieces  that are core 
 to everyday software dev? Stuff you get out of the box with 
 java/python. Is there such considerations beyond fixing the 
 mistakes of v2?

 * Json
 * XML
 * Streams
 * Networking (http, socket, ...)
 ....



 If I understand correctly, v3 is about . How about new module 
 additions to v3 such as the above?


A better solution would be something like this:

```
dub dxml;

void main() {
   // Call functions from the dxml package
}
```

The compiler would know about certain popular packages on 
code.dlang.org. It would download the package if needed, and it 
would know where to find the relevant files.

There are good reasons to not add that stuff to Phobos. The 
standard to get something in Phobos is high enough that it leads 
to a lot of wasted time for many people, it doesn't bloat the 
compiler download size, it takes advantage of work that's already 
been done, and it could be done quickly, rather than the years it 
takes to get stuff done otherwise. (I'm writing this fully aware 
of how things get done around here, but I'm going to put it out 
there anyway.)
Feb 02
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Friday, 2 February 2024 at 16:46:29 UTC, Lance Bachmeier wrote:

 On Friday, 2 February 2024 at 14:46:45 UTC, aberba wrote:

 [...]


 A better solution would be something like this:

 ```
 dub dxml;

 [...]


You think dub should be integrated into the language?
Feb 02
parent Lance Bachmeier <no spam.net> writes:
On Friday, 2 February 2024 at 18:03:32 UTC, jmh530 wrote:

 On Friday, 2 February 2024 at 16:46:29 UTC, Lance Bachmeier 
 wrote:

 On Friday, 2 February 2024 at 14:46:45 UTC, aberba wrote:

 [...]


 A better solution would be something like this:

 ```
 dub dxml;

 [...]


 You think dub should be integrated into the language?


It wouldn't be necessary. The compiler would only have to know 
how to download the source files of a Dub package if they're not 
currently available. Other than that, there'd be no difference to 
the compiler/language between this and the `import std.whatever` 
statements we use right now.
Feb 02
prev sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On Friday, 2 February 2024 at 16:46:29 UTC, Lance Bachmeier wrote:

 On Friday, 2 February 2024 at 14:46:45 UTC, aberba wrote:

 A better solution would be something like this:

 ```
 dub dxml;

 void main() {
   // Call functions from the dxml package
 }
 ```

 The compiler would know about certain popular packages on 
 code.dlang.org. It would download the package if needed, and it 
 would know where to find the relevant files.


I cannot argue strenuously enough against putting the packaging 
system in to the compiler directly. Not only will it balloon the 
build times by orders of magnitude, it is a serious violation of 
separation-of-concerns. The compiler should concern itself with 
compiling code and nothing else. The compiler is the wrong place 
to be handling third-party packages, that is properly the venue 
of the build system.


 There are good reasons to not add that stuff to Phobos. The 
 standard to get something in Phobos is high enough that it 
 leads to a lot of wasted time for many people, it doesn't bloat 
 the compiler download size, it takes advantage of work that's 
 already been done, and it could be done quickly, rather than 
 the years it takes to get stuff done otherwise. (I'm writing 
 this fully aware of how things get done around here, but I'm 
 going to put it out there anyway.)


People always say there are good reasons, but are light on actual 
reasons. Saying that Phobos has a high bar is not itself a reason 
so much as an excuse to not make the effort.

In any case, I have a slightly different philosophy around PR's. 
I do not consider bikeshedding over formatting and other trivia 
to be a valid reason to delay a PR. I've posted about this 
elsewhere. If your code passes it's tests, delivers what it says 
it delivers, and helps us achieve our stated goals for the 
release, it goes in. Will bugs escape, sure, bugs escaping is 
normal, but I find our habit of using bikeshedding to stall out 
PRs we don't agree with to be rather odious. Merge buttons exist 
to prevent people from tying up PR's in bureaucratic nonsense. I 
intend to push them.

Furthermore, if you disagree with how a PR does something that is 
non-trivial, either offer up a competing PR or remain silent. If 
you do not offer up a competing PR then we will assume that you 
are either not as concerned as your comments make you sound, or 
you are not actually sure that your solution is better. In either 
case, you can always submit a PR to improve upon the original PR. 
PR's are free, let's make more of them.

Under my watch (if the community doesn't kick me out) Phobos will 
move faster. And yes, there might even be more bugs that escape. 
My plan is to use the monthly(Atila?) releases between the major 
roll-up releases mentioned in my original post to catch them 
before they get to the wider population.
Feb 02
parent reply Lance Bachmeier <no spam.net> writes:
On Friday, 2 February 2024 at 18:57:48 UTC, Adam Wilson wrote:

 On Friday, 2 February 2024 at 16:46:29 UTC, Lance Bachmeier 
 wrote:

 On Friday, 2 February 2024 at 14:46:45 UTC, aberba wrote:

 A better solution would be something like this:

 ```
 dub dxml;

 void main() {
   // Call functions from the dxml package
 }
 ```

 The compiler would know about certain popular packages on 
 code.dlang.org. It would download the package if needed, and 
 it would know where to find the relevant files.


 I cannot argue strenuously enough against putting the packaging 
 system in to the compiler directly. Not only will it balloon 
 the build times by orders of magnitude, it is a serious 
 violation of separation-of-concerns. The compiler should 
 concern itself with compiling code and nothing else. The 
 compiler is the wrong place to be handling third-party 
 packages, that is properly the venue of the build system.


I responded to the other comment, but I'll give the same answer. 
What I'm proposing doesn't have anything to do with the packaging 
system, Dub, or any of that. The only thing the compiler would 
have to know is how to download the source files of a package if 
they haven't already been downloaded.


 There are good reasons to not add that stuff to Phobos. The 
 standard to get something in Phobos is high enough that it 
 leads to a lot of wasted time for many people, it doesn't 
 bloat the compiler download size, it takes advantage of work 
 that's already been done, and it could be done quickly, rather 
 than the years it takes to get stuff done otherwise. (I'm 
 writing this fully aware of how things get done around here, 
 but I'm going to put it out there anyway.)


 People always say there are good reasons, but are light on 
 actual reasons. Saying that Phobos has a high bar is not itself 
 a reason so much as an excuse to not make the effort.


A sufficient reason is that I know how things get done around 
here, and an updated Phobos with these additions will 
optimistically be released in 2032. Most of the work has already 
been done. Why not reuse it?

It's been eight months since I proposed a simple solution to the 
problem that library writers have to support too many compiler 
releases. Label some compiler releases as not being supported by 
library writers. It would have solved the problem fully and it 
could have been done by the end of the day. Obviously we couldn't 
do that. Instead we got editions. Eight months later, we still 
don't have an initial proposal, and the problem for library 
writers continues.


 Under my watch (if the community doesn't kick me out) Phobos 
 will move faster. And yes, there might even be more bugs that 
 escape. My plan is to use the monthly(Atila?) releases between 
 the major roll-up releases mentioned in my original post to 
 catch them before they get to the wider population.


I hope that's true. It's hard to change the culture of any 
organization, especially one built on volunteer labor.
Feb 02
next sibling parent Mengu <mengukagan gmail.com> writes:
On Friday, 2 February 2024 at 19:39:34 UTC, Lance Bachmeier wrote:

 A sufficient reason is that I know how things get done around 
 here, and an updated Phobos with these additions will 
 optimistically be released in 2032. Most of the work has 
 already been done. Why not reuse it?


If I'm not mistaken Go has this for over 10 years now. I'm not 
aware of any other language/ecosystem that followed this.
Feb 02
prev sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On Friday, 2 February 2024 at 19:39:34 UTC, Lance Bachmeier wrote:

 On Friday, 2 February 2024 at 18:57:48 UTC, Adam Wilson wrote:

 I cannot argue strenuously enough against putting the 
 packaging system in to the compiler directly. Not only will it 
 balloon the build times by orders of magnitude, it is a 
 serious violation of separation-of-concerns. The compiler 
 should concern itself with compiling code and nothing else. 
 The compiler is the wrong place to be handling third-party 
 packages, that is properly the venue of the build system.


 I responded to the other comment, but I'll give the same 
 answer. What I'm proposing doesn't have anything to do with the 
 packaging system, Dub, or any of that. The only thing the 
 compiler would have to know is how to download the source files 
 of a package if they haven't already been downloaded.


But that's not true is it? How does it know which versions of the 
package are acceptable? Which of the acceptable versions does it 
pull? How does it handle the packages' dependencies? You simply 
cannot have the compiler pull a package without going through the 
entire dependency resolution process. Which means you're building 
DUB into DMD. Compilers are not package managers. I've asked 
Walter about this idea before and the reception was ... not 
positive, so probably best to just drop it.


 People always say there are good reasons, but are light on 
 actual reasons. Saying that Phobos has a high bar is not 
 itself a reason so much as an excuse to not make the effort.


 A sufficient reason is that I know how things get done around 
 here, and an updated Phobos with these additions will 
 optimistically be released in 2032. Most of the work has 
 already been done. Why not reuse it?


That's not a reason. You're saying that you know what the reason 
is, that you're not going to state the reason, but that your 
special knowledge is so perfect that you can confidently state a 
ridiculous timeline as fact? I would suggest that you might want 
to consider how superlatively arrogant that sounds. DConf 2024 is 
probably a stretch, but sometime in 2025 is probably realistic, 
unless the money floodgates open and time can be purchased, but 
I'm not holding my breath for that outcome.


 It's been eight months since I proposed a simple solution to 
 the problem that library writers have to support too many 
 compiler releases. Label some compiler releases as not being 
 supported by library writers. It would have solved the problem 
 fully and it could have been done by the end of the day. 
 Obviously we couldn't do that. Instead we got editions. Eight 
 months later, we still don't have an initial proposal, and the 
 problem for library writers continues.


That is pretty much where this is going to end. Walter and I 
discussed tying Phobos releases to Compiler Editions, and he was 
amenable to the idea, his only sticking point was to not use 
"Editions" to describe Phobos, and "Versions" or "Releases" works 
just fine for our purposes. I would expect to see Phobos 3 land 
with the first Edition. Both of which I would expect in 2025.

I am sorry to hear that it's not getting there as fast as you 
want it to, but that is the nature of volunteer projects, so 
unless you're willing to step up and help out, it's going to 
happen at pretty much whatever pace the rest of us can afford. 
But don't expect much sympathy when you're whining about a 
problem you're not doing anything to help solve.


 Under my watch (if the community doesn't kick me out) Phobos 
 will move faster. And yes, there might even be more bugs that 
 escape. My plan is to use the monthly(Atila?) releases between 
 the major roll-up releases mentioned in my original post to 
 catch them before they get to the wider population.


 I hope that's true. It's hard to change the culture of any 
 organization, especially one built on volunteer labor.


Yes and no, so long as the primary volunteers working on a 
specific piece stay the same then yes, expect similar results. 
But change out the personnel and I would expect the results to 
change. This is also true in a for-profit corp, but to a lesser 
degree, as volunteers are usually far less restricted by onerous 
policies.

If I run Phobos development, I will run it differently. Walter 
has asked that I not go crazy, but so far everything I've talked 
about has been met with some variant of "that's ... different" so 
skepticism yes, but so far Walter and the DLF have been willing 
to extend me that latitude.
Feb 02
parent reply Lance Bachmeier <no spam.net> writes:
On Saturday, 3 February 2024 at 03:36:08 UTC, Adam Wilson wrote:

 On Friday, 2 February 2024 at 19:39:34 UTC, Lance Bachmeier 
 wrote:

 On Friday, 2 February 2024 at 18:57:48 UTC, Adam Wilson wrote:

 I cannot argue strenuously enough against putting the 
 packaging system in to the compiler directly. Not only will 
 it balloon the build times by orders of magnitude, it is a 
 serious violation of separation-of-concerns. The compiler 
 should concern itself with compiling code and nothing else. 
 The compiler is the wrong place to be handling third-party 
 packages, that is properly the venue of the build system.


 I responded to the other comment, but I'll give the same 
 answer. What I'm proposing doesn't have anything to do with 
 the packaging system, Dub, or any of that. The only thing the 
 compiler would have to know is how to download the source 
 files of a package if they haven't already been downloaded.


 But that's not true is it? How does it know which versions of 
 the package are acceptable?


It doesn't need to if only the most recent version as of the 
compiler release is supported.


 Which of the acceptable versions does it pull?


Answered above.


 How does it handle the packages' dependencies? You simply 
 cannot have the compiler pull a package without going through 
 the entire dependency resolution process.


This is actually a non-issue for many packages on code.dlang.org. 
I've been downloading the D source files and putting them in my 
repos for years. I specified that this would be for some of the 
popular packages, not arbitrary packages.



 People always say there are good reasons, but are light on 
 actual reasons. Saying that Phobos has a high bar is not 
 itself a reason so much as an excuse to not make the effort.


 A sufficient reason is that I know how things get done around 
 here, and an updated Phobos with these additions will 
 optimistically be released in 2032. Most of the work has 
 already been done. Why not reuse it?


 That's not a reason. You're saying that you know what the 
 reason is, that you're not going to state the reason,


I actually listed multiple good reasons in my original comment, 
all of which you ignored, so I gave another.


 It's been eight months since I proposed a simple solution to 
 the problem that library writers have to support too many 
 compiler releases. Label some compiler releases as not being 
 supported by library writers. It would have solved the problem 
 fully and it could have been done by the end of the day. 
 Obviously we couldn't do that. Instead we got editions. Eight 
 months later, we still don't have an initial proposal, and the 
 problem for library writers continues.


 That is pretty much where this is going to end. Walter and I 
 discussed tying Phobos releases to Compiler Editions, and he 
 was amenable to the idea, his only sticking point was to not 
 use "Editions" to describe Phobos, and "Versions" or "Releases" 
 works just fine for our purposes. I would expect to see Phobos 
 3 land with the first Edition. Both of which I would expect in 
 2025.

 I am sorry to hear that it's not getting there as fast as you 
 want it to, but that is the nature of volunteer projects, so 
 unless you're willing to step up and help out, it's going to 
 happen at pretty much whatever pace the rest of us can afford. 
 But don't expect much sympathy when you're whining about a 
 problem you're not doing anything to help solve.


This...doesn't make sense. There was a complaint that library 
writers have to support too many compiler releases. Well, that's 
an easy problem to solve, which the open source world has used 
for decades. Just say "we only support releases X and Y". I even 
gave examples. Wouldn't take any work at all, just a simple 
management decision, done.

Now you're telling me I should just shut up because I'm not the 
one doing any of the work that doesn't need to be done?

I give up.
Feb 03
parent Adam Wilson <flyboynw gmail.com> writes:
On Saturday, 3 February 2024 at 19:08:34 UTC, Lance Bachmeier 
wrote:

 It doesn't need to if only the most recent version as of the 
 compiler release is supported.


This makes a number of assumptions about both the package itself 
and is inherently limiting. Problems:

1. Updating a package for security/bugs/etc. now requires a 
compiler update.
2. Chicken-egg problems. Compiler makes breaking change that 
breaks the package, but the package maintainer cannot update the 
package until after the compiler has shipped. Therefore the 
package, and all of it's users, are broken for an entire release 
cycle. This could theoretically be solved with extremely tight 
coordination with package maintainers, but nothing in D's history 
suggests that there is even a remote possibility of this 
occurring on even a semi-repeatable basis. The package 
maintainers are volunteers too, if they miss the deadline, broken 
code ships and people yell at us for shipping broken code.
3. More chicken-egg problems. The syntax you propose implies the 
execution of DUB in the background. OK, but DUB is built with 
DMD, so by this design, you need DUB to pull a Phobos package to 
build DUB. Yes, there are bootstrapping designs, but ask Iain or 
Kinke about the bootstrapping rules for GCC or LLVM some time, 
they are highly restrictive.
4. Enforces the use of DUB, no other tool/package system can be 
built or substituted without changing the compiler syntax. D has 
a relatively diverse build tool ecosystem, are we telling them to 
just go pound sand?
5. Not all packages conform to Phobos' standards. Many 
gc-allocate heavily, or use exceptions, etc. By doing this we are 
explicitly telling entire groups of the D userspace that chunks 
of Phobos aren't for them, but that we can't easily tell them 
which chunks those are because we don't control them, so now they 
have to go lobby individual maintainers to support them. The 
screaming about Phobos' problems are loud enough without inviting 
even more trouble by pulling in packages that were never intended 
or designed to be held to Phobos' standards.

And this doesn't address the fact that the first thing the 
community will ask for is the ability to specify package versions.

I can hear Atila (Mr. Build-Speed himself) warming up his fingers 
from here for an explanation as to why any one of the above four 
problems make this idea untenable. But all five make it a 
complete non-starter with everybody who maintains the compiler. I 
submit that there are more useful ways to expend energy than 
pining for something that is simply never going to happen.


 This is actually a non-issue for many packages on 
 code.dlang.org. I've been downloading the D source files and 
 putting them in my repos for years. I specified that this would 
 be for some of the popular packages, not arbitrary packages.


This implies that there are design constraints on the package 
then (no dependencies?), correct?


 I actually listed multiple good reasons in my original comment, 
 all of which you ignored, so I gave another.


You have correctly pointed out that I did not do a point-by-point 
answer to your claims so here you go. (Relevant snippet below 
with responses interspersed)


 There are good reasons to not add that stuff to Phobos.


I disagree, expanding Phobos expands the range of software that 
can be built without needing DUB, thus expanding the capabilities 
of software that can be built where DUB is unavailable or 
untenable, while avoiding the problems listed above.


 The standard to get something in Phobos is high enough that it 
 leads to a lot of wasted time for many people,


It is high, and there is a good reason for that. Phobos has to 
work everywhere, it has to meet a broad variety of use case, and 
user environments.

I do agree that there are adjustments that could be made. For 
example, we tend to spend a lot of time on unnecessary 
bikeshedding and in some cases bikeshedding is used to 
intentionally stall out PRs. So one change than can be made is to 
focus on more qualitative reviews and not allow bikeshedding to 
stall a PR. Another, more controversial, change that I've been 
considering proposing, but know is going to receive considerable 
resistance, is allowing PR's that do not fulfill all the 
environmental rules (like not using the GC or using exceptions) 
to pass and then inviting others to modify the code to fit those 
rules. This is a deeply controversial idea and the debates I've 
had that only nibble at the edges of it have been ... emotional.


 it doesn't bloat the compiler download size,


Just how much does the library code add to the download though? 
Most of the download size in the binaries, not the library code. 
Assuming an atrocious 5:1 text-compression ratio we would have to 
add 10MB of code to add just 2MB of download size. For reference, 
Phobos V 2.106 is 12.6MB on my NTFS partition. We could triple 
the capability of the standard library for the cost of an 
additional 4MB on a 32MB installer. In this modern era of 
Cable/DSL/Fiber I fail to see how an additional megabyte or two 
of download size is a serious concern, especially since 10-20MB 
of library code is a serious amount of additional utility that 
would be available OOTB.


 it takes advantage of work that's already been done,


This is true, but is the work that's been done of the quality we 
want to ship as a top-line package? Does it follow the standards 
and idioms that the users of Phobos are used too?


 and it could be done quickly, rather than the years it takes to 
 get stuff done otherwise.


This is likely untrue. First, even using DUB as an external 
executable, you still need to teach DMD how to execute DUB and 
parse the `dub -v` output. That is a non-trivial amount of work. 
And we would have to figure out how this interacts with 
traditional `dub` project files (if two different versions are 
specified, what happens? The interactions are non-trivial and 
will require significant amount of code and testing of the 
compiler. Then you have to teach the library which packages are 
acceptable and manually maintain lists of versions match which 
compilers.


 This...doesn't make sense. There was a complaint that library 
 writers have to support too many compiler releases. Well, 
 that's an easy problem to solve, which the open source world 
 has used for decades. Just say "we only support releases X and 
 Y". I even gave examples. Wouldn't take any work at all, just a 
 simple management decision, done.

 Now you're telling me I should just shut up because I'm not the 
 one doing any of the work that doesn't need to be done?

 I give up.


As I pointed out above, there is in fact significant work that 
would need to be done.
Feb 03
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 When then moved on to a conversation about how Walter envisions 
 editions actually working. Since none of have seen the document 
 that Atila is working on, Walter shared his opinions on how it 
 should work. [...]

 [...] Atila, if you're reading this, this is what Walter was 
 thinking/hoping would appear, but since we don't have your 
 document we were mostly left to speculate. We await your 
 document drop with anticipation!


Why isn't Atila's document stored in a respository or shared 
drive somewhere that Walter has read access to? I understand not 
wanting to release it to the public before it's ready, but the 
fact that even the two leaders of the D project are forced to 
*guess* what the other is working on is completely ridiculous.

The good news is that this is a really easy problem to solve. If 
setting up a private Github repository and pushing to it is too 
much effort, even something as simple as using Google docs would 
work.


 If you would like to participate in the design discussion on 
 Phobos 3, I am currently hosting a repo on my GitHub with GH 
 Discussions for unresolved/undesigned topics, and PR's for 
 editing the actual design document. Link is here: 
 https://github.com/LightBender/PhobosV3-Design


Great to see. Is there a plan to update this repository and/or 
its Discussions page to reflect the material covered in this 
post? If not, how can I volunteer to help make that happen?
Feb 02
parent Adam Wilson <flyboynw gmail.com> writes:
On Friday, 2 February 2024 at 16:46:08 UTC, Paul Backus wrote:

 On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 When then moved on to a conversation about how Walter 
 envisions editions actually working. Since none of have seen 
 the document that Atila is working on, Walter shared his 
 opinions on how it should work. [...]

 [...] Atila, if you're reading this, this is what Walter was 
 thinking/hoping would appear, but since we don't have your 
 document we were mostly left to speculate. We await your 
 document drop with anticipation!


 Why isn't Atila's document stored in a respository or shared 
 drive somewhere that Walter has read access to? I understand 
 not wanting to release it to the public before it's ready, but 
 the fact that even the two leaders of the D project are forced 
 to *guess* what the other is working on is completely 
 ridiculous.


Probably because reaching for GitHub is not how people 
think-by-default when thinking about a document. I only stumbled 
to it because Walter sent me a markdown document for his Phobos 3 
ideas and that randomly triggered the connection in my head. I 
agree that the present situation is less than ideal, but it's 
what we've got to work with.


 The good news is that this is a really easy problem to solve. 
 If setting up a private Github repository and pushing to it is 
 too much effort, even something as simple as using Google docs 
 would work.


 If you would like to participate in the design discussion on 
 Phobos 3, I am currently hosting a repo on my GitHub with GH 
 Discussions for unresolved/undesigned topics, and PR's for 
 editing the actual design document. Link is here: 
 https://github.com/LightBender/PhobosV3-Design


 Great to see. Is there a plan to update this repository and/or 
 its Discussions page to reflect the material covered in this 
 post? If not, how can I volunteer to help make that happen?


I merged two of the open PRs on the document last night based on 
the discussion. I have one more to do (DUB Only Builds) but I ran 
out of steam and needed to get some sleep. There is a discussion 
open on Versioned Roots, but I talked Walter out of that, so I 
will post updates there and I have opened a PR to my proposed 
multi-root design here: 
https://github.com/LightBender/PhobosV3-Design/pull/3 but I need 
to update it on the basis of the discussion. Time is, as always, 
in short supply.

If you have ideas about the general design that aren't fully 
formed or need to debated please open up a new Discussion (if I 
need some form of access let me know, this is my first time using 
Discussions). If you have a specific change proposal to the 
document please open up a PR and we'll discuss it in the Review 
comments.
Feb 02
prev sibling next sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
Concerning allocators, needs a simpler API in `core.memory`, 
without assumptions and without typeinfo, so libraries can depend 
on it from bare metal to higher level desktop applications, i 
dream of a way to change the associated array's allocator for a 
custom one for example

The one in `std.experimental` is too over engineered to depend on 
it

Concerning `std.traits` is this package really needed? perhaps 
work should go in improving `__traits` instead?

I agree with aberba, i view the STD as building blocks for your 
program, instead of just utility functions, `std.net` `std.event` 
`std.container` `std.database` etc

Hence i view the "fix std.traits` a waste of time, at least for a 
Phobos 3 work

To me Phobos 3 should be an opportunity to make D's std 
competitive with other language's std, `std.net` shouldn't be a 
binding to curl for example, and if that's not in phobos 3's 
goals, then what's the point? Too many people asking how to ship 
with curl on windows.. that shouldn't be a problem when someone 
is using the language's std, it should just work
Feb 02
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/2/2024 1:09 AM, Adam Wilson wrote:

 Finally, we touched briefly on the major changes we would like to see in
Phobos 
 3 and these are the major changes we are committing to for Phobos 3 so far:
 - Removal of Autodecoding.
 - Promoting allocators out of experimental.
 - Range interface redesign (see JMD's thread 
 [here](https://forum.dlang.org/thread/mailman.588.1705813271.3719.digitalmars-d puremagic.com)).
 - Fix std.traits.


- Drop all support for wchar and dchar from the templates. Instead, use the 
adaptors from std.utf to convert wchar/dchar to/from char.
Feb 02
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about 
 Phobos 3 and we felt it would be appropriate to share some 
 notes on our discussion.

 [snip]


When it comes to discussions around functions that allocate to 
the heap, I think there is some benefit in implementing 
lower-level functionality in functions that do not do any 
allocations whatsoever. You can then have higher level API above 
that that controls allocation and is more convenient to use. That 
way users can opt in to whatever they want or write their own 
more convenient API on top of the lower level functions.

May not work for everything, but could be a useful approach.
Feb 02
next sibling parent reply Hipreme <msnmancini hotmail.com> writes:
On Friday, 2 February 2024 at 18:19:47 UTC, jmh530 wrote:

 On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about 
 Phobos 3 and we felt it would be appropriate to share some 
 notes on our discussion.

 [snip]


 When it comes to discussions around functions that allocate to 
 the heap, I think there is some benefit in implementing 
 lower-level functionality in functions that do not do any 
 allocations whatsoever. You can then have higher level API 
 above that that controls allocation and is more convenient to 
 use. That way users can opt in to whatever they want or write 
 their own more convenient API on top of the lower level 
 functions.

 May not work for everything, but could be a useful approach.



It may be still a bad solution. Countless times I have done 
functions which know how much to preallocate. The eager solution 
always seems to be a lot faster, and I have noticed the pattern 
of using `map` or `filter` or anything more + `.array` looking to 
infinitely slower than if you had a direct version. And I don't 
know about `release` versions. The thing is that 99% of the time 
I'm testing on debug builds and getting less speed only because 
I'm on debug looks really bad.

This should be addressed.
Feb 02
parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 2 February 2024 at 18:39:33 UTC, Hipreme wrote:

 On Friday, 2 February 2024 at 18:19:47 UTC, jmh530 wrote:

 On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about 
 Phobos 3 and we felt it would be appropriate to share some 
 notes on our discussion.

 [snip]


 When it comes to discussions around functions that allocate to 
 the heap, I think there is some benefit in implementing 
 lower-level functionality in functions that do not do any 
 allocations whatsoever. You can then have higher level API 
 above that that controls allocation and is more convenient to 
 use. That way users can opt in to whatever they want or write 
 their own more convenient API on top of the lower level 
 functions.

 May not work for everything, but could be a useful approach.



 It may be still a bad solution. Countless times I have done 
 functions which know how much to preallocate. The eager 
 solution always seems to be a lot faster, and I have noticed 
 the pattern of using `map` or `filter` or anything more + 
 `.array` looking to infinitely slower than if you had a direct 
 version. And I don't know about `release` versions. The thing 
 is that 99% of the time I'm testing on debug builds and getting 
 less speed only because I'm on debug looks really bad.


The work I’ve been doing is related to linear algebra. Nothing 
really lazy or map-like. For instance, think of it like dgemm 
doing matrix multiplication without doing any allocations. It 
takes a pointer to the result matrix as an input and requires 
information about how  if that will be. A higher level D version 
could work in terms of slices so that the API is cleaner. A 
higher level API could handle the case where the result is 
allocated with GC.
Feb 04
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, February 2, 2024 11:19:47 AM MST jmh530 via Digitalmars-d wrote:

 On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about
 Phobos 3 and we felt it would be appropriate to share some
 notes on our discussion.

 [snip]


 When it comes to discussions around functions that allocate to
 the heap, I think there is some benefit in implementing
 lower-level functionality in functions that do not do any
 allocations whatsoever. You can then have higher level API above
 that that controls allocation and is more convenient to use. That
 way users can opt in to whatever they want or write their own
 more convenient API on top of the lower level functions.

 May not work for everything, but could be a useful approach.


That's certainly the approach that I think that we should be taking for a
number of things (and to an extent, it's what Phobos already does with some
stuff - just not everywhere). There are plenty of cases where if you really
want something user-friendly, you need to be giving fewer options and just
allocating stuff on the heap, but it often makes perfect sense to build that
on top of much less user-friendly stuff that provides more functionality for
those who really need it.

This is exactly the approach that I've taken with my socket library that I
need to finish one of these days. It provides a low-level API using structs
and  nogc, whereas for higher level socket stuff (e.g. code that needs to
operate on sockets without caring whether they're using SSL/TLS or not),
you're almost certainly going to want classes - and you're definitely going
to want stuff on the heap given the need to do stuff like point to sockets
on other threads to close them.

A much simpler example of this sort of thing that we already have in Phobos
is range-based functions vs array functions - e.g. splitter vs split. The
range-based ones are much more flexible, and they typically don't allocate
(though in some cases might due to closures), but they're also much more
verbose when you actually want to operate on arrays, since then you end up
with calls to array all over the place, and you potentially lose out on
optimization opportunites for those cases where an algorithm can be
implemented more efficiently for an array. So, by having both, you have the
option to avoid the allocations and work with a wider range of types - or
you can use the array-specific version when you don't need those extra
capabilities.

- Jonathan M Davis
Feb 02
prev sibling next sibling parent monkyyy <crazymonkyyy gmail.com> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Finally, we touched briefly on the major changes we would like 
 to see in Phobos 3 and these are the major changes we are 
 committing to for Phobos 3 so far:
 - Range interface redesign


https://forum.dlang.org/thread/ghsihtmlurzsuslyzgnv forum.dlang.org
Feb 02
prev sibling next sibling parent reply Andrea Fontana <nospam example.org> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 First, it would be very easy to accidentally type `std.` in an 
 `std2.` module. We realized that this would be a problem no 
 matter what root name was used and that the appropriate answer 
 would be to specify a DStyle rule specific to Phobos that only 
 modules in `std.` can import `std.` modules.


What about std.v1.*, std.v2.* and so on? And obviously std.v1.* 
== std.*


 Therefore, the compiler should default to use the latest 
 version and then provide the ability via a switch to set the 
 edition, or lack thereof, of the modules an import path. This 
 solves the problem of abandon-ware packages being accessible 
 without presenting the new user with an ever more decayed 
 version of the compiler. We want to put out best foot forward 
 and presenting the last pre-editions release, which is 
 constantly getting old as time passes, does not do that. No 
 specific agreement was reached but Walter agreed to consider it.


Is the newest compiler compatible with oldest phobos?
Maybe some sort of automagical import std.old.v108;


 After that we had a discussion about how to distribute Phobos. 
 This mostly centered on what release cadence to use. I argued 
 for linking the Phobos version to the edition release schedule. 
 I think this is sensible and makes it easier for people to 
 reason about which compiler/library pairing they are using. 
 Walter was fine with that, but he does not want to use the 
 "Edition" language to describe Phobos releases. I think this 
 makes sense as Phobos doesn't really have editions, but 
 following SemVer, it would have yearly major releases. This 
 would mean that new features are released once per year, and 
 bugfixes would be released on whatever the fast-cadence 
 schedule is for DMD in between Yearly Edition roll-ups.


Can phobos and dmd split themselves, just like any other library, 
or are we supposed to keep them together?


 Finally, we touched briefly on the major changes we would like 
 to see in Phobos 3 and these are the major changes we are 
 committing to for Phobos 3 so far:
 - Removal of Autodecoding.


+1


 - Promoting allocators out of experimental.


+1


 - Range interface redesign


+1


 - Fix std.traits.


+1

- Removing ext dependencies could make things easier (f.e. 
libcurl) for multi-platform development (it's nice to statically 
compile everything in a single executable, but it's not that 
easy: I did it for https://github.com/trikko/tshare)
- Phobos with webassembly
- Phobos with betterC

Andrea
Feb 02
parent reply Adam Wilson <flyboynw gmail.com> writes:
On Friday, 2 February 2024 at 21:11:51 UTC, Andrea Fontana wrote:

 On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 First, it would be very easy to accidentally type `std.` in an 
 `std2.` module. We realized that this would be a problem no 
 matter what root name was used and that the appropriate answer 
 would be to specify a DStyle rule specific to Phobos that only 
 modules in `std.` can import `std.` modules.


 What about std.v1.*, std.v2.* and so on? And obviously std.v1.* 
 == std.*


 Therefore, the compiler should default to use the latest 
 version and then provide the ability via a switch to set the 
 edition, or lack thereof, of the modules an import path. This 
 solves the problem of abandon-ware packages being accessible 
 without presenting the new user with an ever more decayed 
 version of the compiler. We want to put out best foot forward 
 and presenting the last pre-editions release, which is 
 constantly getting old as time passes, does not do that. No 
 specific agreement was reached but Walter agreed to consider 
 it.


 Is the newest compiler compatible with oldest phobos?
 Maybe some sort of automagical import std.old.v108;


This hasn't been fully decided yet, and I probably need to have a 
chat with Atila so we can sort out some of the practical issues. 
But what I am pushing for is tying Phobos versions to matching 
editions. Example: Phobos 3 == Edition 2025 (2024?), Phobos 4 == 
Edition 2026, etc. The plan as I understand it is that Phobos 3 
code can be built with later editions. So the "Edition 2025" is a 
minimum supported edition level.


 After that we had a discussion about how to distribute Phobos. 
 This mostly centered on what release cadence to use. I argued 
 for linking the Phobos version to the edition release 
 schedule. I think this is sensible and makes it easier for 
 people to reason about which compiler/library pairing they are 
 using. Walter was fine with that, but he does not want to use 
 the "Edition" language to describe Phobos releases. I think 
 this makes sense as Phobos doesn't really have editions, but 
 following SemVer, it would have yearly major releases. This 
 would mean that new features are released once per year, and 
 bugfixes would be released on whatever the fast-cadence 
 schedule is for DMD in between Yearly Edition roll-ups.


 Can phobos and dmd split themselves, just like any other 
 library, or are we supposed to keep them together?


I've heard rumors about going mono-repo. But for the moment they 
are in separate repos, and the initial Phobos 3 development will 
likely be done in a separate repo to avoid waiting on and 
conflicting with the primary development path. After that I would 
expect it to land in the same repo as Phobos 2.


 Finally, we touched briefly on the major changes we would like 
 to see in Phobos 3 and these are the major changes we are 
 committing to for Phobos 3 so far:
 - Removal of Autodecoding.


 +1


 - Promoting allocators out of experimental.


 +1


 - Range interface redesign


 +1


 - Fix std.traits.


 +1

 - Removing ext dependencies could make things easier (f.e. 
 libcurl) for multi-platform development (it's nice to 
 statically compile everything in a single executable, but it's 
 not that easy: I did it for https://github.com/trikko/tshare)


Easier said that done, because the one thing you cannot do 
without an external library is cryptography, and if the advice is 
to just use OpenSSL, we would be better served telling them to 
skip that entirely and just check themselves into a mental 
institution directly (I implemented crypto using OpenSSL, AMA).

Specific to libcurl I think we should use the built-in sockets, 
but this requires a bunch of work from JMD, who is also heading 
up ranges. We have a lot of work and few people to help, so we 
have to be judicious here, and C libraries via ImportC offer an 
appealing amount of leverage.


 - Phobos with webassembly


This is mostly a DMD/DRT problem. The big hang-up with WASM has 
been that WASM only recently got GC support and the current 
compiler integrations haven't been updated to support, it you 
want to help out with that, look for a call for volunteers from 
Mike Parker some time in the near future. WASM is pretty high up 
the priority list.


 - Phobos with betterC


This is not in the cards. BetterC would seriously constrain the 
capability of Phobos as we expect to support TypeInfo/ModuleInfo, 
threading, Dynamic/Associative arrays, and static module 
constructors/destructors.


 Andrea
Feb 02
next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Saturday, 3 February 2024 at 03:49:54 UTC, Adam Wilson wrote:

 - Phobos with betterC


 This is not in the cards. BetterC would seriously constrain the 
 capability of Phobos as we expect to support 
 TypeInfo/ModuleInfo, threading, Dynamic/Associative arrays, and 
 static module constructors/destructors.


 Andrea





What typeinfo/moduleinfo has to do with a networking module in 
the STD for example?

Dynamic array/associative array can be made to work with 
-betterC, i mentioned in my reply above, minimal allocator API in 
`core.memory` make associative array able to change its 
allocator, and build APIs in phobos around allocators and people 
who use betterC will be able to consume phobos without people 
working on phobos having to know/care about betterC

If you care about simplicity and efficiency, obviously
Feb 03
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Saturday, 3 February 2024 at 11:53:05 UTC, ryuukk_ wrote:

 On Saturday, 3 February 2024 at 03:49:54 UTC, Adam Wilson wrote:

 - Phobos with betterC


 This is not in the cards. BetterC would seriously constrain 
 the capability of Phobos as we expect to support 
 TypeInfo/ModuleInfo, threading, Dynamic/Associative arrays, 
 and static module constructors/destructors.


 Andrea





 What typeinfo/moduleinfo has to do with a networking module in 
 the STD for example?

 Dynamic array/associative array can be made to work with 
 -betterC, i mentioned in my reply above, minimal allocator API 
 in `core.memory` make associative array able to change its 
 allocator, and build APIs in phobos around allocators and 
 people who use betterC will be able to consume phobos without 
 people working on phobos having to know/care about betterC

 If you care about simplicity and efficiency, obviously


Oh, and Exception Handling?

Get in native tuple and tagged union in the language, and you'll 
be able to build nice error handling code

I hope no EH in Phobos 3
Feb 03
next sibling parent reply Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Saturday, 3 February 2024 at 11:54:48 UTC, ryuukk_ wrote:

 Oh, and Exception Handling?

 Get in native tuple and tagged union in the language, and 
 you'll be able to build nice error handling code

 I hope no EH in Phobos 3


Existing try catch syntax could be improved to support tagged 
union you've mentioned btw.  Compiler then would inject extra 
code on each statement that returns a tagged union to 
automatically forward to appropriate catch clause, if an 
exception is returned from said statement.
Feb 03
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 04/02/2024 1:40 AM, Alexandru Ermicioi wrote:

 On Saturday, 3 February 2024 at 11:54:48 UTC, ryuukk_ wrote:

 Oh, and Exception Handling?

 Get in native tuple and tagged union in the language, and you'll be 
 able to build nice error handling code

 I hope no EH in Phobos 3


 
 Existing try catch syntax could be improved to support tagged union 
 you've mentioned btw.  Compiler then would inject extra code on each 
 statement that returns a tagged union to automatically forward to 
 appropriate catch clause, if an exception is returned from said statement.


Yes it is well known that we want a different exception handling 
implementation in addition to the runtime one we have now.

I suspect I should probably begin doing a write up on sum types. I'm not 
happy with Walter's design. It leaves some things out that are kinda 
needed to match the problem domain.

https://github.com/rikkimax/DIPs/blob/value_type_exceptions/DIPs/DIP1xxx-RC.md
Feb 03
prev sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On Saturday, 3 February 2024 at 11:54:48 UTC, ryuukk_ wrote:

 Oh, and Exception Handling?

 Get in native tuple and tagged union in the language, and 
 you'll be able to build nice error handling code

 I hope no EH in Phobos 3


The EH problem has two potential answers.

The first is the sumtype answer. That would be zero runtime cost 
and would achieve a similar result as the current EH scheme.

The second is that, absent sumtype exceptions, Phobos will be 
designed to not use exceptions wherever possible. There will 
always be cases where exceptions are needed so we can't entirely 
rule them out, but any time one is added it will require some 
serious justification.

This goes back to not wanting to box ourselves into any tight 
corners or inferior designs in the name of slavishly attempting 
to a too-restrictive standard. Consider this potential problem, 
we completely eradicate EH in Phobos 3, create a number of 
less-than-ideal implementations, and then the next Edition of D 
drops with sumtype exceptions and now we have to go back and 
rebuild all those known inferior designs because EH became 
acceptable to the crowd again.

It is better to build the best design up-front, pay a small 
penalty for it, then work on minimizing the penalty.
Feb 03
parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 4 February 2024 at 01:14:56 UTC, Adam Wilson wrote:

 On Saturday, 3 February 2024 at 11:54:48 UTC, ryuukk_ wrote:

 Oh, and Exception Handling?

 Get in native tuple and tagged union in the language, and 
 you'll be able to build nice error handling code

 I hope no EH in Phobos 3


 The EH problem has two potential answers.


Three: copy Herb Sutter's idea.


 The first is the sumtype answer. That would be zero runtime 
 cost and would achieve a similar result as the current EH 
 scheme.


Not unless we solve the "bubbling up" issue to avoid pattern 
matching over and over again when none of the N functions in the 
call stack between `main` and the one "throwing" care. So far all 
I know of is Rust's `?` and monads.
Feb 05
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 06/02/2024 3:41 AM, Atila Neves wrote:

     The EH problem has two potential answers.
 
 Three: copy Herb Sutter's idea.
 


Oh look, someone has already done it!

https://github.com/rikkimax/DIPs/blob/value_type_exceptions/DIPs/DIP1xxx-RC.md
Feb 05
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 5 February 2024 at 15:45:36 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 On 06/02/2024 3:41 AM, Atila Neves wrote:

     The EH problem has two potential answers.
 
 Three: copy Herb Sutter's idea.
 


 Oh look, someone has already done it!

 https://github.com/rikkimax/DIPs/blob/value_type_exceptions/DIPs/DIP1xxx-RC.md


Nice! How did I miss this??
Feb 05
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 06/02/2024 6:38 AM, Atila Neves wrote:

 On Monday, 5 February 2024 at 15:45:36 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 On 06/02/2024 3:41 AM, Atila Neves wrote:

     The EH problem has two potential answers.

 Three: copy Herb Sutter's idea.


 Oh look, someone has already done it!

 https://github.com/rikkimax/DIPs/blob/value_type_exceptions/DIPs/DIP1xxx-RC.md


 
 Nice! How did I miss this??


Probably because you weren't on Discord or reading any of my comments on 
the N.G. ;)
Feb 05
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 3 February 2024 at 03:49:54 UTC, Adam Wilson wrote:

 On Friday, 2 February 2024 at 21:11:51 UTC, Andrea Fontana 
 wrote:

 - Phobos with betterC


 This is not in the cards. BetterC would seriously constrain the 
 capability of Phobos as we expect to support 
 TypeInfo/ModuleInfo, threading, Dynamic/Associative arrays, and 
 static module constructors/destructors.


Obviously we are not going to get all of Phobos to work with 
BetterC, but significant chunks of it already do, and there's a 
good deal more that could be considered low-hanging fruit.

I think the right approach here is to try to make code 
BetterC-compatible when we can, and to at least keep BetterC 
compatibility in mind as a "nice-to-have" goal (but not a 
requirement) when doing API design. For example, if the only 
thing keeping a function from being BetterC-compatible is that it 
uses the GC to allocate memory for an error message, it's 
probably worth a little bit of extra effort to either get rid of 
that GC dependency, or provide a `version (D_BetterC)` 
alternative.
Feb 03
next sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On Saturday, 3 February 2024 at 12:53:34 UTC, Paul Backus wrote:

 Obviously we are not going to get all of Phobos to work with 
 BetterC, but significant chunks of it already do, and there's a 
 good deal more that could be considered low-hanging fruit.

 I think the right approach here is to try to make code 
 BetterC-compatible when we can, and to at least keep BetterC 
 compatibility in mind as a "nice-to-have" goal (but not a 
 requirement) when doing API design. For example, if the only 
 thing keeping a function from being BetterC-compatible is that 
 it uses the GC to allocate memory for an error message, it's 
 probably worth a little bit of extra effort to either get rid 
 of that GC dependency, or provide a `version (D_BetterC)` 
 alternative.


This is the path that we're taking. We want to make Phobos as 
widely usable as possible, but there is a balancing act we have 
to perform. If we restrict the language features we allow 
ourselves to much, there are entire classes of capabilities we 
cannot deliver. BetterC is too much, for example, we wouldn't be 
able to use classes/interfaces or the GC where it's appropriate.

JMD and I were talking about sockets and I think he came up with 
an interesting approach to this problem. He built a low-level API 
that is either BetterC compliant or very close to it, and then 
built a higher level API on top of that. The low-level API is 
painful to use but it allowed him to build the high-level API he 
wanted in a way that made sense. You can use the low-level API if 
you absolutely need that level of performance/control/etc, but 
most projects will find the high-level API to be sufficient for 
their needs.

Perhaps this is a concept that we should investigate for other 
areas of Phobos?
Feb 03
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 4 February 2024 at 01:01:28 UTC, Adam Wilson wrote:

 JMD and I were talking about sockets and I think he came up 
 with an interesting approach to this problem. He built a 
 low-level API that is either BetterC compliant or very close to 
 it, and then built a higher level API on top of that. The 
 low-level API is painful to use but it allowed him to build the 
 high-level API he wanted in a way that made sense. You can use 
 the low-level API if you absolutely need that level of 
 performance/control/etc, but most projects will find the 
 high-level API to be sufficient for their needs.

 Perhaps this is a concept that we should investigate for other 
 areas of Phobos?


Absolutely.

A lot of things in Phobos unnecessarily use classes. I feel very 
strongly that V3 should go in the direction of using structs 
where possible, non-copyable/scoped/unique ones if needed.
Feb 05
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 3 February 2024 at 12:53:34 UTC, Paul Backus wrote:

 On Saturday, 3 February 2024 at 03:49:54 UTC, Adam Wilson wrote:

 [...]


 Obviously we are not going to get all of Phobos to work with 
 BetterC, but significant chunks of it already do, and there's a 
 good deal more that could be considered low-hanging fruit.

 I think the right approach here is to try to make code 
 BetterC-compatible when we can, and to at least keep BetterC 
 compatibility in mind as a "nice-to-have" goal (but not a 
 requirement) when doing API design. For example, if the only 
 thing keeping a function from being BetterC-compatible is that 
 it uses the GC to allocate memory for an error message, it's 
 probably worth a little bit of extra effort to either get rid 
 of that GC dependency, or provide a `version (D_BetterC)` 
 alternative.


I think our goal is to make `-betterC` obsolete. As in: if you 
don't use the feature, you don't pay for it, and it's implicit.
Feb 05
next sibling parent zjh <fqbqrr 163.com> writes:
On Monday, 5 February 2024 at 14:38:56 UTC, Atila Neves wrote:

 I think our goal is to make `-betterC` obsolete. As in: if you 
 don't use the feature, you don't pay for it, and it's implicit.



Very nice.
Feb 05
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 5 February 2024 at 14:38:56 UTC, Atila Neves wrote:

 On Saturday, 3 February 2024 at 12:53:34 UTC, Paul Backus wrote:

 I think the right approach here is to try to make code 
 BetterC-compatible when we can, and to at least keep BetterC 
 compatibility in mind as a "nice-to-have" goal (but not a 
 requirement) when doing API design. For example, if the only 
 thing keeping a function from being BetterC-compatible is that 
 it uses the GC to allocate memory for an error message, it's 
 probably worth a little bit of extra effort to either get rid 
 of that GC dependency, or provide a `version (D_BetterC)` 
 alternative.


 I think our goal is to make `-betterC` obsolete. As in: if you 
 don't use the feature, you don't pay for it, and it's implicit.


This is more of a goal for druntime than Phobos, but yes, agreed.

Even in the 100% pay-as-you-go world, I think we'll still want 
Phobos APIs to avoid depending on (and paying for) more druntime 
features than they really need to.
Feb 05
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Monday, 5 February 2024 at 15:40:11 UTC, Paul Backus wrote:

 On Monday, 5 February 2024 at 14:38:56 UTC, Atila Neves wrote:

 On Saturday, 3 February 2024 at 12:53:34 UTC, Paul Backus 
 wrote:

 I think the right approach here is to try to make code 
 BetterC-compatible when we can, and to at least keep BetterC 
 compatibility in mind as a "nice-to-have" goal (but not a 
 requirement) when doing API design. For example, if the only 
 thing keeping a function from being BetterC-compatible is 
 that it uses the GC to allocate memory for an error message, 
 it's probably worth a little bit of extra effort to either 
 get rid of that GC dependency, or provide a `version 
 (D_BetterC)` alternative.


 I think our goal is to make `-betterC` obsolete. As in: if you 
 don't use the feature, you don't pay for it, and it's implicit.


 This is more of a goal for druntime than Phobos, but yes, 
 agreed.

 Even in the 100% pay-as-you-go world, I think we'll still want 
 Phobos APIs to avoid depending on (and paying for) more 
 druntime features than they really need to.


Yeah, that's a good point. For instance: not allocating for no 
reason, and leaving it up to the user instead. If they *want* to 
call `.array` on the result, that's up to them. Preferring "sink" 
`toString` implementations, that kind of thing.
Feb 05
prev sibling parent reply monkyyy <crazymonkyyy gmail.com> writes:
On Monday, 5 February 2024 at 15:40:11 UTC, Paul Backus wrote:

 I think our goal is to make `-betterC` obsolete. As in: if you 
 don't use the feature, you don't pay for it, and it's implicit.


 Even in the 100% pay-as-you-go world, I think we'll still want 
 Phobos APIs to avoid depending on (and paying for) more 
 druntime features than they really need to.


This feels delusional to me; like I don't know how the formal 
style with its contracts and extra asserts and 5 layered datetime 
will ever avoid rogue imports that aren't part of a blessed 
compile environment.

I have a wasm build script, feel free to test if any single 
function of any of the std can compile with libc.math imports 
t_time which isnt found with the broken wasm enverment.

The current style leads to highly incestuous, overly reliant on 
long correct versions of code that just leads to a highly 
connected graph. I see no suggesting from any "pay-as-you-go std" 
advocate that isn't "we wil be more careful", "we will compile 
with more  nogc" when I dont even know if it would work but a 
solution id see would need to make local imports only be allowed 
in templated code(so it actually functioned as pay as you go), 
drastically cutting back on complexity(no 500 line nullable and 
100 line template contracts)
Feb 05
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 5 February 2024 at 16:36:02 UTC, monkyyy wrote:

 On Monday, 5 February 2024 at 15:40:11 UTC, Paul Backus wrote:

 I think our goal is to make `-betterC` obsolete. As in: if 
 you don't use the feature, you don't pay for it, and it's 
 implicit.


 Even in the 100% pay-as-you-go world, I think we'll still want 
 Phobos APIs to avoid depending on (and paying for) more 
 druntime features than they really need to.


 This feels delusional to me; like I don't know how the formal 
 style with its contracts and extra asserts and 5 layered 
 datetime will ever avoid rogue imports that aren't part of a 
 blessed compile environment.


The way you avoid this stuff is by testing. Run your unit tests 
with `-betterC` and any accidental druntime dependencies you add 
will be revealed to you very quickly.

The current version of Phobos was written before BetterC existed 
and is kind of lazy about testing in general, so it doesn't do 
this, but I'm hoping that will change with Phobos V3.

I've tried to set an example with `std.sumtype`. Unlike every 
other module in Phobos, its unit tests are enabled in BetterC by 
default, and those that depend on druntime have to make this 
explicit by using a `version` condition.
Feb 05
parent monkyyy <crazymonkyyy gmail.com> writes:
On Monday, 5 February 2024 at 16:46:20 UTC, Paul Backus wrote:

 On Monday, 5 February 2024 at 16:36:02 UTC, monkyyy wrote:

 On Monday, 5 February 2024 at 15:40:11 UTC, Paul Backus wrote:

 I think our goal is to make `-betterC` obsolete. As in: if 
 you don't use the feature, you don't pay for it, and it's 
 implicit.


 Even in the 100% pay-as-you-go world, I think we'll still 
 want Phobos APIs to avoid depending on (and paying for) more 
 druntime features than they really need to.


 This feels delusional to me; like I don't know how the formal 
 style with its contracts and extra asserts and 5 layered 
 datetime will ever avoid rogue imports that aren't part of a 
 blessed compile environment.


 The way you avoid this stuff is by testing. Run your unit tests 
 with `-betterC` and any accidental druntime dependencies you 
 add will be revealed to you very quickly.


That just adds a singluar blessed environment, it will be very 
very unlikely that it will compile on wasm, a new embedded chip 
that releases in 2025, bsd, or a new rust os in 2040.

Which is a potential option but it's not a mythical 
"pay-as-you-go" and then you'll see nogc avocates being like "oi 
I needed to import toStringz but your (algorthim thats 100x 
simplier with allocation) allocates and broke my code, because of 
a 9 long import chain of unused code"

https://www.youtube.com/watch?v=a-767WnbaCQ

"just do ___"; no we are not even close to the phase change 
point, I suggest someone needs an answer for how you reduce the 
fundamental "order"/R0 of imports, which I think is 2 imports per 
file, or being extremely strict about local imports being in 
templates so they dont compile if unused
Feb 05
prev sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Mon, Feb 05, 2024 at 02:38:56PM +0000, Atila Neves via Digitalmars-d wrote:
[...]

 I think our goal is to make `-betterC` obsolete. As in: if you don't
 use the feature, you don't pay for it, and it's implicit.


+1, this is the right approach.


T

-- 
It said to install Windows 2000 or better, so I installed Linux instead.
Feb 05
prev sibling next sibling parent Mengu <mengukagan gmail.com> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about 
 Phobos 3 and we felt it would be appropriate to share some 
 notes on our discussion.

 [...]


Will there be a plan to put Phobos 2 on life support and 
eventually sunset it? It doesn't make sense to support that 
forever.
Feb 02
prev sibling next sibling parent reply Meta <jared771 gmail.com> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Fix std.traits


Can you expand on this a bit more? I've contributed to std.traits 
in the past and would like to help improve it for Phobos v2.
Feb 04
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, February 4, 2024 2:07:44 AM MST Meta via Digitalmars-d wrote:

 On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Fix std.traits


 Can you expand on this a bit more? I've contributed to std.traits
 in the past and would like to help improve it for Phobos v2.


The main thing that it needs IMHO is going through it and making sure that
it's doing the right thing for each trait and that the exact set of traits
that it has is really what we want, since right now, several of them are
doing the wrong thing (e.g. treating enum as if it were its base type with
stuff like isIntegral, isNumeric, etc., which is a source of bugs). The
names also potentially need to be cleaned up, and it just needs to be looked
over in general to make sure that it's really what we want. It's grown
organically over the years, the current situation with D is not the same as
when std.traits was started, and we've learned more since then, so the exact
set of traits that we want to have in there likely should be adjusted now
that we have an opportunity to essentially start from scratch with it.

That being said, as I understand it, what initiated the idea to rework it as
one of Phobos v3's priorities is that Walter doesn't understand traits like
isSomeChar and finds them confusing, though I suspect that that's probably
because he hasn't done much with them, if nothing else, because dmd doesn't
use Phobos, and I would expect that it does a lot less with templated code
than a lot of D code does, though I haven't dug through its code lately.

I also get the impression that some folks just plain don't like what the
module does, but given how D's template constraints work, we fundamentally
need traits like it has even if the exact set that is in there needs work.

There's also likely some debate to be had over std.traits vs __traits. On
the whole, std.traits adds stuff on top of __traits, so plenty of folks use
both, but part of the idea behind std.traits was to make it so that most
code wouldn't need to use __traits. The advantage of that is that the result
is cleaner code (since __traits tends to be pretty ugly), but it also means
more template instantiations. So, I expect that there is going to be some
disagreement on what we should be doing there (though there's no reason why
we can't have traits in std.traits which simply wrap __traits stuff even if
some folks want to use __traits directly as much as possible). But
regardless, there are a number of things that std.traits tests that aren't
in __traits.

In any case, as things stand, I don't think there really is agreement on
what should be done with std.traits, and deciding what exactly needs to
happen with it is one of the things that we're going to need to figure out.

- Jonathan M Davis
Feb 04
next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Sunday, 4 February 2024 at 09:30:58 UTC, Jonathan M Davis 
wrote:

 So, I expect that there is going to be some disagreement on 
 what we should be doing there (though there's no reason why we 
 can't have traits in std.traits which simply wrap __traits 
 stuff even if some folks want to use __traits directly as much 
 as possible). But regardless, there are a number of things that 
 std.traits tests that aren't in __traits.

 In any case, as things stand, I don't think there really is 
 agreement on what should be done with std.traits, and deciding 
 what exactly needs to happen with it is one of the things that 
 we're going to need to figure out.

 - Jonathan M Davis



On Sunday, 4 February 2024 at 02:25:25 UTC, Adam Wilson wrote:

 This goes back to not wanting to box ourselves into any tight


 corners or inferior designs in the name of slavishly attempting 
 to a too-restrictive standard. Consider this potential problem, 
 we completely eradicate EH in Phobos 3, create a number of 
 less-than-ideal implementations, and then the next Edition of D 
 drops with sumtype exceptions and now we have to go back and 
 rebuild all those known inferior designs because EH became 
 acceptable to the crowd again.



D's feature has problem, and is annoying to use? Don't be like 
C++, don't bloat the std, fix the language

STD should not be the place to fix language's shortcoming


EH is evil, some platforms has banned it completly (apple), and 

application in Rust too, not just C++, it is an inferior design


https://www.theregister.com/2024/01/31/microsoft_seeks_rust_developers/
Feb 04
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, February 4, 2024 3:13:39 AM MST ryuukk_ via Digitalmars-d wrote:

 EH is evil, some platforms has banned it completly (apple), and

 application in Rust too, not just C++, it is an inferior design


 https://www.theregister.com/2024/01/31/microsoft_seeks_rust_developers/


A number of us do not agree with you, and I for one never will. For many
situations, exceptions are by far the best error handling mechanism that
there is. D's implementation of it could certainly use some improvements
(e.g. while being able to have a hierarchy for exceptions is valuable, the
fact that they're classes and thus have to be heap allocated is a problem
for code that needs to avoid the GC), so it would be great if D's
implementation for exceptions could be improved, but the basic concept is
solid and makes a lot of code cleaner as a result (e.g. I'd never want to
write a parser that didn't use exceptions; being able to separate the error
handling code from the actual parsing makes the code _far_ cleaner and far
less error-prone than any other error handling mechanism that I've ever
encountered).

Obviously, there are situations where exceptions are not appropriate (e.g.
it was definitely a mistake to have Phobos decoding Unicode all of the place
and potentially throw a UTFException from practically anywhere as a result),
and any properly written library is going to have to be intelligent about
when any particular error handling mechanism is used, but exceptions are
far, far too useful to not use - especially when the main alternative is
checking return values all over the place. So, while there are places that
Phobos currently uses exceptions where it shouldn't, there are others where
using them is very much the right thing to do IMHO, and I would use them
again in the next version of Phobos (and argue strongly against anyone
trying to not use them there), because if I didn't, the result would be
worse code that was harder to use.

In general, exceptions are best when

1. It's cleaner to assume that an operation will succeed, and it's
reasonable to assume that it will succeed. Parsing is a great example of
this. You're not going to do something like validate that an XML document is
going to parse correctly and then parse it, since that would basically mean
parsing it twice. You're just going to parse it and then report when that
fails so that the caller can handle it. And exceptions make that very clean,
because then you only have to have the one place at the top that catches the
exception, and the rest of the code can just do its thing. The way that
exceptions bubble up, allowing code along the way to completely ignore the
error handling and then have somewhere higher up the stack that is actually
in a position to handle the error condition catch the exception and handle
it is an amazing feature.

2. It's also best to use exceptions when you can't guarantee that an
operation will succeed but where it's reasonable to assume that it will -
particularly when it's actually impossible to check that an operation will
succeed beforehand. A great example of this would be reading a file. It's
good practice to check that the file exists first, but even then, you can't
guarantee that opening it will succeed, because the file could be removed
between the time you check and the time you go to open it (or a variety of
other errors could occur which cause opening or reading the file to fail).
The same goes for most file operations really. Any of them could fail, and
there needs to be a way to report that, and it clutters up the code
considerably if you have to deal with checking return values all over the
place - on top of the fact that it makes it impossible to simply return an
object or buffer from a function and use it if you also have to check an
error condition. You're stuck either returning the object via one of the
parameters or using a compound type where you have to check if the operation
succeeded and then extract the actual return value from it, which absolutely
destroys your ability to chain function calls - on top of making it far more
likely that code will fail to check the return value like it should,
resulting in it assuming that the call succeeded when it didn't.

3. For constructors, exceptions are really your only option unless you want
to get into doing two-part initialization, which is known to be error-prone,
and the advice I've almost always seen is to avoid it like the plague. So,
if a constructor needs to validate its arguments - or report any other error
condition that might occur while it's being constructed - an exception is
going to be the way to do it unless you want to do something like add a
member function to the type just so that you can check whether it was
constructed correctly, and at that point, you might as well just be doing
two-part initialization.

4. And exceptions are an excellent choice in any situation where you need to
be sure that an error condition is not ignored - or at least put the code in
a position where either the exception is caught and handled in whatever
manner is appropriate, or the program is killed, because the error condition
wasn't handled. Obviously, not all error conditions fall into that category,
but there are plenty of situations where error conditions must be handled
for the application to operate properly, and that's much more likely to
happen with an exception than with any other error reporting mechanism that
D has. Some other languages force that by forcing you to check return
values, but that can be incredibly annoying, and it results in far more
verbose code, because you have to put error-handling code everywhere instead
of just letting the exception bubble up to the code that can actually handle
it properly.

Now, obviously, there are situations where exceptions aren't appropriate. An
obvious one is where an error is likely, because using an exception in that
kind of situation is going to be inefficient in comparison to other options,
and if the error path is that likely, then trying to separate out the error
handling code like you do with try-catch statements really doesn't make
sense anyway.

And of course there are situations where validating stuff ahead of time is
perfectly reasonable and allows the code beyond that to just assume that
everything works without needing to report error conditions at all, because
the validation was already done. Unicode decoding is one such case where
that often makes sense. If you validate the Unicode when you read in a file,
then the rest of the code doesn't have to. Phobos, unfortunately, currently
picks the worst of both worlds, because readText checks the Unicode, and
then you have auto-decoding all over the place which validates the text
again and throws an exception if it's invalid. So, we end up with a bunch of
code that can't be nothrow or  nogc just because of a potential UTFException
which can't possibly be thrown if the text was already validated.

And of course there are also situations where it's actually reasonable to
ignore an error, in which case, throwing an exception wouldn't make sense -
but no other reporting mechanism would either. Unicode can actually be
handled in such a fashion in some cases, because it has the replacement
character so that you can use it anywhere that you encounter an invalid code
point, which gives text processing a way to process text without worrying
about invalid Unicode. That's obviously not a good choice in all cases, but
without auto-decoding, that choice is left up to the programmer as it should
be.

And with D, there are situations where exceptions can't be used where they
could be in a language like C++, because D's exceptions are allocated on the
heap, which not only means allocating memory, but it typically means using
the GC, which is fine for most code but not fine for all code. So, anyone in
that kind of situation is going to need to avoid exceptions even if they
would otherwise be the best choice.

And naturally, there are idiots who use exceptions for stupid things (e.g.
throwing an exception when you're checking a condition instead of just
returning bool, or having a type implement an interface and then throw an
exception if you call a function that it doesn't actualy support), which I
expect is part of the reason that some folks dislike exceptions. So, there's
no question that exceptions can be a problem, but overall, in the hands of
someone who knows what they're doing, they can be an excellent tool and
result in much cleaner, less error-prone APIs.

Obviously, you're free to hate exceptions for whatever reasons you may have,
and we're obviously not going to use them for everything in Phobos, because
that would be dumb, but given that there are cases where code is clearly
cleaner and easier to use correctly when exceptions are used, you're not
going to find general agreement with the idea that exceptions are inherently
bad. I expect that for most people, the disagreement is going to be on which
cases exactly they make sense for, and which they don't, not whether they're
a bad idea in general.

And personally, I wouldn't use D if it didn't have exceptions. It would be
far too miserable to not have them. So, while D's exceptions could certainly
use some improvement, I never would have started using D if it hadn't
supported exceptions, and if Walter seriously tried to get rid of them, I'd
go find another language to use instead. The exact form exceptions in D take
may change at some point in the future, but their basic functionality is
IMHO critical to writing good APIs, and part of the reason that languages
like C are miserable to use is that they don't have a comparable feature.

- Jonathan M Davis
Feb 04
next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Sunday, 4 February 2024 at 12:35:02 UTC, Jonathan M Davis 
wrote:

I get it, you are interested in the propagation mechanic, it can 
be done with error handling code too, take a look at Swift/Zig, 
take a look at Go2's proposal, many have succesfully moved away 
from EH, I learnt from them

There is no reason to be stubborn about the way it is currently 
done in phobos, DMD was successfully built without exceptions, 
let's embrace and enhance this

My game+engine are also successfully built without EH, and i'd 
love to move away from the C way of checking errors, and i'm 
envious of how other people are doing it with their other 
languages
Feb 04
next sibling parent Sergey <kornburn yandex.ru> writes:
On Sunday, 4 February 2024 at 15:54:53 UTC, ryuukk_ wrote:

 On Sunday, 4 February 2024 at 12:35:02 UTC, Jonathan M Davis 
 i'd love to move away from the C way of checking errors, and 
 i'm envious of how other people are doing it with their other 
 languages


Why to be envious, and not just use this one for example 
https://tchaloupka.github.io/expected/expected.html ?
Feb 04
prev sibling parent aberba <karabutaworld gmail.com> writes:
On Sunday, 4 February 2024 at 15:54:53 UTC, ryuukk_ wrote:

 On Sunday, 4 February 2024 at 12:35:02 UTC, Jonathan M Davis 
 wrote:

 I get it, you are interested in the propagation mechanic, it 
 can be done with error handling code too, take a look at 
 Swift/Zig, take a look at Go2's proposal, many have succesfully 
 moved away from EH, I learnt from them

 There is no reason to be stubborn about the way it is currently 
 done in phobos, DMD was successfully built without exceptions, 
 let's embrace and enhance this


No a fun of checking errors by return. Such code looks more 
difficult to read than it should. So it possible but not ideal. 
These language have gotten very little use out there. Exception 
is proven and works.


 My game+engine are also successfully built without EH, and i'd 
 love to move away from the C way of checking errors, and i'm 
 envious of how other people are doing it with their other 
 languages
Feb 04
prev sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 4 February 2024 at 12:35:02 UTC, Jonathan M Davis 
wrote:

 On Sunday, February 4, 2024 3:13:39 AM MST ryuukk_ via 
 Digitalmars-d wrote:

 EH is evil, some platforms has banned it completly (apple), 

 application in Rust too, not just C++, it is an inferior design

 https://www.theregister.com/2024/01/31/microsoft_seeks_rust_developers/


 A number of us do not agree with you, and I for one never will. 
 For many situations, exceptions are by far the best error 
 handling mechanism that there is.


I personally find exceptions useful only when I want to jump many 
stack frames at once.

Overall I find them taxing because they add another possible 
'return' value to a function. Sometimes its documented, but 9 out 
of 10 times you'll find out at runtime.

Knowing all the possible error states of a function helps me 
enormously in avoiding surprises later on. While it might seem 
overwhelming at first, I find it liberating because the complete 
state space is right in front of me. I guess it is similar to the 
benefits of local reasoning.

Note I often write server programs. YMMV with e.g. batch programs.
Feb 04
parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Sun, Feb 04, 2024 at 09:10:20PM +0000, Sebastiaan Koppe via Digitalmars-d
wrote:
[...]

 I personally find exceptions useful only when I want to jump many
 stack frames at once.


But I thought exceptions were not intended to be used in that kind of
way. They're more like an abort condition to short-circuit the normal
flow of control. If a failure condition is an expected part of the
program, throwing an exception is probably not the best way of
implementing it.



 Overall I find them taxing because they add another possible 'return'
 value to a function. Sometimes its documented, but 9 out of 10 times
 you'll find out at runtime.


But isn't it enough to just have a blanket catch block in the dispatch
code to swallow any exceptions that downstream code might generate? You
don't have to know exactly what it is, you just catch it, abort the
offending task, and move on to the next one?



 Knowing all the possible error states of a function helps me
 enormously in avoiding surprises later on. While it might seem
 overwhelming at first, I find it liberating because the complete state
 space is right in front of me.  I guess it is similar to the benefits
 of local reasoning.
 
 Note I often write server programs. YMMV with e.g. batch programs.


There are certainly valid considerations about the predictability of
control flow. I.e., if you call a series of functions in a row and they
are all nothrow, you can rest assured that the control flow will reach
the end of the block, and not bail out on you in the middle.

Still, when a fatal error happens deep inside some low-level code and
there's nothing else you can do until you get back to the high-level
business logic, exceptions are very useful. To manually bubble the error
state all the way back up the call stack is just too onerous. It forces
all intermediate levels of code to be dependent on the exact error codes
/ sumtypes so that it can be properly propagated up to the level where
the code has enough context to react to it meaningfully. This increases
coupling between orthogonal code. Using exceptions avoids this spurious
coupling.


T

-- 
People tell me I'm stubborn, but I refuse to accept it!
Feb 04
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 4 February 2024 at 23:16:36 UTC, H. S. Teoh wrote:

 On Sun, Feb 04, 2024 at 09:10:20PM +0000, Sebastiaan Koppe via 
 Digitalmars-d wrote: [...]

 I personally find exceptions useful only when I want to jump 
 many stack frames at once.


 But I thought exceptions were not intended to be used in that 
 kind of way. They're more like an abort condition to 
 short-circuit the normal flow of control. If a failure 
 condition is an expected part of the program, throwing an 
 exception is probably not the best way of implementing it.


Sorry, I didn't phrase it correctly. You are right, exceptions 
obviously aren't for control flow.

I meant that I find them very useful when I'm in the bowels of a 
library and need to bubble up an error to the original caller. 
With exceptions you can do this without involving the middle 
layer; just throw and let someone else care about it.

This is very convenient. However, that is about the only thing 
that they have going for me.


 Overall I find them taxing because they add another possible 
 'return' value to a function. Sometimes its documented, but 9 
 out of 10 times you'll find out at runtime.


 But isn't it enough to just have a blanket catch block in the 
 dispatch code to swallow any exceptions that downstream code 
 might generate? You don't have to know exactly what it is, you 
 just catch it, abort the offending task, and move on to the 
 next one?


Sometimes that can work yes.

I find that when I'm writing server software however, I often 
want to handle a subset of the possible failures. That might 
involve a blanket retry, a retry with tweaked parameters, a 
possible fallback, etc.

Using algebraic datatypes has always been a lot more explicit for 
me.


 Note I often write server programs. YMMV with e.g. batch 
 programs.


 There are certainly valid considerations about the 
 predictability of control flow. I.e., if you call a series of 
 functions in a row and they are all nothrow, you can rest 
 assured that the control flow will reach the end of the block, 
 and not bail out on you in the middle.


Yes, exactly. I value that a lot.


 Still, when a fatal error happens deep inside some low-level 
 code and there's nothing else you can do until you get back to 
 the high-level business logic, exceptions are very useful. To 
 manually bubble the error state all the way back up the call 
 stack is just too onerous. It forces all intermediate levels of 
 code to be dependent on the exact error codes / sumtypes so 
 that it can be properly propagated up to the level where the 
 code has enough context to react to it meaningfully. This 
 increases coupling between orthogonal code. Using exceptions 
 avoids this spurious coupling.


With exceptions there is coupling as well. In fact, I would argue 
the coupling is worse since its over a wider distance. Plus you 
have to 'discover' them over time.

It is true that it can sometimes be a bit onerous. In practice 
its only in a few rare cases, so I do it and move on. The 
possible set of failures is pretty static so its not like you 
have to extend the set all the time.

---

To conclude, I guess I consider exceptions to be very 
unstructured. Not only do they rarely contain the right 
information in order to decide the next step (unless you want to 
parse strings), because it is an open hierarchy, basically 
anything goes and you have no idea what gets thrown at you.

Not to mention that a refactor might change or add an exception, 
and there is no way you get informed about that during 
compilation.
Feb 05
prev sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Sun, Feb 04, 2024 at 05:35:02AM -0700, Jonathan M Davis via Digitalmars-d
wrote:

 On Sunday, February 4, 2024 3:13:39 AM MST ryuukk_ via Digitalmars-d wrote:

 EH is evil, some platforms has banned it completly (apple), and

 in Rust too, not just C++, it is an inferior design




[...]

 A number of us do not agree with you, and I for one never will. For
 many situations, exceptions are by far the best error handling
 mechanism that there is.


Exceptions are like the GC: they free your APIs from being littered with
error-handling paraphrenalia that crowd out the actual business logic.
Just like not having the GC forces your code and APIs to be crowded with
memory-management paraphrenalia. Both memory management and
error-handling are secondary concerns; the primary concern is actually
making progress in your problem domain: doing what the code set out to
achieve in the first place. As such, code should be focused as much as
possible on the business logic rather than have secondary concerns
occupy most of the space. Ideally, the main code should *only* express
the business logic; memory management and error-handling should be done
in a footnote rather than in the main body of the code.

Try/catch blocks let you do this to a large extent; whereas having to
manually check error codes or unwrap error sumtypes just adds excessive
verbosity to the primary logic, which makes code more tedious to write,
harder to read, and less maintainable as a result.

I suspect, though, that most objections to EH are directed primarily at
the specific implementation of it using libunwind like in C++, rather
than the concept itself of EH. That part I agree with: the way EH is
implemented could be done better. But as far as the concept is concerned
it's fine, and trying to get rid of it is just silly.


T

-- 
The easy way is the wrong way, and the hard way is the stupid way. Pick one.
Feb 04
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 4 February 2024 at 09:30:58 UTC, Jonathan M Davis 
wrote:

 The main thing that it needs IMHO is going through it and 
 making sure that it's doing the right thing for each trait and 
 that the exact set of traits that it has is really what we 
 want, since right now, several of them are doing the wrong 
 thing (e.g. treating enum as if it were its base type with 
 stuff like isIntegral, isNumeric, etc., which is a source of 
 bugs).


I have bad news for you. This behavior is not the fault of 
std.traits, but is actually baked into the language itself. For 
example, the spec for __traits(isIntegral) says:


 The integral types are: byte, ubyte, short, ushort, int, uint, 
 long, ulong, cent, ucent, bool, char, wchar, dchar, vectors of 
 integral types, and enums with an integral base type.


https://dlang.org/spec/traits.html#isIntegral

Maybe we can clean this up in a new language edition, but as long 
as this is how the underlying __traits work, std.traits should 
work the same way.
Feb 04
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, February 4, 2024 8:04:00 AM MST Paul Backus via Digitalmars-d 
wrote:

 On Sunday, 4 February 2024 at 09:30:58 UTC, Jonathan M Davis

 wrote:

 The main thing that it needs IMHO is going through it and
 making sure that it's doing the right thing for each trait and
 that the exact set of traits that it has is really what we
 want, since right now, several of them are doing the wrong
 thing (e.g. treating enum as if it were its base type with
 stuff like isIntegral, isNumeric, etc., which is a source of
 bugs).


 I have bad news for you. This behavior is not the fault of
 std.traits, but is actually baked into the language itself. For

 example, the spec for __traits(isIntegral) says:

 The integral types are: byte, ubyte, short, ushort, int, uint,
 long, ulong, cent, ucent, bool, char, wchar, dchar, vectors of
 integral types, and enums with an integral base type.


 https://dlang.org/spec/traits.html#isIntegral

 Maybe we can clean this up in a new language edition, but as long
 as this is how the underlying __traits work, std.traits should
 work the same way.


std.traits doesn't actually rely on that. It does its own checks for enum,
and it does so inconsistently (e.g. isSomeString doesn't allow enums, and it
was a source of bugs and frustration when it did years ago, because it meant
that range-based code was accepting enums with a base type of string and
then not working, because they aren't actually ranges). Traits and template
constraints in general need to be checking for either an exact list of types
or for whether the provided type has a particular set of capabilities.
Allowing implicit conversions is almost always a source of bugs. So, while
it should obviously be possible to check for implicit conversions where
appropriate, IMHO, it's a big mistake for Phobos to encourage it or make it
the easy the default. It should not be necessary to put !is(T == enum) all
over the place in code to stop enums or alias this from making it into a
function and then not behaving properly, because the type wasn't actually
converted, and the fact that std.traits is not principled about that is one
of its biggest problems IMHO - though as with a number of things that need
to be fixed in Phobos, they're mistakes that really only become clear after
more experience with what we attempted the first time around.

So, regardless of what __traits is doing, I will argue quite strongly that
std.traits should not be treating anything as its base type unless that's
truly what makes sense for that particular trait, and if __traits isn't
doing that, then it should probably fixed in a future edition as well. But
ultimately, __traits is a building block for other traits, so what
std.traits does does not have to be the same thing, though having the same
name in __traits and std.traits do something different from each other is
something that should probably be avoided. So, I agree that having
__traits(isIntegral, T) accept enums and std.traits.isIntegral reject them
is not a good idea. But my solution at that point is to not call the trait
in std.traits isIntegral but rather given a distinct name.

- Jonathan M Davis
Feb 04
prev sibling parent Meta <jared771 gmail.com> writes:
On Sunday, 4 February 2024 at 09:30:58 UTC, Jonathan M Davis 
wrote:

 The advantage of that is that the result is cleaner code (since 
 __traits tends to be pretty ugly), but it also means more 
 template instantiations. So, I expect that there is going to be 
 some disagreement on what we should be doing there


Man this is such an easily-fixable problem. I wish Walter weren't 
so stubborn about having macros in D. All it would take is 
something like:

```
macro template isNumeric(T) {
     __traits(isNumeric, T)
}

enum isStringNumeric = isNumeric!String;
assert(!isStringNumeric);
```

No symbol template instantiation or symbol generation necessary. 
It's like mixin templates but in reverse. Constrain macro 
templates to only be allowed to expand to an expression or 
something. It completely fixes any issues with std.traits, and is 
probably useful in a lot more areas as well.
Feb 04
prev sibling next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:


 Second, this design implies that the '2' in `std2.` is a 
 version specifier that would be incremented with each Phobos 
 release. This was not the intention and we agreed that it would 
 be confusing. I proposed using `sys.` as the root name for 
 Phobos 3 and Walter found that acceptable. We briefly discussed 
 splitting up Phobos into multiple roots and no firm agreement 
 was reached.


And then subsequent versions get put in a package with a version 
number under that?


 The other major topic of discussion was what I've been calling 
 the "Crippled by Default" design of editions, where the oldest 
 edition (technically the last pre-edition release) is the 
 default edition if no edition is specified. This poses a few 
 challenges from an end-user standpoint


Which ones?


 , but the argument that ended up resonating was the idea that 
 in engineering we always want to make the "right" way the 
 default or easiest way to do something, and then provide escape 
 hatches where necessary.


And the right way should be to not break existing code, even if 
they upgrade the compiler.


 Therefore, the compiler should default to use the latest version


The latest version of Phobos? Of the language?


 and then provide the ability via a switch to set the edition, 
 or lack thereof, of the modules an import path.


I think this option is useful for trying to upgrade a project to 
a new edition, but I'm not sure how it applies here.


 This solves the problem of abandon-ware packages being 
 accessible


Abandon-ware Phobos packages?


 without presenting the new user with an ever more decayed 
 version of the compiler.


I don't understand what this means.


 We want to put out best foot forward and presenting the last 
 pre-editions release, which is constantly getting old as time 
 passes, does not do that.


It's unclear to me how we'd be doing that if the new Phobos 
version(s) is namespaced differently.


 When then moved on to a conversation about how Walter envisions 
 editions actually working. Since none of have seen the document 
 that Atila is working on, Walter shared his opinions on how it 
 should work. Essentially, Walter would like to see a "hybrid" 
 approach having edition attributes for specific experimental 
 features, and then having a yearly "roll-up" edition that 
 includes all the promoted features from the prior year. So if 
 DIP1000 gets promoted to Edition 2025, then DIP1000 would be 
 active by default in that edition and all subsequent editions 
 without having to specifically enable it.


I'm not sure every year is a good time interval.


 I did bring up that this was likely to cause another "function 
 attribute soup" problem


How?


 but in general I wholeheartedly agree with the idea that 

 something similar so it would be conceptually comfortable to 
 users coming from those languages. Atila, if you're reading 
 this, this is what Walter was thinking/hoping would appear,


The reason for that is because I told him that's how I was 
thinking of doing it ;)


 After that we had a discussion about how to distribute Phobos. 
 This mostly centered on what release cadence to use. I argued 
 for linking the Phobos version to the edition release schedule. 
 I think this is sensible and makes it easier for people to 
 reason about which compiler/library pairing they are using. 
 Walter was fine with that, but he does not want to use the 
 "Edition" language to describe Phobos releases.


This is the part I'm really not clear about yet.


 I think this makes sense as Phobos doesn't really have editions,


Unless it opts in to a new one. And it should, because we want to 
lead by example.


 Finally, we touched briefly on the major changes we would like 
 to see in Phobos 3 and these are the major changes we are 
 committing to for Phobos 3 so far:
 - Promoting allocators out of experimental.


This requires solving a number of thorny issues, including "does 
this API even make sense?", which I'm told Paul Backus is working 
on.


 - Range interface redesign (see JMD's thread 
 [here](https://forum.dlang.org/thread/mailman.588.1705813271.3719.digitalmars-d puremagic.com)).
 - Fix std.traits.


What does "fix" mean in this context?


 The above list is not exhaustive and we are open to further 
 suggestions.


- XML
- JSON
- YAML
- SDL?
- Channels
- ...
Feb 05
prev sibling parent reply Martyn <martyn.developer googlemail.com> writes:
On Friday, 2 February 2024 at 09:09:37 UTC, Adam Wilson wrote:

 Walter and I had a productive conversation yesterday about 
 Phobos 3 and we felt it would be appropriate to share some 
 notes on our discussion.

 Snip...

 Finally, we touched briefly on the major changes we would like 
 to see in Phobos 3 and these are the major changes we are 
 committing to for Phobos 3 so far:
 - Removal of Autodecoding.
 - Promoting allocators out of experimental.
 - Range interface redesign (see JMD's thread 
 [here](https://forum.dlang.org/thread/mailman.588.1705813271.3719.digitalmars-d puremagic.com)).
 - Fix std.traits.

 The above list is not exhaustive and we are open to further 
 suggestions. For myself, I would love to see Cryptography and 
 Stream API's make it into Phobos, but I am sure that the list 
 of what the community wants to add is legion so those could end 
 up being lower priority.

 Snip...

 If you would like to participate in the design discussion on 
 Phobos 3, I am currently hosting a repo on my GitHub with GH 
 Discussions for unresolved/undesigned topics, and PR's for 
 editing the actual design document. Link is here: 
 https://github.com/LightBender/PhobosV3-Design


Very interesting!

I really like the Allocator idea. Something I think will benefit 
D and is overdue if being honest.

Curious to know how this will work for D. Will Allocators be 
available for BetterC as well? I certainly hope so!

If so, I guess the **default** Allocator will be the GC one, and 
can still be disabled. Being able to change the default (or 
change locally like in a function or pass it as parameter) would 
provide a lot of flexibility.

It's just I see possiblities of BetterC being able to use certain 
features that are not available, like dynamic or associative 
arrays. To be designed for allowing us to specify the Allocator 
to use (or default if not specified at all) and it doesn't need 
to be a GC one gives the programmer a lot of control for low 
level as well as high.
Feb 06
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 6 February 2024 at 14:41:27 UTC, Martyn wrote:

 Curious to know how this will work for D. Will Allocators be 
 available for BetterC as well? I certainly hope so!


Nothing's set in stone yet, but in the proposal I'm working on, 
there is nothing stopping allocators from being available in 
BetterC.


 If so, I guess the **default** Allocator will be the GC one, 
 and can still be disabled. Being able to change the default (or 
 change locally like in a function or pass it as parameter) 
 would provide a lot of flexibility.


In my proposal, when you use a library container (like an array, 
a hash table, a binary tree, etc.), you can specify what type of 
allocator you want to use as a template parameter (as in, 
`Array!(int, GC)`). It will probably default to the GC, but you 
can just as easily use malloc, or even a custom allocator that 
you write yourself.

I'm not planning to include a global default allocator. Built-in 
language features like `new`, dynamic arrays, associative arrays, 
and delegate contexts will always use the GC, and there will not 
be an option to change this.
Feb 06
next sibling parent Martyn <martyn.developer googlemail.com> writes:
On Wednesday, 7 February 2024 at 05:55:04 UTC, Paul Backus wrote:

 On Tuesday, 6 February 2024 at 14:41:27 UTC, Martyn wrote:

 
 ... Snip


 Nothing's set in stone yet, but in the proposal I'm working on, 
 there is nothing stopping allocators from being available in 
 BetterC.


 ... Snip


 In my proposal, when you use a library container (like an 
 array, a hash table, a binary tree, etc.), you can specify what 
 type of allocator you want to use as a template parameter (as 
 in, `Array!(int, GC)`). It will probably default to the GC, but 
 you can just as easily use malloc, or even a custom allocator 
 that you write yourself.

 I'm not planning to include a global default allocator. 
 Built-in language features like `new`, dynamic arrays, 
 associative arrays, and delegate contexts will always use the 
 GC, and there will not be an option to change this.


Thank you for taking the time to respond. I understand a proposal 
is working on, but your answers above sound very promising. 
Thanks again.
Feb 07
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 7 February 2024 at 05:55:04 UTC, Paul Backus wrote:

 On Tuesday, 6 February 2024 at 14:41:27 UTC, Martyn wrote:

 Curious to know how this will work for D. Will Allocators be 
 available for BetterC as well? I certainly hope so!


 Nothing's set in stone yet, but in the proposal I'm working on, 
 there is nothing stopping allocators from being available in 
 BetterC.


 If so, I guess the **default** Allocator will be the GC one, 
 and can still be disabled. Being able to change the default 
 (or change locally like in a function or pass it as parameter) 
 would provide a lot of flexibility.


 In my proposal, when you use a library container (like an 
 array, a hash table, a binary tree, etc.), you can specify what 
 type of allocator you want to use as a template parameter (as 
 in, `Array!(int, GC)`). It will probably default to the GC, but 
 you can just as easily use malloc, or even a custom allocator 
 that you write yourself.

 I'm not planning to include a global default allocator. 
 Built-in language features like `new`, dynamic arrays, 
 associative arrays, and delegate contexts will always use the 
 GC, and there will not be an option to change this.


The problem with this approach, as C++ found out, is that 
`Vector!(int, MyAlloc)` is a different type from `Vector!(int, 
YourAlloc)`. The way I got around that is by defaulting to a 
global allocator. This has its drawbacks as well of course, 
because now everything is a virtual call.
Feb 07
next sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
On Wednesday, 7 February 2024 at 10:10:27 UTC, Atila Neves wrote:

 On Wednesday, 7 February 2024 at 05:55:04 UTC, Paul Backus 
 wrote:

 On Tuesday, 6 February 2024 at 14:41:27 UTC, Martyn wrote:

 Curious to know how this will work for D. Will Allocators be 
 available for BetterC as well? I certainly hope so!


 Nothing's set in stone yet, but in the proposal I'm working 
 on, there is nothing stopping allocators from being available 
 in BetterC.


 If so, I guess the **default** Allocator will be the GC one, 
 and can still be disabled. Being able to change the default 
 (or change locally like in a function or pass it as 
 parameter) would provide a lot of flexibility.


 In my proposal, when you use a library container (like an 
 array, a hash table, a binary tree, etc.), you can specify 
 what type of allocator you want to use as a template parameter 
 (as in, `Array!(int, GC)`). It will probably default to the 
 GC, but you can just as easily use malloc, or even a custom 
 allocator that you write yourself.

 I'm not planning to include a global default allocator. 
 Built-in language features like `new`, dynamic arrays, 
 associative arrays, and delegate contexts will always use the 
 GC, and there will not be an option to change this.


 The problem with this approach, as C++ found out, is that 
 `Vector!(int, MyAlloc)` is a different type from `Vector!(int, 
 YourAlloc)`. The way I got around that is by defaulting to a 
 global allocator. This has its drawbacks as well of course, 
 because now everything is a virtual call.


I do that as well

```D
struct Array(T)
{
     T[] items;
     Allocator allocator;
     size_t count = 0;
```


Just store the allocator in the struct, it's no big deal, granted 
your allocator type is a simple and compact struct

```D
struct Allocator
{
     void* ptr;
     AllocatorFN* fn;
}
struct AllocatorFN
{
     alloc_d alloc;
     resize_d resize;
     free_d free;
}

// libc
enum c_allocator = Allocator(null, &CAllocator.fn);
struct CAllocator
{
     __gshared AllocatorFN fn = {
         alloc: &alloc_impl,
         resize: &resize_impl,
         free: &free_impl
     };
}
```

That's it

```D

Allocator heap = c_allocator;

auto entities = Array!(Entity).create(heap);

```
Feb 07
prev sibling next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 07/02/2024 11:10 PM, Atila Neves wrote:

 On Wednesday, 7 February 2024 at 05:55:04 UTC, Paul Backus wrote:

 On Tuesday, 6 February 2024 at 14:41:27 UTC, Martyn wrote:

 Curious to know how this will work for D. Will Allocators be 
 available for BetterC as well? I certainly hope so!


 Nothing's set in stone yet, but in the proposal I'm working on, there 
 is nothing stopping allocators from being available in BetterC.


 If so, I guess the **default** Allocator will be the GC one, and can 
 still be disabled. Being able to change the default (or change 
 locally like in a function or pass it as parameter) would provide a 
 lot of flexibility.


 In my proposal, when you use a library container (like an array, a 
 hash table, a binary tree, etc.), you can specify what type of 
 allocator you want to use as a template parameter (as in, `Array!(int, 
 GC)`). It will probably default to the GC, but you can just as easily 
 use malloc, or even a custom allocator that you write yourself.

 I'm not planning to include a global default allocator. Built-in 
 language features like `new`, dynamic arrays, associative arrays, and 
 delegate contexts will always use the GC, and there will not be an 
 option to change this.


 
 The problem with this approach, as C++ found out, is that `Vector!(int, 
 MyAlloc)` is a different type from `Vector!(int, YourAlloc)`. The way I 
 got around that is by defaulting to a global allocator. This has its 
 drawbacks as well of course, because now everything is a virtual call.


I came to a similar conclusion from my usage of std.experimental.allocator.

My stuff doesn't use template parameters for things like memory allocators.

My conclusion was that you really only have two use cases for composable 
allocators:

1. You know about memory patterns, locking, type sizes ext. Use 
composable directly.
2. Otherwise, use virtual.

There doesn't seem to be anything in between.
Feb 07
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 7 February 2024 at 11:46:11 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 My conclusion was that you really only have two use cases for 
 composable allocators:

 1. You know about memory patterns, locking, type sizes ext. Use 
 composable directly.
 2. Otherwise, use virtual.

 There doesn't seem to be anything in between.


The relevant constraint here isn't composability, it's safety.

In order to make allocators  safe (which is the goal of my 
proposal), you have to guarantee somehow that memory blocks can 
only be deallocated by the same allocator that originally 
allocated them.

It is possible to do this with a polymorphic allocator, but it 
incurs an enormous amount of runtime overhead, which rules out 
making it the default approach.

Of course, if you don't care about  safe, you can ignore all of 
this and use a much simpler design. Personally, I think Phobos V3 
should support  safe as much as possible, but ultimately that 
choice is up to project leadership.
Feb 07
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, February 7, 2024 9:38:38 AM MST Paul Backus via Digitalmars-d 
wrote:

 On Wednesday, 7 February 2024 at 11:46:11 UTC, Richard (Rikki)

 Andrew Cattermole wrote:

 My conclusion was that you really only have two use cases for
 composable allocators:

 1. You know about memory patterns, locking, type sizes ext. Use
 composable directly.
 2. Otherwise, use virtual.

 There doesn't seem to be anything in between.


 The relevant constraint here isn't composability, it's safety.

 In order to make allocators  safe (which is the goal of my
 proposal), you have to guarantee somehow that memory blocks can
 only be deallocated by the same allocator that originally
 allocated them.

 It is possible to do this with a polymorphic allocator, but it
 incurs an enormous amount of runtime overhead, which rules out
 making it the default approach.

 Of course, if you don't care about  safe, you can ignore all of
 this and use a much simpler design. Personally, I think Phobos V3
 should support  safe as much as possible, but ultimately that
 choice is up to project leadership.


Allocators should be  safe where they can be, but I think that a number of
us came to the conclusion a while ago that they couldn't be  safe in the
general case. So, it'll be interesting to see what you can come up with, but
if  safe is a requirement, I would expect that certain kinds of allocators
simply won't work. But that's no reason not to support  safe as much as
possible (particularly when code is templated).

But with regards to virtual vs templated, we don't necessarily have to
decide in that we could design allocators so that they can be used with
either approach. The templated ones would be more flexible with regards to
attributes and could be better optimized, whereas there could then be
virtual ones that wrapped those in classes but put harder requirements on
the attributes (be it by requiring more or allowing fewer), because they're
then supposed to work with most or all of the various allocator types. Then
both approaches would be available.

That being said, I would expect most D code to go with the templated
approach. It's what D typically does, and the kind of folks who want to use
allocators are usually also the kind of folks who aren't going to be very
happy about passing classes around.

And based on both what has historically been done with containers in D and
the few discussions that we've had on it recently, I expect that they're
very much going to be taking the Design by Introspection approach and be
templated based on their capabilities (of which the allocator would be only
one, though one of the possible approaches there is for one of the options
be to choose between a statically chosen alocator and a dynamically chosen
one).

- Jonathan M Davis
Feb 07
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 08/02/2024 11:00 AM, Jonathan M Davis wrote:

 That being said, I would expect most D code to go with the templated
 approach. It's what D typically does, and the kind of folks who want to use
 allocators are usually also the kind of folks who aren't going to be very
 happy about passing classes around.


I do not.

There isn't enough of a win to introducing allocators, unless you have 
knowledge of internals.

At which point you don't need to template your container upon the 
composable allocator. You'll have a dedicated one internally that isn't 
virtual.

This is one of my key take aways, don't pretend you need to customize 
the memory allocator to its usage if you don't know how it would be 
used. Go slower, go virtual, it'll keep you sane and allow you to build 
cool things.

Remember to profile. Not guess.
Feb 07
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 7 February 2024 at 22:00:49 UTC, Jonathan M Davis 
wrote:

 Allocators should be  safe where they can be, but I think that 
 a number of us came to the conclusion a while ago that they 
 couldn't be  safe in the general case. So, it'll be interesting 
 to see what you can come up with, but if  safe is a 
 requirement, I would expect that certain kinds of allocators 
 simply won't work. But that's no reason not to support  safe as 
 much as possible (particularly when code is templated).


With -preview=dip1000 and -preview=systemVariables, it is 
possible to make pretty much any allocator  safe--although for 
some of them, the safety checking adds a little bit of runtime 
overhead.

The main obstacle is that both of these -preview features are 
still very much unfinished, and will need a lot of work before 
they are ready to be used in real library code.

By the way, one of the key techniques that makes this work is 
having the allocators wrap each void[] they return in a 
non-copyable struct, so that it can only be deallocated once. So 
you can probably understand why I'm so concerned about support 
for non-copyable types in Phobos v3. :)


 But with regards to virtual vs templated, we don't necessarily 
 have to decide in that we could design allocators so that they 
 can be used with either approach.


Yes, that's the idea.


 And based on both what has historically been done with 
 containers in D and the few discussions that we've had on it 
 recently, I expect that they're very much going to be taking 
 the Design by Introspection approach and be templated based on 
 their capabilities (of which the allocator would be only one, 
 though one of the possible approaches there is for one of the 
 options be to choose between a statically chosen alocator and a 
 dynamically chosen one).


This is also how I'd expect containers to work.
Feb 07
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, February 7, 2024 10:12:06 PM MST Paul Backus via Digitalmars-d 
wrote:

 On Wednesday, 7 February 2024 at 22:00:49 UTC, Jonathan M Davis

 wrote:

 Allocators should be  safe where they can be, but I think that
 a number of us came to the conclusion a while ago that they
 couldn't be  safe in the general case. So, it'll be interesting
 to see what you can come up with, but if  safe is a
 requirement, I would expect that certain kinds of allocators
 simply won't work. But that's no reason not to support  safe as
 much as possible (particularly when code is templated).


 With -preview=dip1000 and -preview=systemVariables, it is
 possible to make pretty much any allocator  safe--although for
 some of them, the safety checking adds a little bit of runtime
 overhead.

 The main obstacle is that both of these -preview features are
 still very much unfinished, and will need a lot of work before
 they are ready to be used in real library code.

 By the way, one of the key techniques that makes this work is
 having the allocators wrap each void[] they return in a
 non-copyable struct, so that it can only be deallocated once. So
 you can probably understand why I'm so concerned about support
 for non-copyable types in Phobos v3. :)


Well, regardless of what we end up doing iwth non-copyable types in Phobos,
it's clear that the language needs improvements with regards to moving
objects, which we discussed in one of the recent DLF planning meetings. DIP
1048 (move constructors) was approved ages ago, but it still hasn't been
implemented yet, and to really make moving objects around even sort of
user-friendly, we need the compiler to start doing things like do a move
instead of a copy when it's the last use of a variable. And in particular,
if we decide that making basic input ranges non-copyable is the best
approach, then that's going to put pressure on getting that sorted out. The
range situation in Phobos v3 is still very much in the air though.

- Jonathan M Davis
Feb 08
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 8 February 2024 at 05:12:06 UTC, Paul Backus wrote:

 On Wednesday, 7 February 2024 at 22:00:49 UTC, Jonathan M Davis 
 wrote:

 [...]


 With -preview=dip1000 and -preview=systemVariables, it is 
 possible to make pretty much any allocator  safe--although for 
 some of them, the safety checking adds a little bit of runtime 
 overhead.

 The main obstacle is that both of these -preview features are 
 still very much unfinished, and will need a lot of work before 
 they are ready to be used in real library code.

 By the way, one of the key techniques that makes this work is 
 having the allocators wrap each void[] they return in a 
 non-copyable struct, so that it can only be deallocated once. 
 So you can probably understand why I'm so concerned about 
 support for non-copyable types in Phobos v3. :)


What specifically would we need support for?
Feb 12
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 12 February 2024 at 17:07:21 UTC, Atila Neves wrote:

 On Thursday, 8 February 2024 at 05:12:06 UTC, Paul Backus wrote:

 By the way, one of the key techniques that makes this work is 
 having the allocators wrap each void[] they return in a 
 non-copyable struct, so that it can only be deallocated once. 
 So you can probably understand why I'm so concerned about 
 support for non-copyable types in Phobos v3. :)


 What specifically would we need support for?


It's not really any specific thing; it's more that Phobos's 
general attitude towards non-copyable types needs to change.

Currently, there are a huge number of interfaces in Phobos that, 
in principle, could work with non-copyable types, but just don't, 
because no one ever thought of it or cared enough. Such 
interfaces include:

- Large portions of std.range and std.algorithm
- std.format in its entirety
- std.typecons.Tuple
- std.array.Appender
- etc.

 From how widespread it is, it's clear that this is not an issue 
of specific bugs or design mistakes. It's a systemic problem in 
the way Phobos's code is written, tested, and reviewed.

The attitude that leads to this problem is exemplified by 
Jonathan M. Davis's comments in the "Range Redesign" thread [1], 
but I do not want to single him out. Every single one of us who 
writes and reviews code for Phobos, and allows PRs to be merged 
that fail to handle non-copyable types, is a contributor to the 
problem (including myself!).

If we're going to fix this, whoever's leading the development of 
Phobos V3 (you? Adam Wilson?) needs to make a policy-level 
decision that no PR gets merged unless it either supports 
non-copyable types, or has a good reason not to.

[1]: 
https://forum.dlang.org/post/mailman.618.1705964882.3719.digitalmars-d puremagic.com
Feb 12
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 12 February 2024 at 19:49:39 UTC, Paul Backus wrote:

 On Monday, 12 February 2024 at 17:07:21 UTC, Atila Neves wrote:

 On Thursday, 8 February 2024 at 05:12:06 UTC, Paul Backus 
 wrote:

 By the way, one of the key techniques that makes this work is 
 having the allocators wrap each void[] they return in a 
 non-copyable struct, so that it can only be deallocated once. 
 So you can probably understand why I'm so concerned about 
 support for non-copyable types in Phobos v3. :)


 What specifically would we need support for?


 It's not really any specific thing; it's more that Phobos's 
 general attitude towards non-copyable types needs to change.

 Currently, there are a huge number of interfaces in Phobos 
 that, in principle, could work with non-copyable types, but 
 just don't, because no one ever thought of it or cared enough. 
 Such interfaces include:

 - Large portions of std.range and std.algorithm
 - std.format in its entirety
 - std.typecons.Tuple
 - std.array.Appender
 - etc.

 From how widespread it is, it's clear that this is not an issue 
 of specific bugs or design mistakes. It's a systemic problem in 
 the way Phobos's code is written, tested, and reviewed.


I see. This is an instance of a more general problem of "nasty 
types" that do "weird" things such as being uncopyable. Ideally 
we'd have one in Phobos just for testing and feed it to basically 
everything. FeepingCreature mentioned something like to me at 
DConf but I can't find it in github after googling.
Feb 13
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 13 February 2024 at 14:08:24 UTC, Atila Neves wrote:

 I see. This is an instance of a more general problem of "nasty 
 types" that do "weird" things such as being uncopyable. Ideally 
 we'd have one in Phobos just for testing and feed it to 
 basically everything. FeepingCreature mentioned something like 
 to me at DConf but I can't find it in github after googling.


If you want some examples to crib from, check out the unittests 
in std.sumtype. I've done my best to put every weird type I can 
think of in there. :)
Feb 13
parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 13 February 2024 at 15:27:16 UTC, Paul Backus wrote:

 On Tuesday, 13 February 2024 at 14:08:24 UTC, Atila Neves wrote:

 I see. This is an instance of a more general problem of "nasty 
 types" that do "weird" things such as being uncopyable. 
 Ideally we'd have one in Phobos just for testing and feed it 
 to basically everything. FeepingCreature mentioned something 
 like to me at DConf but I can't find it in github after 
 googling.


 If you want some examples to crib from, check out the unittests 
 in std.sumtype. I've done my best to put every weird type I can 
 think of in there. :)


Nice! Now we "just" have to move them somewhere common (so that 
other Phobos modules don't depend on sumtype for no reason) and 
shove them down the rest of Phobos. Somehow.
Feb 13
prev sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
On Wednesday, 7 February 2024 at 22:00:49 UTC, Jonathan M Davis 
wrote:

 That being said, I would expect most D code to go with the 
 templated approach. It's what D typically does,


We are here, we said we don't like them templated



 and the kind of folks who want to use allocators are usually 
 also the kind of folks who aren't going to be very happy about 
 passing classes around.


"classes"? the kind of people who want to use allocators are not 
the kind of people to make them "classes" another red flag..

Passing them around? yes, if you care about using allocators, you 
won't mind doing this, besides, it's limited to the places you 
want to control your allocations, it's not a kind of type you 
pass around _everywhere_
Feb 08
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 7 February 2024 at 10:10:27 UTC, Atila Neves wrote:

 On Wednesday, 7 February 2024 at 05:55:04 UTC, Paul Backus 
 wrote:

 In my proposal, when you use a library container (like an 
 array, a hash table, a binary tree, etc.), you can specify 
 what type of allocator you want to use as a template parameter 
 (as in, `Array!(int, GC)`). It will probably default to the 
 GC, but you can just as easily use malloc, or even a custom 
 allocator that you write yourself.


 The problem with this approach, as C++ found out, is that 
 `Vector!(int, MyAlloc)` is a different type from `Vector!(int, 
 YourAlloc)`. The way I got around that is by defaulting to a 
 global allocator. This has its drawbacks as well of course, 
 because now everything is a virtual call.


Yes, I'm aware of this problem, but I don't see a way around it.

The thing is, the compiler has to know what allocator you're 
using at compile time in order to infer the correct function 
attributes for the container. If you want `Array!int` to default 
to using the GC, but also work in ` nogc` code when you're using 
`malloc`...how would that even work?

I have two potential mitigations for this problem in mind.

The first is to realize that D already has a solution for writing 
data-structure-agnostic code: ranges! If `Array` implements all 
of the relevant range interfaces, the rest of your code doesn't 
have to depend on any single concrete `Array` type--it can just 
depend on the range interface.

If you really need a concrete type, there's always 
`std.range.interfaces`, although that does come with some 
sacrifices (virtual calls, not ` safe` or ` nogc`, etc.).

The second is to have the allocator library provide a 
runtime-polymorphic allocator, which users can depend on if they 
absolutely need a single concrete container type for whatever 
reason. We'd have to make some serious tradeoffs between runtime 
overhead and attribute compatibility, and it's possible that no 
one-size-fits-all solution exists, but in principle, it can be 
done.

Finally, on the subject of global allocators: I don't think 
having a global allocator is a good idea, period, whether it's 
used as a default or not. Containers generally cannot tolerate 
having their allocators mutated "behind their backs," and a 
user-accessible global variable would make it very easy and 
tempting to write code that does this.
Feb 07
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 7 February 2024 at 16:20:31 UTC, Paul Backus wrote:

 On Wednesday, 7 February 2024 at 10:10:27 UTC, Atila Neves 
 wrote:

 [...]


 Yes, I'm aware of this problem, but I don't see a way around it.

 [...]


All good points, thanks. I like the "just use ranges" approach, 
algorithms really shouldn't have to care what allocation strategy 
is backing the range they just got, and this is an insight I 
won't soon forget.

Appending from a "different" vector will mean element copies, I 
guess. Or OOP. Trade offs. Trade offs everywhere.
Feb 07
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, February 7, 2024 9:54:49 AM MST Atila Neves via Digitalmars-d 
wrote:

 On Wednesday, 7 February 2024 at 16:20:31 UTC, Paul Backus wrote:

 On Wednesday, 7 February 2024 at 10:10:27 UTC, Atila Neves

 wrote:

 [...]


 Yes, I'm aware of this problem, but I don't see a way around it.

 [...]


 All good points, thanks. I like the "just use ranges" approach,
 algorithms really shouldn't have to care what allocation strategy
 is backing the range they just got, and this is an insight I
 won't soon forget.

 Appending from a "different" vector will mean element copies, I
 guess. Or OOP. Trade offs. Trade offs everywhere.


Well, stuff that actually has to operate on the container itself is likely
going to care about the exact container type (particularly things like
removing an element or a range of elements based on an
iterator/cursor/range), but algorithm code generally isn't going to care,
because it's all going to be templatized anyway, and it'll infer the
attributes as appropriate. While our container situation in Phobos needs
work, I wouldn't expect things to change much with regards to algorithms.
If you want to operate on the elements of a container, then you get a range
from it and then all of the algorithm stuff doesn't care what kind of
container you're dealing with.

But egardless of what exactly we do with allocators, I would fully expect
that we're going to end up with container types that are not just templated
on the element type but which are also templated on requirements that you've
placed on them (i.e. Design by Introspection), and that's clearly what
Robert is proposing with what he's discussed on the topic in the DLF
meetings and will be talking about in his dconf online talk. There will
probably be a default choice where you don't tell the container much of what
you want and just get the default (e.g. GC-allocated,  safe, etc.), but I
don't think that code in general is going to be passing around specific
container types. Most of the time, what you really want is to pass around
ranges - and when you do need to pass around an actual container, well,
that's probably specific to what you're doing in your code, and then you can
assume that you're using whatever it is you're using without templatizing
the code using it. But in general, I wouldn't expect a library like Phobos
to be passing containers around.

- Jonathan M Davis
Feb 07
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 7 February 2024 at 10:10:27 UTC, Atila Neves wrote:

 [snip]

 The problem with this approach, as C++ found out, is that 
 `Vector!(int, MyAlloc)` is a different type from `Vector!(int, 
 YourAlloc)`. The way I got around that is by defaulting to a 
 global allocator. This has its drawbacks as well of course, 
 because now everything is a virtual call.


If you need to allocate or deallocate, then it's useful to treat 
them as separate types. Otherwise it's not. Ideally the compiler 
(or maybe LTO?) would know when this leads to template bloat and 
fix it for you.

What about an `alias this` approach? For instance:

```d
struct BaseVector(T) {
     T* ptr;
     size_t length;
     // member functions that are common to all vectors
}

struct Vector(T, Alloc) {
     BaseVector!T baseVector;
     alias this = baseVector;
     // member functions to handle allocation and de-allocation
}
```

If you want to write functions that don't concern themselves with 
allocation, you can write them in terms of `BaseVector`. You 
avoid unnecessary template bloat because of implicit conversions. 
You can say that both `Vector!(int, MyAlloc)` and `Vector!(int, 
YourAlloc)` are both `BaseVector!int`s.

Is there a downside to this?
Feb 07
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 7 February 2024 at 21:26:51 UTC, jmh530 wrote:

 On Wednesday, 7 February 2024 at 10:10:27 UTC, Atila Neves 
 wrote:

 [snip]

 The problem with this approach, as C++ found out, is that 
 `Vector!(int, MyAlloc)` is a different type from `Vector!(int, 
 YourAlloc)`. The way I got around that is by defaulting to a 
 global allocator. This has its drawbacks as well of course, 
 because now everything is a virtual call.


 If you need to allocate or deallocate, then it's useful to 
 treat them as separate types. Otherwise it's not.


     alias MyVec = Vector(MyType, Mallocator);
     void fun(MyVec vec) {
          // just before the curly brace, the elements and storage 
for them get deallocated.
     }
Feb 12
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 12 February 2024 at 17:00:01 UTC, Atila Neves wrote:

 On Wednesday, 7 February 2024 at 21:26:51 UTC, jmh530 wrote:

 On Wednesday, 7 February 2024 at 10:10:27 UTC, Atila Neves 
 wrote:

 [snip]

 The problem with this approach, as C++ found out, is that 
 `Vector!(int, MyAlloc)` is a different type from 
 `Vector!(int, YourAlloc)`. The way I got around that is by 
 defaulting to a global allocator. This has its drawbacks as 
 well of course, because now everything is a virtual call.


 If you need to allocate or deallocate, then it's useful to 
 treat them as separate types. Otherwise it's not.


     alias MyVec = Vector(MyType, Mallocator);
     void fun(MyVec vec) {
          // just before the curly brace, the elements and 
 storage for them get deallocated.
     }


I assume you mean `alias MyVec = Vector!(MyType, Mallocator);`.

I'm not completely sure I'm getting to the heart of what you're 
saying since your post is a bit sparse...but I'm doing my best.

Your `fun` function only works for `MyVec`, which depends on 
`MyType` and `Mallocator`. It isn't general across either types 
or allocators. It's kind of the essence of your complaint about 
C++ and treating `Vector!(int, MyAlloc)` and `Vector!(int, 
YourAlloc)` as different types, except you are pushing all the 
work downstream to functions. In other words, you have to have 
separate `fun` for all the different types and allocators you 
pass, and then have different allocation strategies for each. 
Multiply that across all the other functions.

If all the allocators are consistent with the interface from 
`std.experimental.allocator`, then you might be able to make the 
allocation more general. But, it becomes frustrating to rely on 
template aliases. The compiler can't see through them due to 
issue 1807 [1]. The more general approach is to use template 
constraints (see example below). The problem with the template 
constraint approach is that it leads to template bloat, which is 
why I suggested the `alias this` approach above.

Another approach would be to pass the allocator to the function 
instead of making it part of the type. The problem with this is 
that for some allocators you need to know what did the allocating 
in order to free it safely.

```d
import std.stdio;

struct Vector(T, U) { T x; U y;}
alias MyVec = Vector!(double, int);
void fun(MyVec vec)
{
     writeln("works");
}
alias MyVec2(T) = Vector!(T, int);
void fun2(T)(MyVec2!T vec)
{
     writeln("works2");
}
void fun3(T)(T vec)
     if (is(T : Vector!(U, int), U))
{
     writeln("works3");
}

void main() {
     MyVec vec = MyVec(1.0, 1);
     fun(vec); // prints works
     //fun2(vec); //error
     fun2!double(vec); // prints works2
     fun3(vec); // prints works3
}
```

[1] https://issues.dlang.org/show_bug.cgi?id=1807
Feb 12
parent Atila Neves <atila.neves gmail.com> writes:
On Monday, 12 February 2024 at 18:13:45 UTC, jmh530 wrote:

 On Monday, 12 February 2024 at 17:00:01 UTC, Atila Neves wrote:

     [...]


 I assume you mean `alias MyVec = Vector!(MyType, Mallocator);`.

 I'm not completely sure I'm getting to the heart of what you're 
 saying since your post is a bit sparse...but I'm doing my best.

 [...]


My point is that things going out of scope can trigger 
deallocations. You're right that passing the allocator is 
error-prone. That's why automem, unlike C++'s smart pointers, 
doesn't let you construct the thing and pass it in; it's more 
"tell me what you want and I'll construct it for you".
Feb 13