www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - memory management and the standard library

reply Sjoerd Nijboer <dlang sjoerdnijboer.com> writes:
So I pop in every now and then to look how D is progressing, and 
the last time I concidered D was for a problem a friend of me had 
with a small game. In that game there was some colission 
detection that required a custom datastructure and then some 
operations on top of that, with a hard limit on time.

For this I concidered using D for all its metaprogramming, but I 
couldn't find how people use manual memory management for common 
types in the standard library. Is there no way to use custom 
memory management for types as strings, arrays, every function in 
the std.algorith module and slices? Am I missing something 
obvious?

What's the view of the creators of D with regards to custom 
(manual or gc) memory management for the standard library? C++ 
Makes a template of about everything that could allocate just so 
it can extract the allocator. (std::vector for instance) Will D 
do that too? Will D use a different approach or conclude that 
this problem only exists for a select few and not the main 
audience of D?
Mar 12
next sibling parent reply Kagamin <spam here.lot> writes:
The allocator API is experimental, so currently only 3rd party 
libraries use manual memory management.
Mar 13
parent reply Sjoerd Nijboer <dlang sjoerdnijboer.com> writes:
On Wednesday, 13 March 2019 at 08:22:30 UTC, Kagamin wrote:
 The allocator API is experimental, so currently only 3rd party 
 libraries use manual memory management.
If the allocator API would become non experimental, would there be any attempt to extract allocators out using templates?
Mar 13
parent Seb <seb wilzba.ch> writes:
On Wednesday, 13 March 2019 at 16:34:29 UTC, Sjoerd Nijboer wrote:
 On Wednesday, 13 March 2019 at 08:22:30 UTC, Kagamin wrote:
 The allocator API is experimental, so currently only 3rd party 
 libraries use manual memory management.
If the allocator API would become non experimental, would there be any attempt to extract allocators out using templates?
What do you mean by this? You can already use the allocator API today, though I would recommend to use the DUB package [1] as the API there is frozen and thus won't break with potential changes. Also v3 of the DUB package [1] works with BetterC. [1] https://github.com/dlang-community/stdx-allocator
Mar 13
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 12 March 2019 at 21:12:49 UTC, Sjoerd Nijboer wrote:
 So I pop in every now and then to look how D is progressing, 
 and the last time I concidered D was for a problem a friend of 
 me had with a small game. In that game there was some colission 
 detection that required a custom datastructure and then some 
 operations on top of that, with a hard limit on time.

 For this I concidered using D for all its metaprogramming, but 
 I couldn't find how people use manual memory management for 
 common types in the standard library. Is there no way to use 
 custom memory management for types as strings, arrays, every 
 function in the std.algorith module and slices? Am I missing 
 something obvious?

 What's the view of the creators of D with regards to custom 
 (manual or gc) memory management for the standard library? C++ 
 Makes a template of about everything that could allocate just 
 so it can extract the allocator. (std::vector for instance) 
 Will D do that too? Will D use a different approach or conclude 
 that this problem only exists for a select few and not the main 
 audience of D?
std.algorithm mostly doesn't care about allocations. I guess one *could* want to control how the returned ranges are allocated, but I'm not sure how useful that would be. Otherwise, I wrote this: https://github.com/atilaneves/automem
Mar 13
parent reply Sjoerd Nijboer <dlang sjoerdnijboer.com> writes:
On Wednesday, 13 March 2019 at 09:05:27 UTC, Atila Neves wrote:
 std.algorithm mostly doesn't care about allocations. I guess 
 one *could* want to control how the returned ranges are 
 allocated, but I'm not sure how useful that would be.

 Otherwise, I wrote this:

 https://github.com/atilaneves/automem
Automem with the vector template looks like i would expect D's arrays to be. Will we ever get something like this in the stdlib?
Mar 13
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 13 March 2019 at 16:45:35 UTC, Sjoerd Nijboer wrote:
 On Wednesday, 13 March 2019 at 09:05:27 UTC, Atila Neves wrote:
 std.algorithm mostly doesn't care about allocations. I guess 
 one *could* want to control how the returned ranges are 
 allocated, but I'm not sure how useful that would be.

 Otherwise, I wrote this:

 https://github.com/atilaneves/automem
Automem with the vector template looks like i would expect D's arrays to be. Will we ever get something like this in the stdlib?
Maybe, see e.g. https://github.com/dlang/druntime/pull/2348 Though Phobos has been declared "stable" these days, which means that breaking changes can't be done anymore and if you're looking for sth. modern and nice, dub is the place these days. I wonder if we ever get to the point where we can replace Phobos with a modern standard library...
Mar 13
parent reply Francesco Mecca <me francescomecca.eu> writes:
On Wednesday, 13 March 2019 at 17:50:57 UTC, Seb wrote:
 [...]
 Though Phobos has been declared "stable" these days, which 
 means that breaking changes can't be done anymore and if you're 
 looking for sth. modern and nice, dub is the place these days.
 I wonder if we ever get to the point where we can replace 
 Phobos with a modern standard library...
https://github.com/wilzbach/hellas Do you have any plans for this?
Mar 13
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 13 March 2019 at 19:22:37 UTC, Francesco Mecca 
wrote:
 On Wednesday, 13 March 2019 at 17:50:57 UTC, Seb wrote:
 [...]
 Though Phobos has been declared "stable" these days, which 
 means that breaking changes can't be done anymore and if 
 you're looking for sth. modern and nice, dub is the place 
 these days.
 I wonder if we ever get to the point where we can replace 
 Phobos with a modern standard library...
https://github.com/wilzbach/hellas Do you have any plans for this?
Yes, I do (if I ever get time to get it). The rough outline is here: https://github.com/wilzbach/dts/issues The rough vision of dts is to remove the ugly bits of Phobos that we can't remove or change and make the design/structure more fitting. However, there are a few limiting language/runtime issues like D strings being used for exceptions etc. Anyhow, I should probably point out that Jonathan is pushing a more radical approach with Dragon: https://github.com/dragon-lang The main vision of Dragon is to do the radical D3 changes that would never be accepted in mainline dmd, see e.g. https://github.com/dragon-lang/dc/issues/43
Mar 14
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 14 March 2019 at 09:25:47 UTC, Seb wrote:
 On Wednesday, 13 March 2019 at 19:22:37 UTC, Francesco Mecca 
 wrote:
 On Wednesday, 13 March 2019 at 17:50:57 UTC, Seb wrote:
 [...]
 Though Phobos has been declared "stable" these days, which 
 means that breaking changes can't be done anymore and if 
 you're looking for sth. modern and nice, dub is the place 
 these days.
 I wonder if we ever get to the point where we can replace 
 Phobos with a modern standard library...
https://github.com/wilzbach/hellas Do you have any plans for this?
Yes, I do (if I ever get time to get it). The rough outline is here: https://github.com/wilzbach/dts/issues The rough vision of dts is to remove the ugly bits of Phobos that we can't remove or change and make the design/structure more fitting. However, there are a few limiting language/runtime issues like D strings being used for exceptions etc. Anyhow, I should probably point out that Jonathan is pushing a more radical approach with Dragon: https://github.com/dragon-lang The main vision of Dragon is to do the radical D3 changes that would never be accepted in mainline dmd, see e.g. https://github.com/dragon-lang/dc/issues/43
That's interesting! It is no secret that I am strongly in favor of the evolution of the library / language even with radical changes. And this implies that I am in favor of D3 + an evolution of Phobos. Sociomantic has its own library, the same Weka, I would like a commercial effort to support and define D3 + Phobos. Something similar to a common organized work group, very very pragmatic. If this thing were feasible, my company would be interested in contributing. - Paolo
Mar 14
parent reply JN <666total wp.pl> writes:
On Thursday, 14 March 2019 at 10:02:20 UTC, Paolo Invernizzi 
wrote:
 On Thursday, 14 March 2019 at 09:25:47 UTC, Seb wrote:
 On Wednesday, 13 March 2019 at 19:22:37 UTC, Francesco Mecca 
 wrote:
 On Wednesday, 13 March 2019 at 17:50:57 UTC, Seb wrote:
 [...]
 Though Phobos has been declared "stable" these days, which 
 means that breaking changes can't be done anymore and if 
 you're looking for sth. modern and nice, dub is the place 
 these days.
 I wonder if we ever get to the point where we can replace 
 Phobos with a modern standard library...
https://github.com/wilzbach/hellas Do you have any plans for this?
Yes, I do (if I ever get time to get it). The rough outline is here: https://github.com/wilzbach/dts/issues The rough vision of dts is to remove the ugly bits of Phobos that we can't remove or change and make the design/structure more fitting. However, there are a few limiting language/runtime issues like D strings being used for exceptions etc. Anyhow, I should probably point out that Jonathan is pushing a more radical approach with Dragon: https://github.com/dragon-lang The main vision of Dragon is to do the radical D3 changes that would never be accepted in mainline dmd, see e.g. https://github.com/dragon-lang/dc/issues/43
That's interesting! It is no secret that I am strongly in favor of the evolution of the library / language even with radical changes. And this implies that I am in favor of D3 + an evolution of Phobos. Sociomantic has its own library, the same Weka, I would like a commercial effort to support and define D3 + Phobos. Something similar to a common organized work group, very very pragmatic. If this thing were feasible, my company would be interested in contributing. - Paolo
To be frank, even though I'd love the idea of D3, I don't think it's just a matter of Phobos. Phobos is a victim of the underlying ideology of D, which is both it's blessing and a curse. That ideology is the idea that all paradigms are equal and every usecase sh someone asks how to do manual memory management, the answer is either "you can't" or "interop with C". In D, the answer is "oh, you can totally do that, just this won't work, and this is needed, and this will probably not compile, oh and no one tested that". It adds a lot of cognitive overhead, and unfortunately there are many points of division like that. nogc/RC, GC, add to that -betterC too. Which one would a "new" Phobos support? I don't think you can support them all. You'd have to go for the lowest common denominator, which is nogc, but if stdlib doesn't require a GC, what is the point of a GC at all. Especially since GC is an issue when porting D to Webassembly (need to use -betterC for that right now). I think if D3 was to be successful, it'd have to be a bit more opinionated. I don't mean the language has to become a OOP behemoth (I'd love that though haha) or a functional monadic transformer combinator, but it'd be nice to better define a scope of the language and be able to answer a question "Can D do XYZ?" with "No, it's not useful and you shouldn't do it this way". Also, some language features seem dubious or hardly used at all. Does anyone really use contracts? I know they're sweet and Eiffel and everything, but does anyone ACTUALLY use them and doesn't stick to good old asserts? Perhaps rather than implementing (and supporting! each feature has to work well with other features which adds support overhead) such features, it'd be more useful to add features like struct initialization outside of assginments, which feels to me like a much nicer improvement to the language.
Mar 14
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 14, 2019 at 09:32:21PM +0000, JN via Digitalmars-d wrote:
[...]
 I think if D3 was to be successful, it'd have to be a bit more
 opinionated. I don't mean the language has to become a OOP behemoth
 (I'd love that though haha) or a functional monadic transformer
 combinator, but it'd be nice to better define a scope of the language
 and be able to answer a question "Can D do XYZ?" with "No, it's not
 useful and you shouldn't do it this way".
TBH I like the fact that D is not OOP-centric, because IMNSHO OOP is overhyped far beyond its usefulness. It has its place, but shoehorning everything into an OO paradigm just because it's the cool thing to do, is silly and leads to stupidity like singleton classes (nothing more than politically-correct euphemism for global functions and global variables) and excessive boilerplate. And I like the fact that you can write functional-like code in D without being bound in the straitjacket of strict functional purity and immutable values. Again, such a thing is useful for certain things and has its place, but I'd hate to be forced to write *everything* that way just because the language dictates so. One reason I chose to use D over other languages is because I'm sick and tired of opinionated languages dictating what I can or cannot do. I want to be able to what makes sense for the problem domain, not to waste time fighting with a language insists that I have to do it Their Way or the highway. I think a lot of the perceived complexity problems in D come from later additions that invalidate earlier assumptions. For example, the core language was originally designed with the GC being an inherent part of it, and the foundations of Phobos were laid upon that assumption. Then the no-GC crowd clamored and clamored for nogc, so they got nogc, and suddenly we realized that Phobos was completely incompatible with nogc, and core language features like arrays, AA's, and exceptions didn't play nice with it too. But since nogc is now officially supported, people inevitably clamored for nogc code to be put on equal footing with GC-reliant code. Hence a lot of churn just to "liberate" Phobos from the GC. And anyone who has worked with "enterprise" code knows what happens when a foundational assumption of a large, entrenched codebase is invalidated: an explosion of code complexity, special cases, and a labyrinth of dark corners for bugs to hide in. Similarly, before the range-based idiom entered into the language, you have things like std.string and std.array with freely-allocating functions. In order to retain backward compatibility, a lot of range functions with similar semantics had to be named differently, leading to ugly names like .joiner (rather than .join), .splitter (rather than .split), and so on. Had all these features been integrated into the original language design, these special cases and complexities would not be like they are today. OTOH, though, these twists and turns in the development of the language are also an integral part of what it is; UFCS, for example, came about from a clever hack added in order to make built-in arrays compatible with the then newly-introduced range API. Later on, people clamored for this "hack" to be extended to other types and become officially accepted. It was a huge success IMO. Nowadays, I can't imagine writing D code without UFCS. But still, you can see the seams where UFCS didn't quite fit with the original core language, e.g., the way it interacts with cross-module symbol resolution, operator overloading, etc.. Redesigning the language with all these things built into its core would eliminate these special cases. But it would also be a huge, monumental task, because each of these features, simple in appearance, actually imply a vast amount of complexity. No thanks to combinatorial explosion, any non-trivial set of features inevitably has unexpected corner cases that may behave differently from what one might expect, and the desire to cater to human intuition is always at odds with making the language simpler and more orthogonal.
 Also, some language features seem dubious or hardly used at all. Does
 anyone really use contracts? I know they're sweet and Eiffel and
 everything, but does anyone ACTUALLY use them and doesn't stick to
 good old asserts?
I do. They're useful for documenting the intent of APIs, whereas I generally use asserts for internal, implementation-specific assumptions.
 Perhaps rather than implementing (and supporting! each feature has to
 work well with other features which adds support overhead) such
 features, it'd be more useful to add features like struct
 initialization outside of assginments, which feels to me like a much
 nicer improvement to the language.
What do you mean by struct initialization outside of assignments? D already supports struct initialization syntax of the form S(...) inside expressions. T -- Those who don't understand D are condemned to reinvent it, poorly. -- Daniel N
Mar 14
next sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 14 March 2019 at 23:06:19 UTC, H. S. Teoh wrote:
 What do you mean by struct initialization outside of 
 assignments? D already supports struct initialization syntax of 
 the form S(...) inside expressions.
https://github.com/dlang/DIPs/pull/71
Mar 14
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 14, 2019 at 11:23:35PM +0000, Seb via Digitalmars-d wrote:
 On Thursday, 14 March 2019 at 23:06:19 UTC, H. S. Teoh wrote:
 What do you mean by struct initialization outside of assignments? D
 already supports struct initialization syntax of the form S(...)
 inside expressions.
https://github.com/dlang/DIPs/pull/71
[...] Ah I see. Well it's a minor syntactic convenience, and we do already support ctor calls (albeit without the ability to explicitly name fields), so I don't consider it a major point at all. But certainly nice to have. T -- Some ideas are so stupid that only intellectuals could believe them. -- George Orwell
Mar 15
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-15 00:06, H. S. Teoh wrote:

 OTOH, though, these twists and turns in the development of the language
 are also an integral part of what it is; UFCS, for example, came about
 from a clever hack added in order to make built-in arrays compatible
 with the then newly-introduced range API. Later on, people clamored for
 this "hack" to be extended to other types and become officially
 accepted. It was a huge success IMO. Nowadays, I can't imagine writing D
 code without UFCS.
That's not what happened. UFCS for arrays existed in D1, long before D2 and when ranges were introduced. -- /Jacob Carlborg
Mar 17
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Mar 17, 2019 at 10:50:15AM +0100, Jacob Carlborg via Digitalmars-d
wrote:
 On 2019-03-15 00:06, H. S. Teoh wrote:
 
 OTOH, though, these twists and turns in the development of the
 language are also an integral part of what it is; UFCS, for example,
 came about from a clever hack added in order to make built-in arrays
 compatible with the then newly-introduced range API. Later on,
 people clamored for this "hack" to be extended to other types and
 become officially accepted. It was a huge success IMO. Nowadays, I
 can't imagine writing D code without UFCS.
That's not what happened. UFCS for arrays existed in D1, long before D2 and when ranges were introduced.
[...] Wasn't that a parser quirk or something that was later taken advantage of? T -- It always amuses me that Windows has a Safe Mode during bootup. Does that mean that Windows is normally unsafe?
Mar 17
prev sibling next sibling parent James Blachly <james.blachly gmail.com> writes:
On 3/14/19 5:32 PM, JN wrote:
 Also, some language features seem dubious or hardly used at all. Does 
 anyone really use contracts? I know they're sweet and Eiffel and 
 everything, but does anyone ACTUALLY use them and doesn't stick to good 
 old asserts? Perhaps rather than implementing (and supporting! each 
 feature has to work well with other features which adds support 
 overhead) such features, it'd be more useful to add features like struct 
 initialization outside of assginments, which feels to me like a much 
 nicer improvement to the language.
FWIW as a new-ish D programmer I used contracts to great effect to quickly write an interval tree library (which I will release soon). The contracts mechanism sped development tremendously, as well as caught several subtle edge-case errors that would have been nightmare to find without. That being said, I would love struct initialization ...
Mar 14
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
On Thursday, 14 March 2019 at 21:32:21 UTC, JN wrote:
 On Thursday, 14 March 2019 at 10:02:20 UTC, Paolo Invernizzi 
 wrote:
 On Thursday, 14 March 2019 at 09:25:47 UTC, Seb wrote:
 [...]
That's interesting! It is no secret that I am strongly in favor of the evolution of the library / language even with radical changes. And this implies that I am in favor of D3 + an evolution of Phobos. Sociomantic has its own library, the same Weka, I would like a commercial effort to support and define D3 + Phobos. Something similar to a common organized work group, very very pragmatic. If this thing were feasible, my company would be interested in contributing. - Paolo
To be frank, even though I'd love the idea of D3, I don't think it's just a matter of Phobos. Phobos is a victim of the underlying ideology of D, which is both it's blessing and a curse. That ideology is the idea that all paradigms are equal and every usecase should be supported. In languages like Java/C#, if someone asks how to do manual memory management, the answer is either "you can't" or "interop with C". [...]
In Java the answer has also been user java.nio.Buffer, and it will be improved when value types finally become a thing. In C#, it has always been a thing since the early days via System.Runtime.InteropServices.Marshal and value types, improved later via SafeH memory allocation alongside slices and channels. -- Paulo
Mar 15
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Thursday, 14 March 2019 at 21:32:21 UTC, JN wrote:
 In D, the answer is "oh, you can totally do that, just this 
 won't work, and this is needed, and this will probably not 
 compile, oh and no one tested that". It adds a lot of cognitive 
 overhead, and unfortunately there are many points of division 
 like that.
nogc is inherently cognitive overhead.
Mar 15
prev sibling next sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Thursday, 14 March 2019 at 21:32:21 UTC, JN wrote:
 Perhaps rather than implementing (and supporting! each feature 
 has to work well with other features which adds support 
 overhead) such features, it'd be more useful to add features 
 like struct initialization outside of assginments, which feels 
 to me like a much nicer improvement to the language.
The problem when designing a language, is that *everyone* says "Stop wasting your time maintaining [feature I don't use], nobody cares about it. You should implement [feature that I want] instead, the language would be more popular that way!", but nobody can agree on which features are the "good" ones. Eg, some people will say that -betterC is almost useless and we should stop worrying about it, while other people would stop using D if it didn't have that flag.
Mar 16
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-14 22:32, JN wrote:

 To be frank, even though I'd love the idea of D3, I don't think it's 
 just a matter of Phobos. Phobos is a victim of the underlying ideology 
 of D, which is both it's blessing and a curse. That ideology is the idea 
 that all paradigms are equal and every usecase should be supported. In 
 languages like Java/C#, if someone asks how to do manual memory 
 management, the answer is either "you can't" or "interop with C". In D, 
 the answer is "oh, you can totally do that, just this won't work, and 
 this is needed, and this will probably not compile, oh and no one tested 
 that". It adds a lot of cognitive overhead, and unfortunately there are 
 many points of division like that.  nogc/RC, GC, add to that -betterC 
 too. Which one would a "new" Phobos support? I don't think you can 
 support them all. You'd have to go for the lowest common denominator, 
 which is  nogc, but if stdlib doesn't require a GC, what is the point of 
 a GC at all. 
This can be solved by pushing the allocation up the call stack. Instead of having a function allocate memory, pass a buffer, delegate, output range or something similar to the function. Then whatever you pass can decide if it should allocate using the GC, malloc or something else. This will make the API not as nice to work with. But a layer on top of that can be built with a default allocation strategy. I would suggest the default allocation strategy to be the GC. If someone doesn't want to use the GC he/she can use the layer below and decide how the allocation should be done.
 Also, some language features seem dubious or hardly used at all. Does 
 anyone really use contracts? I know they're sweet and Eiffel and 
 everything, but does anyone ACTUALLY use them and doesn't stick to good 
 old asserts? Perhaps rather than implementing (and supporting! each 
 feature has to work well with other features which adds support 
 overhead) such features, it'd be more useful to add features like struct 
 initialization outside of assginments, which feels to me like a much 
 nicer improvement to the language.
I use contracts. But with the current implementation it's only a syntactic difference between an in-contract and a regular assert inside the function body. Instead, the in-contract should be called by the caller, not the callee. -- /Jacob Carlborg
Mar 16
parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Saturday, 16 March 2019 at 20:17:50 UTC, Jacob Carlborg wrote:
 Instead of having a function allocate memory, pass a buffer, 
 delegate, output range or something similar to the function. 
 Then whatever you pass can decide if it should allocate using 
 the GC, malloc or something else.
It would have to be a delegate or an output range, because in most cases the callee wouldn't know in advance how much storage it needs to allocate. Anyway, it would be pretty nice if there were a standardized way to tell a function how it should allocate its memory; it would allow some nice strategies like "I know that I'm going to discard most of what this function allocates, so just put in in a continuous buffer; then deep-copy the relevant data to GC memory, and throw the rest away" (eg in a parser). However, for that to work, you'd probably need a way to pass implicit arguments to a function without having to specify `foobar(defaultGcAllocator, x, y, z)` every time you call a function.
Mar 16
next sibling parent James Blachly <james.blachly gmail.com> writes:
On 3/16/19 5:17 PM, Olivier FAURE wrote:
 Anyway, it would be pretty nice if there were a standardized way to tell 
 a function how it should allocate its memory; it would allow some nice 
 strategies like "I know that I'm going to discard most of what this 
 function allocates, so just put in in a continuous buffer; then 
 deep-copy the relevant data to GC memory, and throw the rest away" (eg 
 in a parser).
Wow, that is a really fascinating idea ...
Mar 16
prev sibling next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 17/03/2019 10:17 AM, Olivier FAURE wrote:
 
 Anyway, it would be pretty nice if there were a standardized way to tell 
 a function how it should allocate its memory; it would allow some nice 
 strategies like "I know that I'm going to discard most of what this 
 function allocates, so just put in in a continuous buffer; then 
 deep-copy the relevant data to GC memory, and throw the rest away" (eg 
 in a parser).
I have questions related to how much garbage this parser is creating. Ignoring how complex of an idea this actually is, we would need clear set of examples demonstrating that this cannot be done manually (choosing between GC and another allocator). But otherwise, neat.
Mar 16
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-03-16 22:17, Olivier FAURE wrote:

 However, for that to work, you'd probably need a way to pass implicit 
 arguments to a function without having to specify 
 `foobar(defaultGcAllocator, x, y, z)` every time you call a function.
With my idea that would be the second layer, sitting on top of the more general layer. I guess a parameter with a default value could work: void foobar(Allocator)(int x, int y, int z, Allocator allocator = defaultGcAllocator); -- /Jacob Carlborg
Mar 17
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 16 Mar 2019 21:17:02 +0000 schrieb Olivier FAURE:
 
 However, for that to work, you'd probably need a way to pass implicit
 arguments to a function without having to specify
 `foobar(defaultGcAllocator, x, y, z)` every time you call a function.
You can implement that using TLS globals, I think this is what std.experimental.allocators theAllocator is supposed to do. Of course you're limited in typing, you can't use templates there. -- Johannes
Mar 17
parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-17 12:27, Johannes Pfau wrote:

 You can implement that using TLS globals, I think this is what
 std.experimental.allocators theAllocator is supposed to do. Of course
 you're limited in typing, you can't use templates there.
Then the functions can't be pure. -- /Jacob Carlborg
Mar 18
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Monday, 18 March 2019 at 10:07:37 UTC, Jacob Carlborg wrote:
 On 2019-03-17 12:27, Johannes Pfau wrote:
 You can implement that using TLS globals, I think this is what
 std.experimental.allocators theAllocator is supposed to do. Of 
 course
 you're limited in typing, you can't use templates there.
Then the functions can't be pure.
Yeah, what you want is either an annotation to tell a function foobar "you're pure except for *these* global variables", and then a way to shim the globals when calling foobar from a pure function, or implicit arguments (which are basically the same thing).
Mar 18
prev sibling next sibling parent reply Sjoerd Nijboer <dlang sjoerdnijboer.com> writes:
On Saturday, 16 March 2019 at 21:17:02 UTC, Olivier FAURE wrote:
 On Saturday, 16 March 2019 at 20:17:50 UTC, Jacob Carlborg 
 wrote:
 Instead of having a function allocate memory, pass a buffer, 
 delegate, output range or something similar to the function. 
 Then whatever you pass can decide if it should allocate using 
 the GC, malloc or something else.
It would have to be a delegate or an output range, because in most cases the callee wouldn't know in advance how much storage it needs to allocate. Anyway, it would be pretty nice if there were a standardized way to tell a function how it should allocate its memory; it would allow some nice strategies like "I know that I'm going to discard most of what this function allocates, so just put in in a continuous buffer; then deep-copy the relevant data to GC memory, and throw the rest away" (eg in a parser). However, for that to work, you'd probably need a way to pass implicit arguments to a function without having to specify `foobar(defaultGcAllocator, x, y, z)` every time you call a function.
How about something like the following? It introduces an overload of keyword new, a keyword 'old' and a keyword 'lifetime' to pass in an IAllocator as a template argument to a function, and change the name mangling to add it as a hidden parameter. /*MyClass is an implicit template which takes an IAllocator. It can always be implicitly casted to a MyClass for backwards compatibility. It can use this allocator using the .allocatorOf property.*/ class MyClass { public MyClass getANEwInstance(MyClass a, MyClass b) { .... //some code return this.allocatorOf new MyClass(); } /*This function takes Myclass A as a parameter and returns a new instance of MyClass using thesame allocator. if no explicit allocator is used for a, or no allocator is known, the default GC_ALLOCATOR is used. This allows for better compatibility with existing D libraries.*/ public lifetime(a) getANewInstanceFromParameter(lifetime MyClass a) { .... //some code return a.allocatorof new MyClass(); } /* a class which has a lifetime associated with it, should be able to decay into a "normal" class so they can still be used in existing codebases. They are an opt-in feature because of it. simpleProperty() demonstrates this. */ public bool simpleProperty(MyClass myClass) { .... return myClass.someProperty(); } } //usage: public void foo(IAllocator A)() { MyClass c1 = new MyClass(); //Use default GC allocator. MyClass c2 = Mallocator new MyClass (); // Allocate using malloc. /*Instantiate MyClas using allocator A, and mangle the allocator A into the resulting type as the first template parameter.*/ MyClass c3 = A new MyClass(); /* since a reference to a class instance is still a pointer, we can discard the allocator of the pointer and call regular D, C++ and C functions with it. */ println(c3.simpleProperty()); // get a new object with thesame allocator as c1, since c1 is a lifetime object. auto c4 = c1.getANEwInstance(); auto c5 = c2.getANewInstanceFromParameter(c3); old c1; // no-operation old c2; // destruct and free c2 old c3; // destruct and free c3 if A is not a GC allocator using introspection. old c4; /*destruct and free using thesame allocator as c1, meaning no-op in this case.*/ old c5; /*destruct and free using thesame allocator as c3, meaning this is conditional on A being a GC allocator.*/ } ` I think that this is an example which would be quite easy to learn, not a huge strain on the developers, backwards compatible with existing D code and potentially immensely powerfull. It would also give helpful info which can be used by debuggers and other tooling. One thing it can never do is return an object which was conditionally allocated with one allocator or another.
Mar 17
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Sunday, 17 March 2019 at 14:28:49 UTC, Sjoerd Nijboer wrote:
 I think that this is an example which would be quite easy to 
 learn, not a huge strain on the developers, backwards 
 compatible with existing D code and potentially immensely 
 powerfull. It would also give helpful info which can be used by 
 debuggers and other tooling. One thing it can never do is 
 return an object which was conditionally allocated with one 
 allocator or another.
I'll be honest, the example you gave didn't help me understand your idea at all. It looks way too complicated, and way too different from D's existing semantics.
Mar 18
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 16 March 2019 at 21:17:02 UTC, Olivier FAURE wrote:
 On Saturday, 16 March 2019 at 20:17:50 UTC, Jacob Carlborg 
 wrote:
 Instead of having a function allocate memory, pass a buffer, 
 delegate, output range or something similar to the function. 
 Then whatever you pass can decide if it should allocate using 
 the GC, malloc or something else.
It would have to be a delegate or an output range, because in most cases the callee wouldn't know in advance how much storage it needs to allocate. Anyway, it would be pretty nice if there were a standardized way to tell a function how it should allocate its memory;
There is: void fun(A)(auto ref A allocator);
 it would allow some nice strategies like "I know that I'm going 
 to discard most of what this function allocates, so just put in 
 in a continuous buffer; then deep-copy the relevant data to GC 
 memory, and throw the rest away" (eg in a parser).
This can all be done with a custom allocator, or one built with the building blocks in std.experimental.allocator.
 However, for that to work, you'd probably need a way to pass 
 implicit arguments to a function without having to specify 
 `foobar(defaultGcAllocator, x, y, z)` every time you call a 
 function.
void foobar(int x, int y, int z) { import std.experimental.allocator.gc_allocator: GCAllocator; foobar(GCAllocator.instance, x, y, z); } void foobar(A)(auto ref A allocator, int x, int y, int z) { // ... } Or use `theAllocator`.
Mar 19
prev sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 14 March 2019 at 21:32:21 UTC, JN wrote:
 On Thursday, 14 March 2019 at 10:02:20 UTC, Paolo Invernizzi 
 wrote:
 [...]
To be frank, even though I'd love the idea of D3, I don't think it's just a matter of Phobos. Phobos is a victim of the underlying ideology of D, which is both it's blessing and a curse. That ideology is the idea that all paradigms are equal and every usecase should be supported. In languages like Java/C#, if someone asks how to do manual memory management, the answer is either "you can't" or "interop with C". In D, the answer is "oh, you can totally do that, just this won't work, and this is needed, and this will probably not compile, oh and no one tested that". It adds a lot of cognitive overhead, and unfortunately there are many points of division like that. nogc/RC, GC, add to that -betterC too. Which one would a "new" Phobos support? I don't think you can support them all. You'd have to go for the lowest common denominator, which is nogc, but if stdlib doesn't require a GC, what is the point of a GC at all. Especially since GC is an issue when porting D to Webassembly (need to use -betterC for that right now). I think if D3 was to be successful, it'd have to be a bit more opinionated. I don't mean the language has to become a OOP behemoth (I'd love that though haha) or a functional monadic transformer combinator, but it'd be nice to better define a scope of the language and be able to answer a question "Can D do XYZ?" with "No, it's not useful and you shouldn't do it this way". Also, some language features seem dubious or hardly used at all. Does anyone really use contracts? I know they're sweet and Eiffel and everything, but does anyone ACTUALLY use them and doesn't stick to good old asserts? Perhaps rather than implementing (and supporting! each feature has to work well with other features which adds support overhead) such features, it'd be more useful to add features like struct initialization outside of assginments, which feels to me like a much nicer improvement to the language.
There are a lot of changes, with large consensus, that are currently not admitted because of existing code. Just **today** another one coming from Walter [1]: "As mentioned earlier, scope guard will catch Errors, but it should really only catch Exceptions. This hasn't been fixed because there's existing code that relies on it catching Errors." And the history of D is full of such examples, low hanging fruits, actionable. / Paolo [1] https://forum.dlang.org/post/mailman.7569.1552776840.29801.digitalmars-d-bugs puremagic.com
Mar 17
prev sibling parent James Blachly <james.blachly gmail.com> writes:
On 3/12/19 5:12 PM, Sjoerd Nijboer wrote:
 ...
 What's the view of the creators of D with regards to custom (manual or 
 gc) memory management for the standard library? C++ Makes a template of 
 about everything that could allocate just so it can extract the 
 allocator. (std::vector for instance) Will D do that too? Will D use a 
 different approach or conclude that this problem only exists for a 
 select few and not the main audience of D?
Partially related: I believe a recent (most recent?) version of D allows pluggable garbage collector, although I am not aware of any plugins just yet.
Mar 13