www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - memory management and the standard library

reply Sjoerd Nijboer <dlang sjoerdnijboer.com> writes:
So I pop in every now and then to look how D is progressing, and 
the last time I concidered D was for a problem a friend of me had 
with a small game. In that game there was some colission 
detection that required a custom datastructure and then some 
operations on top of that, with a hard limit on time.

For this I concidered using D for all its metaprogramming, but I 
couldn't find how people use manual memory management for common 
types in the standard library. Is there no way to use custom 
memory management for types as strings, arrays, every function in 
the std.algorith module and slices? Am I missing something 
obvious?

What's the view of the creators of D with regards to custom 
(manual or gc) memory management for the standard library? C++ 
Makes a template of about everything that could allocate just so 
it can extract the allocator. (std::vector for instance) Will D 
do that too? Will D use a different approach or conclude that 
this problem only exists for a select few and not the main 
audience of D?
Mar 12 2019
next sibling parent reply Kagamin <spam here.lot> writes:
The allocator API is experimental, so currently only 3rd party 
libraries use manual memory management.
Mar 13 2019
parent reply Sjoerd Nijboer <dlang sjoerdnijboer.com> writes:
On Wednesday, 13 March 2019 at 08:22:30 UTC, Kagamin wrote:

 The allocator API is experimental, so currently only 3rd party 
 libraries use manual memory management.


If the allocator API would become non experimental, would there 
be any attempt to extract allocators out using templates?
Mar 13 2019
parent Seb <seb wilzba.ch> writes:
On Wednesday, 13 March 2019 at 16:34:29 UTC, Sjoerd Nijboer wrote:

 On Wednesday, 13 March 2019 at 08:22:30 UTC, Kagamin wrote:

 The allocator API is experimental, so currently only 3rd party 
 libraries use manual memory management.


 If the allocator API would become non experimental, would there 
 be any attempt to extract allocators out using templates?


What do you mean by this? You can already use the allocator API 
today, though I would recommend to use the DUB package [1] as the 
API there is frozen and thus won't break with potential changes. 
Also v3 of the DUB package [1] works with BetterC.

[1] https://github.com/dlang-community/stdx-allocator
Mar 13 2019
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 12 March 2019 at 21:12:49 UTC, Sjoerd Nijboer wrote:

 So I pop in every now and then to look how D is progressing, 
 and the last time I concidered D was for a problem a friend of 
 me had with a small game. In that game there was some colission 
 detection that required a custom datastructure and then some 
 operations on top of that, with a hard limit on time.

 For this I concidered using D for all its metaprogramming, but 
 I couldn't find how people use manual memory management for 
 common types in the standard library. Is there no way to use 
 custom memory management for types as strings, arrays, every 
 function in the std.algorith module and slices? Am I missing 
 something obvious?

 What's the view of the creators of D with regards to custom 
 (manual or gc) memory management for the standard library? C++ 
 Makes a template of about everything that could allocate just 
 so it can extract the allocator. (std::vector for instance) 
 Will D do that too? Will D use a different approach or conclude 
 that this problem only exists for a select few and not the main 
 audience of D?


std.algorithm mostly doesn't care about allocations. I guess one 
*could* want to control how the returned ranges are allocated, 
but I'm not sure how useful that would be.

Otherwise, I wrote this:

https://github.com/atilaneves/automem
Mar 13 2019
parent reply Sjoerd Nijboer <dlang sjoerdnijboer.com> writes:
On Wednesday, 13 March 2019 at 09:05:27 UTC, Atila Neves wrote:

 std.algorithm mostly doesn't care about allocations. I guess 
 one *could* want to control how the returned ranges are 
 allocated, but I'm not sure how useful that would be.

 Otherwise, I wrote this:

 https://github.com/atilaneves/automem


Automem with the vector template looks like i would expect D's 
arrays to be. Will we ever get something like this in the stdlib?
Mar 13 2019
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 13 March 2019 at 16:45:35 UTC, Sjoerd Nijboer wrote:

 On Wednesday, 13 March 2019 at 09:05:27 UTC, Atila Neves wrote:

 std.algorithm mostly doesn't care about allocations. I guess 
 one *could* want to control how the returned ranges are 
 allocated, but I'm not sure how useful that would be.

 Otherwise, I wrote this:

 https://github.com/atilaneves/automem


 Automem with the vector template looks like i would expect D's 
 arrays to be. Will we ever get something like this in the 
 stdlib?


Maybe, see e.g. https://github.com/dlang/druntime/pull/2348

Though Phobos has been declared "stable" these days, which means 
that breaking changes can't be done anymore and if you're looking 
for sth. modern and nice, dub is the place these days.
I wonder if we ever get to the point where we can replace Phobos 
with a modern standard library...
Mar 13 2019
parent reply Francesco Mecca <me francescomecca.eu> writes:
On Wednesday, 13 March 2019 at 17:50:57 UTC, Seb wrote:

 [...]
 Though Phobos has been declared "stable" these days, which 
 means that breaking changes can't be done anymore and if you're 
 looking for sth. modern and nice, dub is the place these days.
 I wonder if we ever get to the point where we can replace 
 Phobos with a modern standard library...


https://github.com/wilzbach/hellas
Do you have any plans for this?
Mar 13 2019
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 13 March 2019 at 19:22:37 UTC, Francesco Mecca 
wrote:

 On Wednesday, 13 March 2019 at 17:50:57 UTC, Seb wrote:

 [...]
 Though Phobos has been declared "stable" these days, which 
 means that breaking changes can't be done anymore and if 
 you're looking for sth. modern and nice, dub is the place 
 these days.
 I wonder if we ever get to the point where we can replace 
 Phobos with a modern standard library...


 https://github.com/wilzbach/hellas
 Do you have any plans for this?


Yes, I do (if I ever get time to get it).
The rough outline is here:

https://github.com/wilzbach/dts/issues

The rough vision of dts is to remove the ugly bits of Phobos that 
we can't remove or change and make the design/structure more 
fitting.
However, there are a few limiting language/runtime issues like D 
strings being used for exceptions etc.

Anyhow, I should probably point out that Jonathan is pushing a 
more radical approach with Dragon:

https://github.com/dragon-lang

The main vision of Dragon is to do the radical D3 changes that 
would never be accepted in mainline dmd, see e.g.

https://github.com/dragon-lang/dc/issues/43
Mar 14 2019
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 14 March 2019 at 09:25:47 UTC, Seb wrote:

 On Wednesday, 13 March 2019 at 19:22:37 UTC, Francesco Mecca 
 wrote:

 On Wednesday, 13 March 2019 at 17:50:57 UTC, Seb wrote:

 [...]
 Though Phobos has been declared "stable" these days, which 
 means that breaking changes can't be done anymore and if 
 you're looking for sth. modern and nice, dub is the place 
 these days.
 I wonder if we ever get to the point where we can replace 
 Phobos with a modern standard library...


 https://github.com/wilzbach/hellas
 Do you have any plans for this?


 Yes, I do (if I ever get time to get it).
 The rough outline is here:

 https://github.com/wilzbach/dts/issues

 The rough vision of dts is to remove the ugly bits of Phobos 
 that we can't remove or change and make the design/structure 
 more fitting.
 However, there are a few limiting language/runtime issues like 
 D strings being used for exceptions etc.

 Anyhow, I should probably point out that Jonathan is pushing a 
 more radical approach with Dragon:

 https://github.com/dragon-lang

 The main vision of Dragon is to do the radical D3 changes that 
 would never be accepted in mainline dmd, see e.g.

 https://github.com/dragon-lang/dc/issues/43


That's interesting!

It is no secret that I am strongly in favor of the evolution of 
the library / language even with radical changes. And this 
implies that I am in favor of D3 + an evolution of Phobos.

Sociomantic has its own library, the same Weka, I would like a 
commercial effort to support and define D3 + Phobos.

Something similar to a common organized work group, very very 
pragmatic. If this thing were feasible, my company would be 
interested in contributing.

- Paolo
Mar 14 2019
parent reply JN <666total wp.pl> writes:
On Thursday, 14 March 2019 at 10:02:20 UTC, Paolo Invernizzi 
wrote:

 On Thursday, 14 March 2019 at 09:25:47 UTC, Seb wrote:

 On Wednesday, 13 March 2019 at 19:22:37 UTC, Francesco Mecca 
 wrote:

 On Wednesday, 13 March 2019 at 17:50:57 UTC, Seb wrote:

 [...]
 Though Phobos has been declared "stable" these days, which 
 means that breaking changes can't be done anymore and if 
 you're looking for sth. modern and nice, dub is the place 
 these days.
 I wonder if we ever get to the point where we can replace 
 Phobos with a modern standard library...


 https://github.com/wilzbach/hellas
 Do you have any plans for this?


 Yes, I do (if I ever get time to get it).
 The rough outline is here:

 https://github.com/wilzbach/dts/issues

 The rough vision of dts is to remove the ugly bits of Phobos 
 that we can't remove or change and make the design/structure 
 more fitting.
 However, there are a few limiting language/runtime issues like 
 D strings being used for exceptions etc.

 Anyhow, I should probably point out that Jonathan is pushing a 
 more radical approach with Dragon:

 https://github.com/dragon-lang

 The main vision of Dragon is to do the radical D3 changes that 
 would never be accepted in mainline dmd, see e.g.

 https://github.com/dragon-lang/dc/issues/43


 That's interesting!

 It is no secret that I am strongly in favor of the evolution of 
 the library / language even with radical changes. And this 
 implies that I am in favor of D3 + an evolution of Phobos.

 Sociomantic has its own library, the same Weka, I would like a 
 commercial effort to support and define D3 + Phobos.

 Something similar to a common organized work group, very very 
 pragmatic. If this thing were feasible, my company would be 
 interested in contributing.

 - Paolo


To be frank, even though I'd love the idea of D3, I don't think 
it's just a matter of Phobos. Phobos is a victim of the 
underlying ideology of D, which is both it's blessing and a 
curse. That ideology is the idea that all paradigms are equal and 

someone asks how to do manual memory management, the answer is 
either "you can't" or "interop with C". In D, the answer is "oh, 
you can totally do that, just this won't work, and this is 
needed, and this will probably not compile, oh and no one tested 
that". It adds a lot of cognitive overhead, and unfortunately 
there are many points of division like that.  nogc/RC, GC, add to 
that -betterC too. Which one would a "new" Phobos support? I 
don't think you can support them all. You'd have to go for the 
lowest common denominator, which is  nogc, but if stdlib doesn't 
require a GC, what is the point of a GC at all. Especially since 
GC is an issue when porting D to Webassembly (need to use 
-betterC for that right now). I think if D3 was to be successful, 
it'd have to be a bit more opinionated. I don't mean the language 
has to become a OOP behemoth (I'd love that though haha) or a 
functional monadic transformer combinator, but it'd be nice to 
better define a scope of the language and be able to answer a 
question "Can D do XYZ?" with "No, it's not useful and you 
shouldn't do it this way".

Also, some language features seem dubious or hardly used at all. 
Does anyone really use contracts? I know they're sweet and Eiffel 
and everything, but does anyone ACTUALLY use them and doesn't 
stick to good old asserts? Perhaps rather than implementing (and 
supporting! each feature has to work well with other features 
which adds support overhead) such features, it'd be more useful 
to add features like struct initialization outside of 
assginments, which feels to me like a much nicer improvement to 
the language.
Mar 14 2019
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 14, 2019 at 09:32:21PM +0000, JN via Digitalmars-d wrote:
[...]

 I think if D3 was to be successful, it'd have to be a bit more
 opinionated. I don't mean the language has to become a OOP behemoth
 (I'd love that though haha) or a functional monadic transformer
 combinator, but it'd be nice to better define a scope of the language
 and be able to answer a question "Can D do XYZ?" with "No, it's not
 useful and you shouldn't do it this way".


TBH I like the fact that D is not OOP-centric, because IMNSHO OOP is
overhyped far beyond its usefulness.  It has its place, but shoehorning
everything into an OO paradigm just because it's the cool thing to do,
is silly and leads to stupidity like singleton classes (nothing more
than politically-correct euphemism for global functions and global
variables) and excessive boilerplate.

And I like the fact that you can write functional-like code in D without
being bound in the straitjacket of strict functional purity and
immutable values.  Again, such a thing is useful for certain things and
has its place, but I'd hate to be forced to write *everything* that way
just because the language dictates so.

One reason I chose to use D over other languages is because I'm sick and
tired of opinionated languages dictating what I can or cannot do. I want
to be able to what makes sense for the problem domain, not to waste time
fighting with a language insists that I have to do it Their Way or the
highway.

I think a lot of the perceived complexity problems in D come from later
additions that invalidate earlier assumptions.  For example, the core
language was originally designed with the GC being an inherent part of
it, and the foundations of Phobos were laid upon that assumption.  Then
the no-GC crowd clamored and clamored for  nogc, so they got  nogc, and
suddenly we realized that Phobos was completely incompatible with  nogc,
and core language features like arrays, AA's, and exceptions didn't play
nice with it too.  But since  nogc is now officially supported, people
inevitably clamored for  nogc code to be put on equal footing with
GC-reliant code.  Hence a lot of churn just to "liberate" Phobos from
the GC. And anyone who has worked with "enterprise" code knows what
happens when a foundational assumption of a large, entrenched codebase
is invalidated: an explosion of code complexity, special cases, and a
labyrinth of dark corners for bugs to hide in.

Similarly, before the range-based idiom entered into the language, you
have things like std.string and std.array with freely-allocating
functions.  In order to retain backward compatibility, a lot of range
functions with similar semantics had to be named differently, leading to
ugly names like .joiner (rather than .join), .splitter (rather than
.split), and so on.

Had all these features been integrated into the original language
design, these special cases and complexities would not be like they are
today.

OTOH, though, these twists and turns in the development of the language
are also an integral part of what it is; UFCS, for example, came about
from a clever hack added in order to make built-in arrays compatible
with the then newly-introduced range API. Later on, people clamored for
this "hack" to be extended to other types and become officially
accepted. It was a huge success IMO. Nowadays, I can't imagine writing D
code without UFCS.

But still, you can see the seams where UFCS didn't quite fit with the
original core language, e.g., the way it interacts with cross-module
symbol resolution, operator overloading, etc..

Redesigning the language with all these things built into its core would
eliminate these special cases. But it would also be a huge, monumental
task, because each of these features, simple in appearance, actually
imply a vast amount of complexity. No thanks to combinatorial explosion,
any non-trivial set of features inevitably has unexpected corner cases
that may behave differently from what one might expect, and the desire
to cater to human intuition is always at odds with making the language
simpler and more orthogonal.



 Also, some language features seem dubious or hardly used at all. Does
 anyone really use contracts? I know they're sweet and Eiffel and
 everything, but does anyone ACTUALLY use them and doesn't stick to
 good old asserts?


I do.  They're useful for documenting the intent of APIs, whereas I
generally use asserts for internal, implementation-specific assumptions.



 Perhaps rather than implementing (and supporting! each feature has to
 work well with other features which adds support overhead) such
 features, it'd be more useful to add features like struct
 initialization outside of assginments, which feels to me like a much
 nicer improvement to the language.


What do you mean by struct initialization outside of assignments? D
already supports struct initialization syntax of the form S(...) inside
expressions.


T

-- 
Those who don't understand D are condemned to reinvent it, poorly. -- Daniel N
Mar 14 2019
next sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 14 March 2019 at 23:06:19 UTC, H. S. Teoh wrote:

 What do you mean by struct initialization outside of 
 assignments? D already supports struct initialization syntax of 
 the form S(...) inside expressions.


https://github.com/dlang/DIPs/pull/71
Mar 14 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 14, 2019 at 11:23:35PM +0000, Seb via Digitalmars-d wrote:

 On Thursday, 14 March 2019 at 23:06:19 UTC, H. S. Teoh wrote:

 What do you mean by struct initialization outside of assignments? D
 already supports struct initialization syntax of the form S(...)
 inside expressions.


 
 https://github.com/dlang/DIPs/pull/71


[...]

Ah I see. Well it's a minor syntactic convenience, and we do already
support ctor calls (albeit without the ability to explicitly name
fields), so I don't consider it a major point at all.  But certainly
nice to have.


T

-- 
Some ideas are so stupid that only intellectuals could believe them. -- George
Orwell
Mar 15 2019
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-15 00:06, H. S. Teoh wrote:


 OTOH, though, these twists and turns in the development of the language
 are also an integral part of what it is; UFCS, for example, came about
 from a clever hack added in order to make built-in arrays compatible
 with the then newly-introduced range API. Later on, people clamored for
 this "hack" to be extended to other types and become officially
 accepted. It was a huge success IMO. Nowadays, I can't imagine writing D
 code without UFCS.


That's not what happened. UFCS for arrays existed in D1, long before D2 
and when ranges were introduced.

-- 
/Jacob Carlborg
Mar 17 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Mar 17, 2019 at 10:50:15AM +0100, Jacob Carlborg via Digitalmars-d
wrote:

 On 2019-03-15 00:06, H. S. Teoh wrote:
 

 OTOH, though, these twists and turns in the development of the
 language are also an integral part of what it is; UFCS, for example,
 came about from a clever hack added in order to make built-in arrays
 compatible with the then newly-introduced range API. Later on,
 people clamored for this "hack" to be extended to other types and
 become officially accepted. It was a huge success IMO. Nowadays, I
 can't imagine writing D code without UFCS.


 
 That's not what happened. UFCS for arrays existed in D1, long before
 D2 and when ranges were introduced.


[...]

Wasn't that a parser quirk or something that was later taken advantage
of?


T

-- 
It always amuses me that Windows has a Safe Mode during bootup. Does that mean
that Windows is normally unsafe?
Mar 17 2019
prev sibling next sibling parent James Blachly <james.blachly gmail.com> writes:
On 3/14/19 5:32 PM, JN wrote:

 Also, some language features seem dubious or hardly used at all. Does 
 anyone really use contracts? I know they're sweet and Eiffel and 
 everything, but does anyone ACTUALLY use them and doesn't stick to good 
 old asserts? Perhaps rather than implementing (and supporting! each 
 feature has to work well with other features which adds support 
 overhead) such features, it'd be more useful to add features like struct 
 initialization outside of assginments, which feels to me like a much 
 nicer improvement to the language.


FWIW as a new-ish D programmer I used contracts to great effect to 
quickly write an interval tree library (which I will release soon). The 
contracts mechanism sped development tremendously, as well as caught 
several subtle edge-case errors that would have been nightmare to find 
without.

That being said, I would love struct initialization ...
Mar 14 2019
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
On Thursday, 14 March 2019 at 21:32:21 UTC, JN wrote:

 On Thursday, 14 March 2019 at 10:02:20 UTC, Paolo Invernizzi 
 wrote:

 On Thursday, 14 March 2019 at 09:25:47 UTC, Seb wrote:

 [...]


 That's interesting!

 It is no secret that I am strongly in favor of the evolution 
 of the library / language even with radical changes. And this 
 implies that I am in favor of D3 + an evolution of Phobos.

 Sociomantic has its own library, the same Weka, I would like a 
 commercial effort to support and define D3 + Phobos.

 Something similar to a common organized work group, very very 
 pragmatic. If this thing were feasible, my company would be 
 interested in contributing.

 - Paolo


 To be frank, even though I'd love the idea of D3, I don't think 
 it's just a matter of Phobos. Phobos is a victim of the 
 underlying ideology of D, which is both it's blessing and a 
 curse. That ideology is the idea that all paradigms are equal 
 and every usecase should be supported. In languages like 

 the answer is either "you can't" or "interop with C". [...]


In Java the answer has also been user java.nio.Buffer, and it 
will be improved when value types finally become a thing.


System.Runtime.InteropServices.Marshal and value types, improved 

memory allocation alongside slices and channels.



--
Paulo
Mar 15 2019
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Thursday, 14 March 2019 at 21:32:21 UTC, JN wrote:

 In D, the answer is "oh, you can totally do that, just this 
 won't work, and this is needed, and this will probably not 
 compile, oh and no one tested that". It adds a lot of cognitive 
 overhead, and unfortunately there are many points of division 
 like that.


 nogc is inherently cognitive overhead.
Mar 15 2019
prev sibling next sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Thursday, 14 March 2019 at 21:32:21 UTC, JN wrote:

 Perhaps rather than implementing (and supporting! each feature 
 has to work well with other features which adds support 
 overhead) such features, it'd be more useful to add features 
 like struct initialization outside of assginments, which feels 
 to me like a much nicer improvement to the language.


The problem when designing a language, is that *everyone* says 
"Stop wasting your time maintaining [feature I don't use], nobody 
cares about it. You should implement [feature that I want] 
instead, the language would be more popular that way!", but 
nobody can agree on which features are the "good" ones.

Eg, some people will say that -betterC is almost useless and we 
should stop worrying about it, while other people would stop 
using D if it didn't have that flag.
Mar 16 2019
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-14 22:32, JN wrote:


 To be frank, even though I'd love the idea of D3, I don't think it's 
 just a matter of Phobos. Phobos is a victim of the underlying ideology 
 of D, which is both it's blessing and a curse. That ideology is the idea 
 that all paradigms are equal and every usecase should be supported. In 

 management, the answer is either "you can't" or "interop with C". In D, 
 the answer is "oh, you can totally do that, just this won't work, and 
 this is needed, and this will probably not compile, oh and no one tested 
 that". It adds a lot of cognitive overhead, and unfortunately there are 
 many points of division like that.  nogc/RC, GC, add to that -betterC 
 too. Which one would a "new" Phobos support? I don't think you can 
 support them all. You'd have to go for the lowest common denominator, 
 which is  nogc, but if stdlib doesn't require a GC, what is the point of 
 a GC at all. 


This can be solved by pushing the allocation up the call stack. Instead 
of having a function allocate memory, pass a buffer, delegate, output 
range or something similar to the function. Then whatever you pass can 
decide if it should allocate using the GC, malloc or something else.

This will make the API not as nice to work with. But a layer on top of 
that can be built with a default allocation strategy. I would suggest 
the default allocation strategy to be the GC. If someone doesn't want to 
use the GC he/she can use the layer below and decide how the allocation 
should be done.


 Also, some language features seem dubious or hardly used at all. Does 
 anyone really use contracts? I know they're sweet and Eiffel and 
 everything, but does anyone ACTUALLY use them and doesn't stick to good 
 old asserts? Perhaps rather than implementing (and supporting! each 
 feature has to work well with other features which adds support 
 overhead) such features, it'd be more useful to add features like struct 
 initialization outside of assginments, which feels to me like a much 
 nicer improvement to the language.


I use contracts. But with the current implementation it's only a 
syntactic difference between an in-contract and a regular assert inside 
the function body. Instead, the in-contract should be called by the 
caller, not the callee.

-- 
/Jacob Carlborg
Mar 16 2019
parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Saturday, 16 March 2019 at 20:17:50 UTC, Jacob Carlborg wrote:

 Instead of having a function allocate memory, pass a buffer, 
 delegate, output range or something similar to the function. 
 Then whatever you pass can decide if it should allocate using 
 the GC, malloc or something else.


It would have to be a delegate or an output range, because in 
most cases the callee wouldn't know in advance how much storage 
it needs to allocate.

Anyway, it would be pretty nice if there were a standardized way 
to tell a function how it should allocate its memory; it would 
allow some nice strategies like "I know that I'm going to discard 
most of what this function allocates, so just put in in a 
continuous buffer; then deep-copy the relevant data to GC memory, 
and throw the rest away" (eg in a parser).

However, for that to work, you'd probably need a way to pass 
implicit arguments to a function without having to specify 
`foobar(defaultGcAllocator, x, y, z)` every time you call a 
function.
Mar 16 2019
next sibling parent James Blachly <james.blachly gmail.com> writes:
On 3/16/19 5:17 PM, Olivier FAURE wrote:

 Anyway, it would be pretty nice if there were a standardized way to tell 
 a function how it should allocate its memory; it would allow some nice 
 strategies like "I know that I'm going to discard most of what this 
 function allocates, so just put in in a continuous buffer; then 
 deep-copy the relevant data to GC memory, and throw the rest away" (eg 
 in a parser).


Wow, that is a really fascinating idea ...
Mar 16 2019
prev sibling next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 17/03/2019 10:17 AM, Olivier FAURE wrote:

 
 Anyway, it would be pretty nice if there were a standardized way to tell 
 a function how it should allocate its memory; it would allow some nice 
 strategies like "I know that I'm going to discard most of what this 
 function allocates, so just put in in a continuous buffer; then 
 deep-copy the relevant data to GC memory, and throw the rest away" (eg 
 in a parser).


I have questions related to how much garbage this parser is creating.

Ignoring how complex of an idea this actually is, we would need clear 
set of examples demonstrating that this cannot be done manually 
(choosing between GC and another allocator).

But otherwise, neat.
Mar 16 2019
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-03-16 22:17, Olivier FAURE wrote:


 However, for that to work, you'd probably need a way to pass implicit 
 arguments to a function without having to specify 
 `foobar(defaultGcAllocator, x, y, z)` every time you call a function.


With my idea that would be the second layer, sitting on top of the more 
general layer.

I guess a parameter with a default value could work:

void foobar(Allocator)(int x, int y, int z, Allocator allocator = 
defaultGcAllocator);

-- 
/Jacob Carlborg
Mar 17 2019
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 16 Mar 2019 21:17:02 +0000 schrieb Olivier FAURE:

 
 However, for that to work, you'd probably need a way to pass implicit
 arguments to a function without having to specify
 `foobar(defaultGcAllocator, x, y, z)` every time you call a function.


You can implement that using TLS globals, I think this is what 
std.experimental.allocators theAllocator is supposed to do. Of course 
you're limited in typing, you can't use templates there.

-- 
Johannes
Mar 17 2019
parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-17 12:27, Johannes Pfau wrote:


 You can implement that using TLS globals, I think this is what
 std.experimental.allocators theAllocator is supposed to do. Of course
 you're limited in typing, you can't use templates there.


Then the functions can't be pure.

-- 
/Jacob Carlborg
Mar 18 2019
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Monday, 18 March 2019 at 10:07:37 UTC, Jacob Carlborg wrote:

 On 2019-03-17 12:27, Johannes Pfau wrote:

 You can implement that using TLS globals, I think this is what
 std.experimental.allocators theAllocator is supposed to do. Of 
 course
 you're limited in typing, you can't use templates there.


 Then the functions can't be pure.


Yeah, what you want is either an annotation to tell a function 
foobar "you're pure except for *these* global variables", and 
then a way to shim the globals when calling foobar from a pure 
function, or implicit arguments (which are basically the same 
thing).
Mar 18 2019
prev sibling next sibling parent reply Sjoerd Nijboer <dlang sjoerdnijboer.com> writes:
On Saturday, 16 March 2019 at 21:17:02 UTC, Olivier FAURE wrote:

 On Saturday, 16 March 2019 at 20:17:50 UTC, Jacob Carlborg 
 wrote:

 Instead of having a function allocate memory, pass a buffer, 
 delegate, output range or something similar to the function. 
 Then whatever you pass can decide if it should allocate using 
 the GC, malloc or something else.


 It would have to be a delegate or an output range, because in 
 most cases the callee wouldn't know in advance how much storage 
 it needs to allocate.

 Anyway, it would be pretty nice if there were a standardized 
 way to tell a function how it should allocate its memory; it 
 would allow some nice strategies like "I know that I'm going to 
 discard most of what this function allocates, so just put in in 
 a continuous buffer; then deep-copy the relevant data to GC 
 memory, and throw the rest away" (eg in a parser).

 However, for that to work, you'd probably need a way to pass 
 implicit arguments to a function without having to specify 
 `foobar(defaultGcAllocator, x, y, z)` every time you call a 
 function.


How about something like the following?
It introduces an overload of keyword new, a keyword 'old' and a 
keyword 'lifetime' to pass in an IAllocator as a template 
argument to a function, and change the name mangling to add it as 
a hidden parameter.

/*MyClass is an implicit template which takes an IAllocator. It 
can always be implicitly casted to a MyClass for backwards 
compatibility. It can use this allocator using the .allocatorOf 
property.*/

class MyClass
{
   public MyClass getANEwInstance(MyClass a, MyClass b)
   {
     .... //some code


       return this.allocatorOf new MyClass();
   }

/*This function takes Myclass A as a parameter and returns a new 
instance of MyClass using thesame allocator. if no explicit 
allocator is used for a, or no allocator is known, the default 
GC_ALLOCATOR is used. This allows for better compatibility with 
existing D libraries.*/

   public lifetime(a) getANewInstanceFromParameter(lifetime 
MyClass a)
   {
     .... //some code
       return a.allocatorof new MyClass();
   }

/* a class which has a lifetime associated with it, should be 
able to decay into a "normal" class so they can still be used in 
existing codebases. They are an opt-in feature because of it. 
simpleProperty() demonstrates this. */

   public bool simpleProperty(MyClass myClass)
   {
     ....
     return myClass.someProperty();
   }
}


//usage:
public void foo(IAllocator A)()
{
   MyClass c1 = new MyClass(); //Use default GC allocator.

   MyClass c2 = Mallocator new MyClass (); // Allocate using 
malloc.

/*Instantiate MyClas using allocator A, and mangle the allocator 
A into the
resulting type as the first template parameter.*/
   MyClass c3 = A new MyClass();

/* since a reference to a class instance is still a pointer, we 
can discard the allocator of the pointer and call regular D, C++ 
and C functions with it. */
   println(c3.simpleProperty());

   // get a new object with thesame allocator as c1, since c1 is a 
 lifetime object.
   auto c4 = c1.getANEwInstance();

   auto c5 = c2.getANewInstanceFromParameter(c3);

   old c1; // no-operation
   old c2; // destruct and free c2
   old c3; // destruct and free c3 if A is not a GC allocator 
using introspection.
   old c4; /*destruct and free using thesame allocator as c1, 
meaning no-op in this case.*/
   old c5; /*destruct and free using thesame allocator as c3, 
meaning this is conditional on A being a GC allocator.*/
}
`

I think that this is an example which would be quite easy to 
learn, not a huge strain on the developers, backwards compatible 
with existing D code and potentially immensely powerfull. It 
would also give helpful info which can be used by debuggers and 
other tooling. One thing it can never do is return an object 
which was conditionally allocated with one allocator or another.
Mar 17 2019
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Sunday, 17 March 2019 at 14:28:49 UTC, Sjoerd Nijboer wrote:

 I think that this is an example which would be quite easy to 
 learn, not a huge strain on the developers, backwards 
 compatible with existing D code and potentially immensely 
 powerfull. It would also give helpful info which can be used by 
 debuggers and other tooling. One thing it can never do is 
 return an object which was conditionally allocated with one 
 allocator or another.


I'll be honest, the example you gave didn't help me understand 
your idea at all.

It looks way too complicated, and way too different from D's 
existing semantics.
Mar 18 2019
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 16 March 2019 at 21:17:02 UTC, Olivier FAURE wrote:

 On Saturday, 16 March 2019 at 20:17:50 UTC, Jacob Carlborg 
 wrote:

 Instead of having a function allocate memory, pass a buffer, 
 delegate, output range or something similar to the function. 
 Then whatever you pass can decide if it should allocate using 
 the GC, malloc or something else.


 It would have to be a delegate or an output range, because in 
 most cases the callee wouldn't know in advance how much storage 
 it needs to allocate.

 Anyway, it would be pretty nice if there were a standardized 
 way to tell a function how it should allocate its memory;


There is:

void fun(A)(auto ref A allocator);


 it would allow some nice strategies like "I know that I'm going 
 to discard most of what this function allocates, so just put in 
 in a continuous buffer; then deep-copy the relevant data to GC 
 memory, and throw the rest away" (eg in a parser).


This can all be done with a custom allocator, or one built with 
the building blocks in std.experimental.allocator.


 However, for that to work, you'd probably need a way to pass 
 implicit arguments to a function without having to specify 
 `foobar(defaultGcAllocator, x, y, z)` every time you call a 
 function.



void foobar(int x, int y, int z) {
     import std.experimental.allocator.gc_allocator: GCAllocator;
     foobar(GCAllocator.instance, x, y, z);
}

void foobar(A)(auto ref A allocator, int x, int y, int z) {
     // ...
}

Or use `theAllocator`.
Mar 19 2019
prev sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 14 March 2019 at 21:32:21 UTC, JN wrote:

 On Thursday, 14 March 2019 at 10:02:20 UTC, Paolo Invernizzi 
 wrote:

 [...]


 To be frank, even though I'd love the idea of D3, I don't think 
 it's just a matter of Phobos. Phobos is a victim of the 
 underlying ideology of D, which is both it's blessing and a 
 curse. That ideology is the idea that all paradigms are equal 
 and every usecase should be supported. In languages like 

 the answer is either "you can't" or "interop with C". In D, the 
 answer is "oh, you can totally do that, just this won't work, 
 and this is needed, and this will probably not compile, oh and 
 no one tested that". It adds a lot of cognitive overhead, and 
 unfortunately there are many points of division like that. 
  nogc/RC, GC, add to that -betterC too. Which one would a "new" 
 Phobos support? I don't think you can support them all. You'd 
 have to go for the lowest common denominator, which is  nogc, 
 but if stdlib doesn't require a GC, what is the point of a GC 
 at all. Especially since GC is an issue when porting D to 
 Webassembly (need to use -betterC for that right now). I think 
 if D3 was to be successful, it'd have to be a bit more 
 opinionated. I don't mean the language has to become a OOP 
 behemoth (I'd love that though haha) or a functional monadic 
 transformer combinator, but it'd be nice to better define a 
 scope of the language and be able to answer a question "Can D 
 do XYZ?" with "No, it's not useful and you shouldn't do it this 
 way".

 Also, some language features seem dubious or hardly used at 
 all. Does anyone really use contracts? I know they're sweet and 
 Eiffel and everything, but does anyone ACTUALLY use them and 
 doesn't stick to good old asserts? Perhaps rather than 
 implementing (and supporting! each feature has to work well 
 with other features which adds support overhead) such features, 
 it'd be more useful to add features like struct initialization 
 outside of assginments, which feels to me like a much nicer 
 improvement to the language.


There are a lot of changes, with large consensus, that are 
currently not admitted because of existing code.

Just **today** another one coming from Walter [1]:

"As mentioned earlier, scope guard will catch Errors, but it 
should really only catch Exceptions. This hasn't been fixed 
because there's existing code that relies on it catching Errors."

And the history of D is full of such examples, low hanging 
fruits, actionable.

/ Paolo


[1] 
https://forum.dlang.org/post/mailman.7569.1552776840.29801.digitalmars-d-bugs puremagic.com
Mar 17 2019
prev sibling parent James Blachly <james.blachly gmail.com> writes:
On 3/12/19 5:12 PM, Sjoerd Nijboer wrote:

 ...
 What's the view of the creators of D with regards to custom (manual or 
 gc) memory management for the standard library? C++ Makes a template of 
 about everything that could allocate just so it can extract the 
 allocator. (std::vector for instance) Will D do that too? Will D use a 
 different approach or conclude that this problem only exists for a 
 select few and not the main audience of D?


Partially related: I believe a recent (most recent?) version of D allows 
pluggable garbage collector, although I am not aware of any plugins just 
yet.
Mar 13 2019