Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

Hello,


I know it's been a long wait. Hopefully it was worth it. The alpha 
release of untyped allocators is ready for tire-kicking and a test drive.

Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

Warning: this is alpha quality. Unit tests are thin, and there are no 
benchmarks. Both would be appreciated, particularly benchmarks to 
validate the gains (which I speculate can be very sizable) of 
custom-built, special-purpose allocators compared to traditional allocators.

I acknowledge I'm clearly in no position to evaluate this design. I have 
been knocking around it for long enough to have no idea how easy it is 
to get into it from the outside, or how good it is. By all signs I could 
gather this feels like good design, and one of the best I've ever put 
together. The allocators defined have an archetypal feeling, are 
flexible both statically and dynamically, and morph and combine in 
infinite ways.

CAllocator and CAllocatorImpl make the link between the static and 
dynamic worlds. Once an allocator is assembled out of pieces and finely 
tuned, wrapping it in a dynamic API is a snap.

Please destroy! I've literally sweat as I'm sending this :o).


Andrei

"bearophile" <bearophileHUGS lycos.com> writes:

Andrei Alexandrescu:

 and morph and combine in infinite ways.

Yes, those combination capabilities look very nice :-)

I presume this module is not in "core" because it's meant to be 
used from normal D code.

- Could some new/extra D language/compiler support/features help 
this module usage safety, efficiency, expressibility or usage 
syntax (beside Issue 11331)?
- Can you create an hierarchical allocator with those?
- Are some operating system memory functions (below C malloc) 
used/supported/addable/useful for those allocators? (Including 
awareness of virtual memory management from the OS).

Bye,
bearophile

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 1:46 PM, bearophile wrote:
 Andrei Alexandrescu:

 and morph and combine in infinite ways.

 Yes, those combination capabilities look very nice :-)

 I presume this module is not in "core" because it's meant to be used
 from normal D code.

Interesting - moving to core is a possibility if we find it necessary.

 - Could some new/extra D language/compiler support/features help this
 module usage safety, efficiency, expressibility or usage syntax (beside
 Issue 11331)?

I thought about that a lot, too, while I was working on the design. 
Kinda analyzing my own going. It's generally been a pleasant experience 
- i.e. whenever I'd find myself in a bind there was some nice way out of 
it. One thing that I found myself wishing is uniformly aliasing 
something to something else. There's plenty of code like:

static if (staticallyKnownAlignment!Primary
         && staticallyKnownAlignment!Fallback)
     enum uint alignment = min(Primary.alignment, Fallback.alignment);
else
     uint alignment()
     {
         return min(primary.alignment, fallback.alignment);
     }

This is aimed at making the alignment either a statically-known constant 
or a dynamic property, depending on two type parameters. There's not a 
lot of duplication but it's quite annoying to have to write this. Would 
be great if I could just write e.g.

alias alignment = min(primary.alignment, fallback.alignment);

and have the thing just automagically do the right thing.

There have been some good idioms I used gainfully, i.e. I think the use 
of unbounded and chooseAtRuntime could become quite common.

"Turtles all the way" are awesome because they allow me to get work done 
without having to pop up and down. Consider for example:

     bool reallocate(ref void[] b, size_t newSize)
     {
         bool crossAllocatorMove(From, To)(ref From from, ref To to)
         {
             auto b1 = to.allocate(newSize);
             if (!b1) return false;
             if (b.length < newSize) b1[0 .. b.length] = b[];
             else b1[] = b[0 .. newSize];
             static if (hasMember!(From, "deallocate"))
                 from.deallocate(b);
             b = b1;
             return true;
         }

         if (primary.owns(b))
         {
             if (primary.reallocate(b, newSize)) return true;
             // Move from primary to fallback
             return crossAllocatorMove(primary, fallback);
         }
         if (fallback.reallocate(b, newSize)) return true;
         // Interesting. Move from fallback to primary.
         return crossAllocatorMove(fallback, primary);
     }

This is code that moves memory across two allocators, in either 
directions. The original version duplicated the actual work. Then I 
considered a version that would define crossAllocatorMove as a private 
method. Then I tried the above, which just worked awesomely nice.

But above all there's the property of Things Just Working (tm). Compared 
to e.g. the time when I designed ranges, there's a lot less toil for a 
lot more effect. Consider e.g. the Segregator with multiple arguments:

template Segregator(Args...) if (Args.length > 3)
{
     // Binary search
     private enum cutPoint = ((Args.length - 2) / 4) * 2;
     static if (cutPoint >= 2)
     {
         alias Segregator = .Segregator!(
             Args[cutPoint],
             .Segregator!(Args[0 .. cutPoint], Args[cutPoint + 1]),
             .Segregator!(Args[cutPoint + 2 .. $])
         );
     }
     else
     {
         // Favor small sizes
         alias Segregator = .Segregator!(
             Args[0],
             Args[1],
             .Segregator!(Args[2 .. $])
         );
     }

     // Linear search
     //alias Segregator = .Segregator!(
     //    Args[0], Args[1],
     //    .Segregator!(Args[2 .. $])
     //);
}

Granted, it's not easy to get into, but can't be made much easier 
because it does something highly non-trivial: a left-leaning binary 
search through a subset of the template arguments. That's in order to 
find the best size class to allocate. (Linear version shown in comment, 
probably we should enable that with a policy.) Such code would have been 
much harder to get going in the past because of random bugs and 
limitations in the compiler and inscrutable error messages.

The entire code base stands at 4000 lines, documentation and all, yet 
allows for a lot of allocator designs.

 - Can you create an hierarchical allocator with those?

Not sure what that means (I recall the phrase has been discussed but 
forgot what it means).

 - Are some operating system memory functions (below C malloc)
 used/supported/addable/useful for those allocators? (Including awareness
 of virtual memory management from the OS).

Definitely!

* http://stackoverflow.com/questions/3839922/aligned-malloc-in-gcc comes 
to mind.

* http://en.wikipedia.org/wiki/Sbrk

* http://en.wikipedia.org/wiki/Mmap

These would be implemented as top allocators (a la Mallocator). On top 
thereof, nice block allocators etc. can be easily defined.

Also the final version should define a few convenience types that are 
pre-assembled allocators that are known to be useful: heaps, reaps, 
freelist batteries come to mind.


Andrei

"bearophile" <bearophileHUGS lycos.com> writes:

Andrei Alexandrescu:

 - Can you create an hierarchical allocator with those?

 Not sure what that means (I recall the phrase has been 
 discussed but forgot what it means).

I meant something like this:




 (Including awareness of virtual memory management from the OS).

 Definitely!

 * 
 http://stackoverflow.com/questions/3839922/aligned-malloc-in-gcc 
 comes to mind.

 * http://en.wikipedia.org/wiki/Sbrk

 * http://en.wikipedia.org/wiki/Mmap

By that "awareness" I meant something like what the Azul JavaVM 
does, here a panoramic view of similar ideas:

http://web.eece.maine.edu/~jyue/papers/iccci.pdf

Bye,
bearophile

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 4:01 PM, bearophile wrote:
 Andrei Alexandrescu:

 - Can you create an hierarchical allocator with those?

 Not sure what that means (I recall the phrase has been discussed but
 forgot what it means).

 I meant something like this:



I see. These are akin to regions to some extent, just a bit more 
dynamic. The design doesn't explicitly address the pattern, so I expect 
there would be some hiccups.

 (Including awareness of virtual memory management from the OS).

 Definitely!

 * http://stackoverflow.com/questions/3839922/aligned-malloc-in-gcc
 comes to mind.

 * http://en.wikipedia.org/wiki/Sbrk

 * http://en.wikipedia.org/wiki/Mmap

 By that "awareness" I meant something like what the Azul JavaVM does,
 here a panoramic view of similar ideas:

 http://web.eece.maine.edu/~jyue/papers/iccci.pdf

Once we get to types and garbage, such stuff will become more relevant.


Andrei

Walter Bright <newshound2 digitalmars.com> writes:

On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha release of
 untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

I'm impressed. I looked for 3 or 4 must haves, and they were there! It covers 
all the cases I've written custom allocators for.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 2:08 PM, Walter Bright wrote:
 On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of
 untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 I'm impressed. I looked for 3 or 4 must haves, and they were there! It
 covers all the cases I've written custom allocators for.

<big sigh of relief>


Andrei

"ChrisG" <christopher.gassib gmail.com> writes:

On Thursday, 24 October 2013 at 21:17:45 UTC, Andrei Alexandrescu 
wrote:
 On 10/24/13 2:08 PM, Walter Bright wrote:
 On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The 
 alpha
 release of
 untyped allocators is ready for tire-kicking and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 I'm impressed. I looked for 3 or 4 must haves, and they were 
 there! It
 covers all the cases I've written custom allocators for.

 <big sigh of relief>


 Andrei

So, is the idea to add an optional allocator template parameter 
to various objects in the phobos library, or is this only 
intended to be used standalone via the with statement?

-Chris

Artem Tarasov <lomereiter gmail.com> writes:

The design looks great. Easy to grasp and covers almost all imaginable use
cases.

'Almost' because it would be nice to also have ScopeAllocator!N ~
FallbackAllocator!(InsituRegion!N, Mallocator) which would automatically
deallocate all heap-allocated blocks when exiting the scope (i.e. no
deallocate method, array of pointers to be freed is kept on a C heap).

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 2:16 PM, Artem Tarasov wrote:
 The design looks great. Easy to grasp and covers almost all imaginable
 use cases.

 'Almost' because it would be nice to also have ScopeAllocator!N ~
 FallbackAllocator!(InsituRegion!N, Mallocator) which would automatically
 deallocate all heap-allocated blocks when exiting the scope (i.e. no
 deallocate method, array of pointers to be freed is kept on a C heap).

Great point, clearly you cut to the chase :o). Yes, I hope to integrate 
that, and I'm a bit pained I didn't have the time to do that.

The way I see that could be done is by defining a ScopedAllocator that 
uses internally an AffixAllocator with a doubly-linked list tracking all 
allocations. In the destructor, ScopedAllocator would walk the list and 
deallocate everything allocated since its construction.

Andrei

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Thursday, 24 October 2013 at 21:20:02 UTC, Andrei Alexandrescu 
wrote:
 The way I see that could be done is by defining a 
 ScopedAllocator that uses internally an AffixAllocator with a 
 doubly-linked list tracking all allocations. In the destructor, 
 ScopedAllocator would walk the list and deallocate everything 
 allocated since its construction.

Can this be generalized as a proxy allocator which simply 
implements deallocateAll, on top of its parent allocator's 
primitives?

The list can be singly-linked if random deallocation is not 
needed.

"Namespace" <rswhite4 googlemail.com> writes:

Awesome! Will Appender get an option to use a suitable allocator?

"Namespace" <rswhite4 googlemail.com> writes:

On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?

A dream of me, that will probably never come true, would be also 
something like this:
----
with (Mallocator) {
     int[] arr;
     arr ~= 42; /// will use Mallocator.it.allocate internal
}
----

Andrej Mitrovic <andrej.mitrovich gmail.com> writes:

On 10/24/13, Namespace <rswhite4 googlemail.com> wrote:
 A dream of me, that will probably never come true, would be also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

Actually this got me thinking..

import core.memory;

GC.useAllocator!Mallocator;
int[] arr;
arr ~= 42;

This could be a useful test-bed for the allocators by using them with
existing D applications and libraries that depend on the GC (most apps
I'd assume).

"Namespace" <rswhite4 googlemail.com> writes:

On Thursday, 24 October 2013 at 21:58:36 UTC, Andrej Mitrovic 
wrote:
 On 10/24/13, Namespace <rswhite4 googlemail.com> wrote:
 A dream of me, that will probably never come true, would be 
 also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

 Actually this got me thinking..

 import core.memory;

 GC.useAllocator!Mallocator;
 int[] arr;
 arr ~= 42;

 This could be a useful test-bed for the allocators by using 
 them with
 existing D applications and libraries that depend on the GC 
 (most apps
 I'd assume).

That would be really awesome. I would test it immediately and 
extensively. :)

"Namespace" <rswhite4 googlemail.com> writes:

On Thursday, 24 October 2013 at 22:22:53 UTC, Namespace wrote:
 On Thursday, 24 October 2013 at 21:58:36 UTC, Andrej Mitrovic 
 wrote:
 On 10/24/13, Namespace <rswhite4 googlemail.com> wrote:
 A dream of me, that will probably never come true, would be 
 also
 something like this:
 ----
 with (Mallocator) {
     int[] arr;
     arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

 Actually this got me thinking..

 import core.memory;

 GC.useAllocator!Mallocator;
 int[] arr;
 arr ~= 42;

 This could be a useful test-bed for the allocators by using 
 them with
 existing D applications and libraries that depend on the GC 
 (most apps
 I'd assume).

 That would be really awesome. I would test it immediately and 
 extensively. :)

Another idea would be that every built-in array has a Allocator 
property which can be set:
----
int[] arr;
arr.allocator = Mallocator;
----

Marco Leise <Marco.Leise gmx.de> writes:

Am Fri, 25 Oct 2013 00:33:40 +0200
schrieb "Namespace" <rswhite4 googlemail.com>:

 Another idea would be that every built-in array has a Allocator 
 property which can be set:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----

The type of arr.allocator would have to be CAllocator to allow
runtime swapping of allocators with polymorphism.

Would you'd implement it like .capacity, as a virtual property
that doesn't take up space in the slice?

-- 
Marco

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 07:50:00 UTC, Marco Leise wrote:
 Am Fri, 25 Oct 2013 00:33:40 +0200
 schrieb "Namespace" <rswhite4 googlemail.com>:

 Another idea would be that every built-in array has a 
 Allocator property which can be set:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----

 The type of arr.allocator would have to be CAllocator to allow
 runtime swapping of allocators with polymorphism.

 Would you'd implement it like .capacity, as a virtual property
 that doesn't take up space in the slice?

Yes I would, but it's not up to me. But I would prefer this 
solution because it is more flexible IMO.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable allocator?

 A dream of me, that will probably never come true, would be also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

Oddly enough this can be actually done.

with (setAllocator!Mallocator)
{
    ...
}

setAllcator returns an rvalue that changes the global allocator to the 
Mallocator in the constructor, and restores it to whatever it was in the 
destructor.


Andrei

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu 
wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?

 A dream of me, that will probably never come true, would be 
 also
 something like this:
 ----
 with (Mallocator) {
     int[] arr;
     arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global allocator 
 to the Mallocator in the constructor, and restores it to 
 whatever it was in the destructor.


 Andrei

Are you saying that this code:
----
with (setAllocator!Mallocator) {
     int[] arr;
     arr ~= 42; [1]
}
----
use the mallocator for [1]? So no GC memory is needed?

Another examaple:
----
with (setAllocator!ScopeAllocator) {
     int[] arr;
     arr ~= 42; [1]
}
----
Did [1] use ScopeAllocator for memory allocation? And is the 
memory of arr automatically collected at the end of the scope 
with ScopeAllocator?

"Maxim Fomin" <maxim maxim-fomin.ru> writes:

On Friday, 25 October 2013 at 07:19:48 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu 
 wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?

 A dream of me, that will probably never come true, would be 
 also
 something like this:
 ----
 with (Mallocator) {
    int[] arr;
    arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
   ...
 }

 setAllcator returns an rvalue that changes the global 
 allocator to the Mallocator in the constructor, and restores 
 it to whatever it was in the destructor.


 Andrei

 Are you saying that this code:
 ----
 with (setAllocator!Mallocator) {
     int[] arr;
     arr ~= 42; [1]
 }
 ----
 use the mallocator for [1]? So no GC memory is needed?

 Another examaple:
 ----
 with (setAllocator!ScopeAllocator) {
     int[] arr;
     arr ~= 42; [1]
 }
 ----
 Did [1] use ScopeAllocator for memory allocation? And is the 
 memory of arr automatically collected at the end of the scope 
 with ScopeAllocator?

This is impossible because with() has already another meaning. 
However nothing stops from inventing another name for this. 
Compiler can insert calls to druntime as well to other context 
dependent functions. After allocator design is defined we can 
move on to step 2, implementing this idea.

"Namespace" <rswhite4 googlemail.com> writes:

 Are you saying that this code:
 ----
 with (setAllocator!Mallocator) {
    int[] arr;
    arr ~= 42; [1]
 }
 ----
 use the mallocator for [1]? So no GC memory is needed?

 Another examaple:
 ----
 with (setAllocator!ScopeAllocator) {
    int[] arr;
    arr ~= 42; [1]
 }
 ----
 Did [1] use ScopeAllocator for memory allocation? And is the 
 memory of arr automatically collected at the end of the scope 
 with ScopeAllocator?

 This is impossible because with() has already another meaning. 
 However nothing stops from inventing another name for this. 
 Compiler can insert calls to druntime as well to other context 
 dependent functions. After allocator design is defined we can 
 move on to step 2, implementing this idea.

You mena something like this?
----
use(Mallocator) {
     int[] arr;
     arr ~= 42;
}
----
Or did I understand you wrong?

Whats about the virtual property idea, that every array has 
internal an allocator? Wouldn't it be easier to implement such a 
thing?

"Maxim Fomin" <maxim maxim-fomin.ru> writes:

On Friday, 25 October 2013 at 08:13:21 UTC, Namespace wrote:
 Are you saying that this code:
 ----
 with (setAllocator!Mallocator) {
   int[] arr;
   arr ~= 42; [1]
 }
 ----
 use the mallocator for [1]? So no GC memory is needed?

 Another examaple:
 ----
 with (setAllocator!ScopeAllocator) {
   int[] arr;
   arr ~= 42; [1]
 }
 ----
 Did [1] use ScopeAllocator for memory allocation? And is the 
 memory of arr automatically collected at the end of the scope 
 with ScopeAllocator?

 This is impossible because with() has already another meaning. 
 However nothing stops from inventing another name for this. 
 Compiler can insert calls to druntime as well to other context 
 dependent functions. After allocator design is defined we can 
 move on to step 2, implementing this idea.

 You mena something like this?
 ----
 use(Mallocator) {
     int[] arr;
     arr ~= 42;
 }
 ----
 Or did I understand you wrong?

It depends on how Mallocator is related to integer array (which 
again boils down to allocator design). If it is appropriate, then 
yes.

 Whats about the virtual property idea, that every array has 
 internal an allocator? Wouldn't it be easier to implement such 
 a thing?

Please provide example.

"Namespace" <rswhite4 googlemail.com> writes:

 You mena something like this?
 ----
 use(Mallocator) {
    int[] arr;
    arr ~= 42;
 }
 ----
 Or did I understand you wrong?

 It depends on how Mallocator is related to integer array (which 
 again boils down to allocator design). If it is appropriate, 
 then yes.

 Whats about the virtual property idea, that every array has 
 internal an allocator? Wouldn't it be easier to implement such 
 a thing?

 Please provide example.

Something like that: 
http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=3#post-pfoxyfzyjxqcqwnvgnpi:40forum.dlang.org

Every array has an internal allocator property which can be reset:
----
int[] arr;
arr.allocator = Mallocator;
----

or

----
int[] arr;
arr.useAllocator(Mallocator);
----

But maybe a design without some alias notation would be more 
preferable:
----
{
     ScopeAllocator m;
     int[] arr;
     arr.useAllocator(m);

     arr ~= 42; /// Use m.allocate
} /// end of scope: ScopeAllocator collects all remaining memory.
----

And:
----
int[] arr;
assert(arr is null);
{
     ScopeAllocator m;
     arr.useAllocator(m);

     arr ~= 42; /// Use m.allocate
} /// end of scope: ScopeAllocator collects all remaining memory.
assert(arr is null);
----

"Maxim Fomin" <maxim maxim-fomin.ru> writes:

On Friday, 25 October 2013 at 08:27:52 UTC, Namespace wrote:
 Something like that: 
 http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=3#post-pfoxyfzyjxqcqwnvgnpi:40forum.dlang.org

 Every array has an internal allocator property which can be 
 reset:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----

 or

 ----
 int[] arr;
 arr.useAllocator(Mallocator);
 ----

That's doable.

 But maybe a design without some alias notation would be more 
 preferable:
 ----
 {
     ScopeAllocator m;
     int[] arr;
     arr.useAllocator(m);

     arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 ----

 And:
 ----
 int[] arr;
 assert(arr is null);
 {
     ScopeAllocator m;
     arr.useAllocator(m);

     arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 assert(arr is null);
 ----

That's also doable. TypeInfo will be bloated more and there would 
be cost of some sort of scope exit, and, ideally, a check that 
reference does not escape, but this is doable.

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 08:50:23 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 08:27:52 UTC, Namespace wrote:
 Something like that: 
 http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=3#post-pfoxyfzyjxqcqwnvgnpi:40forum.dlang.org

 Every array has an internal allocator property which can be 
 reset:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----

 or

 ----
 int[] arr;
 arr.useAllocator(Mallocator);
 ----

 That's doable.

 But maybe a design without some alias notation would be more 
 preferable:
 ----
 {
    ScopeAllocator m;
    int[] arr;
    arr.useAllocator(m);

    arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 ----

 And:
 ----
 int[] arr;
 assert(arr is null);
 {
    ScopeAllocator m;
    arr.useAllocator(m);

    arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 assert(arr is null);
 ----

 That's also doable. TypeInfo will be bloated more and there 
 would be cost of some sort of scope exit, and, ideally, a check 
 that reference does not escape, but this is doable.

A dream could come true.

"Maxim Fomin" <maxim maxim-fomin.ru> writes:

On Friday, 25 October 2013 at 08:53:00 UTC, Namespace wrote:
 A dream could come true.

It depends on how Andrei design allocators, Walter's willingness 
to accept changes and developers to implement it.

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 09:26:46 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 08:53:00 UTC, Namespace wrote:
 A dream could come true.

 It depends on how Andrei design allocators, Walter's 
 willingness to accept changes and developers to implement it.

Therefore the 'could'. But if this would be implemented, I'm sure 
that this would give D good advantages to be a real opponent for 
C++.

We would have then the possibility to manage our memory by 
ourself. One of D's promises is, that the GC can be disabled. 
Yes, it can, but then we have many many things which do not work. 
For example built-in arrays. With the ability of allocators the 
promise could come true.

"Maxim Fomin" <maxim maxim-fomin.ru> writes:

On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be disabled. 
 Yes, it can, but then we have many many things which do not 
 work. For example built-in arrays.

Not only arrays, but classes, throwables, scope exits, new 
operator, nested structs, etc.

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be disabled. 
 Yes, it can, but then we have many many things which do not 
 work. For example built-in arrays.

 Not only arrays, but classes, throwables, scope exits, new 
 operator, nested structs, etc.

Thats right.
But I often use temporary arrays, but I still don't like them 
because they are always consume so much GC memory. But with 
allocators that would end.
Let us hope that Walter has the right intention and that Andrei 
design the allocators for this purpose.

"Jakob Ovrum" <jakobovrum gmail.com> writes:

On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be disabled. 
 Yes, it can, but then we have many many things which do not 
 work. For example built-in arrays.

 Not only arrays, but classes, throwables, scope exits, new 
 operator, nested structs, etc.

 Thats right.
 But I often use temporary arrays, but I still don't like them 
 because they are always consume so much GC memory. But with 
 allocators that would end.
 Let us hope that Walter has the right intention and that Andrei 
 design the allocators for this purpose.

Why does it have to be the opaque druntime dynamic array? Why 
can't you use the hypothetical (planned, rather) 
std.container.Array that supports custom allocators, or a type of 
your own design?

Garbage collected memory management is the only memory management 
paradigm that supports the infinite lifetime model (GC includes 
automatic reference counting). Swapping it out for a different 
memory management technique while libraries or parts of the 
program still rely on infinite lifetime for parts such as the 
slice concatenation operators and closures is disastrous.

The best strategy for avoiding the global GC is to not use the 
language features associated with infinite lifetime, which 
includes slice concatenation and the `new` operator. For implicit 
allocations like closures, we don't have much of a choice, so 
here the capability is useful, but for slice operators? Just use 
a different type!

Currently the `new` operator has capabilities that function calls 
cannot replicate - namely non-static nested class construction 
and construction of classes with non-public constructors - but 
these issues have to be solved anyway to support allocators in 
general.

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 11:21:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be 
 disabled. Yes, it can, but then we have many many things 
 which do not work. For example built-in arrays.

 Not only arrays, but classes, throwables, scope exits, new 
 operator, nested structs, etc.

 Thats right.
 But I often use temporary arrays, but I still don't like them 
 because they are always consume so much GC memory. But with 
 allocators that would end.
 Let us hope that Walter has the right intention and that 
 Andrei design the allocators for this purpose.

 Why does it have to be the opaque druntime dynamic array? Why 
 can't you use the hypothetical (planned, rather) 
 std.container.Array that supports custom allocators, or a type 
 of your own design?

Because Array!int looks a lot more ugly than such a nice thing as 
int[]. And if it is possible to change the allocator for some 
arrays, why shouldn't we implement it?
The default allocator would stay the GC allocator. So if you 
don't want to swap the allocator of your arrays, don't do it.

"Jakob Ovrum" <jakobovrum gmail.com> writes:

On Friday, 25 October 2013 at 12:07:30 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 11:21:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be 
 disabled. Yes, it can, but then we have many many things 
 which do not work. For example built-in arrays.

 Not only arrays, but classes, throwables, scope exits, new 
 operator, nested structs, etc.

 Thats right.
 But I often use temporary arrays, but I still don't like them 
 because they are always consume so much GC memory. But with 
 allocators that would end.
 Let us hope that Walter has the right intention and that 
 Andrei design the allocators for this purpose.

 Why does it have to be the opaque druntime dynamic array? Why 
 can't you use the hypothetical (planned, rather) 
 std.container.Array that supports custom allocators, or a type 
 of your own design?

 Because Array!int looks a lot more ugly than such a nice thing 
 as int[].

If the template syntax is too ugly then we've really failed at 
designing an extensible development platform. Even so, with type 
inference and aliases, the need to write down involved names is 
all but eliminated (not that I think Array!int is a particularly 
involved name).

Not everything belongs in the core language. Conflating slices 
with garbage collected dynamic arrays is a mistake we have to 
live with, but let's not make the situation even more complicated.

 And if it is possible to change the allocator for some arrays, 
 why shouldn't we implement it?

Because it has a significant cost.

 The default allocator would stay the GC allocator. So if you 
 don't want to swap the allocator of your arrays, don't do it.

That's fine as long as you only use the array locally and don't 
pass it to any code that depends on infinite lifetime semantics. 
It also enables circumvention of SafeD unless some language rules 
are changed.

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 14:09:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 12:07:30 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 11:21:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin 
 wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's promises is, that the GC can be 
 disabled. Yes, it can, but then we have many many things 
 which do not work. For example built-in arrays.

 Not only arrays, but classes, throwables, scope exits, new 
 operator, nested structs, etc.

 Thats right.
 But I often use temporary arrays, but I still don't like 
 them because they are always consume so much GC memory. But 
 with allocators that would end.
 Let us hope that Walter has the right intention and that 
 Andrei design the allocators for this purpose.

 Why does it have to be the opaque druntime dynamic array? Why 
 can't you use the hypothetical (planned, rather) 
 std.container.Array that supports custom allocators, or a 
 type of your own design?

 Because Array!int looks a lot more ugly than such a nice thing 
 as int[].

 If the template syntax is too ugly then we've really failed at 
 designing an extensible development platform. Even so, with 
 type inference and aliases, the need to write down involved 
 names is all but eliminated (not that I think Array!int is a 
 particularly involved name).

 Not everything belongs in the core language. Conflating slices 
 with garbage collected dynamic arrays is a mistake we have to 
 live with, but let's not make the situation even more 
 complicated.

I don't get your problem.

 And if it is possible to change the allocator for some arrays, 
 why shouldn't we implement it?

 Because it has a significant cost.

Which cost?

 The default allocator would stay the GC allocator. So if you 
 don't want to swap the allocator of your arrays, don't do it.

 That's fine as long as you only use the array locally and don't 
 pass it to any code that depends on infinite lifetime 
 semantics. It also enables circumvention of SafeD unless some 
 language rules are changed.

The language cannot protect you from every mistake you can do. 
You should know what you do. If you don't, don't use such a 
feature. It's very simple.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 5:07 AM, Namespace wrote:
 On Friday, 25 October 2013 at 11:21:55 UTC, Jakob Ovrum wrote:
 On Friday, 25 October 2013 at 10:02:08 UTC, Namespace wrote:
 On Friday, 25 October 2013 at 09:51:40 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 09:37:23 UTC, Namespace wrote:
 We would have then the possibility to manage our memory by ourself.
 One of D's promises is, that the GC can be disabled. Yes, it can,
 but then we have many many things which do not work. For example
 built-in arrays.

 Not only arrays, but classes, throwables, scope exits, new operator,
 nested structs, etc.

 Thats right.
 But I often use temporary arrays, but I still don't like them because
 they are always consume so much GC memory. But with allocators that
 would end.
 Let us hope that Walter has the right intention and that Andrei
 design the allocators for this purpose.

 Why does it have to be the opaque druntime dynamic array? Why can't
 you use the hypothetical (planned, rather) std.container.Array that
 supports custom allocators, or a type of your own design?

 Because Array!int looks a lot more ugly than such a nice thing as int[].
 And if it is possible to change the allocator for some arrays, why
 shouldn't we implement it?
 The default allocator would stay the GC allocator. So if you don't want
 to swap the allocator of your arrays, don't do it.

Maintaining per-array-object allocators could be quite expensive. Two 
other possibilities are keeping one allocators for all array of a given 
type and of course keeping a global allocator for all arrays.

Andrei

"Namespace" <rswhite4 googlemail.com> writes:

 Because Array!int looks a lot more ugly than such a nice thing 
 as int[].
 And if it is possible to change the allocator for some arrays, 
 why
 shouldn't we implement it?
 The default allocator would stay the GC allocator. So if you 
 don't want
 to swap the allocator of your arrays, don't do it.

 Maintaining per-array-object allocators could be quite 
 expensive. Two other possibilities are keeping one allocators 
 for all array of a given type and of course keeping a global 
 allocator for all arrays.

 Andrei

Sounds also nice. As long as you can customize the way built-in 
arrays allocate, I'm fine with that. :)

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On 25/10/13 11:37, Namespace wrote:
 We would have then the possibility to manage our memory by ourself. One of D's
 promises is, that the GC can be disabled. Yes, it can, but then we have many
 many things which do not work. For example built-in arrays. With the ability of
 allocators the promise could come true.

That's something I'd really like to know more about.

My D code almost invariably works with the "natural" way to handle memory in D, 
which is to use "new" where needed, plus stuff like array appending ~ and 
alterations to array lengths, with all allocations handled behind the scenes by 
the GC.  I've always felt bad about the fact that this therefore imposes use of 
the GC on anyone who uses my code.  It would be great if one could just write 
idiomatic D code and know that others using it could dictate different 
memory-management strategies and have them "just work".

"Namespace" <rswhite4 googlemail.com> writes:

On Saturday, 26 October 2013 at 15:42:48 UTC, Joseph Rushton 
Wakeling wrote:
 On 25/10/13 11:37, Namespace wrote:
 We would have then the possibility to manage our memory by 
 ourself. One of D's
 promises is, that the GC can be disabled. Yes, it can, but 
 then we have many
 many things which do not work. For example built-in arrays. 
 With the ability of
 allocators the promise could come true.

 That's something I'd really like to know more about.

 My D code almost invariably works with the "natural" way to 
 handle memory in D, which is to use "new" where needed, plus 
 stuff like array appending ~ and alterations to array lengths, 
 with all allocations handled behind the scenes by the GC.  I've 
 always felt bad about the fact that this therefore imposes use 
 of the GC on anyone who uses my code.  It would be great if one 
 could just write idiomatic D code and know that others using it 
 could dictate different memory-management strategies and have 
 them "just work".

Read on from here to find out more:
http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=5#post-uqolhuqqygquxnaxahkz:40forum.dlang.org

:)

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On 26/10/13 17:46, Namespace wrote:
 Read on from here to find out more:
 http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=5#post-uqolhuqqygquxnaxahkz:40forum.dlang.org

I already did, and if I understand right, things are looking good ... but I 
wanted to be sure whether or not I understand :-)

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 08:50:23 UTC, Maxim Fomin wrote:
 On Friday, 25 October 2013 at 08:27:52 UTC, Namespace wrote:
 Something like that: 
 http://forum.dlang.org/thread/l4btsk$5u8$1 digitalmars.com?page=3#post-pfoxyfzyjxqcqwnvgnpi:40forum.dlang.org

 Every array has an internal allocator property which can be 
 reset:
 ----
 int[] arr;
 arr.allocator = Mallocator;
 ----

 or

 ----
 int[] arr;
 arr.useAllocator(Mallocator);
 ----

 That's doable.

 But maybe a design without some alias notation would be more 
 preferable:
 ----
 {
    ScopeAllocator m;
    int[] arr;
    arr.useAllocator(m);

    arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 ----

 And:
 ----
 int[] arr;
 assert(arr is null);
 {
    ScopeAllocator m;
    arr.useAllocator(m);

    arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 assert(arr is null);
 ----

 That's also doable. TypeInfo will be bloated more and there 
 would be cost of some sort of scope exit, and, ideally, a check 
 that reference does not escape, but this is doable.

Why the cost of scope(exit)? If ScopeAllocator m get out of 
scope, the DTor is called and ideally the collect method is 
called. That's all. Or am I wrong?

"Maxim Fomin" <maxim maxim-fomin.ru> writes:

On Friday, 25 October 2013 at 09:13:45 UTC, Namespace wrote:
 But maybe a design without some alias notation would be more 
 preferable:
 ----
 {
   ScopeAllocator m;
   int[] arr;
   arr.useAllocator(m);

   arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 ----

 And:
 ----
 int[] arr;
 assert(arr is null);
 {
   ScopeAllocator m;
   arr.useAllocator(m);

   arr ~= 42; /// Use m.allocate
 } /// end of scope: ScopeAllocator collects all remaining 
 memory.
 assert(arr is null);
 ----

 That's also doable. TypeInfo will be bloated more and there 
 would be cost of some sort of scope exit, and, ideally, a 
 check that reference does not escape, but this is doable.

 Why the cost of scope(exit)? If ScopeAllocator m get out of 
 scope, the DTor is called and ideally the collect method is 
 called. That's all. Or am I wrong?

Probably because there may be exception thrown between appending 
to arr and end of block. In this particular case compiler may be 
smart enough to optimize it away, but dmd optimization 
capabilities and nothrow is a separate story.

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Friday, 25 October 2013 at 08:09:30 UTC, Maxim Fomin wrote:
 This is impossible because with() has already another meaning.

I believe Andrei meant using the existing "with" statement as it 
currently is defined in the language. The function setAllocator 
would set the given allocator, and return a structure with a 
destructor. The destructor should be called at the end of the 
"with" block, and it will set the current allocator back to its 
old value.

However, it appears that currently this usage of "with" is buggy:
http://d.puremagic.com/issues/show_bug.cgi?id=11351

Jacob Carlborg <doob me.com> writes:

On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
     ...
 }

 setAllcator returns an rvalue that changes the global allocator to the
 Mallocator in the constructor, and restores it to whatever it was in the
 destructor.

Wouldn't this be very unsafe? Say you call another function inside the 
with-statement and that function assumes the standard GC for allocating 
memory.

-- 
/Jacob Carlborg

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

25-Oct-2013 16:52, Jacob Carlborg пишет:
 On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
     ...
 }

 setAllcator returns an rvalue that changes the global allocator to the
 Mallocator in the constructor, and restores it to whatever it was in the
 destructor.

 Wouldn't this be very unsafe? Say you call another function inside the
 with-statement and that function assumes the standard GC for allocating
 memory.

Very true. To put it simply it's a disastrous idea that sadly is too 
easy to be ignored.

IMHO we'd better start with containers and refitting Phobos from 
built-in AA/arrays to user-defined containers. One interesting way there 
is to accept/adopt containers as OutputRange.

-- 
Dmitry Olshansky

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky 
wrote:
 25-Oct-2013 16:52, Jacob Carlborg пишет:
 On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global 
 allocator to the
 Mallocator in the constructor, and restores it to whatever it 
 was in the
 destructor.

 Wouldn't this be very unsafe? Say you call another function 
 inside the
 with-statement and that function assumes the standard GC for 
 allocating
 memory.

 Very true. To put it simply it's a disastrous idea that sadly 
 is too easy to be ignored.

 IMHO we'd better start with containers and refitting Phobos 
 from built-in AA/arrays to user-defined containers. One 
 interesting way there is to accept/adopt containers as 
 OutputRange.

Did you mean to get rid of built-in arrays / kill int[] and 
replace it with Array!T?

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

25-Oct-2013 22:41, Namespace пишет:
 On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky wrote:
 25-Oct-2013 16:52, Jacob Carlborg пишет:
 On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global allocator to the
 Mallocator in the constructor, and restores it to whatever it was in
 the
 destructor.

 Wouldn't this be very unsafe? Say you call another function inside the
 with-statement and that function assumes the standard GC for allocating
 memory.

 Very true. To put it simply it's a disastrous idea that sadly is too
 easy to be ignored.

 IMHO we'd better start with containers and refitting Phobos from
 built-in AA/arrays to user-defined containers. One interesting way
 there is to accept/adopt containers as OutputRange.

 Did you mean to get rid of built-in arrays / kill int[] and replace it
 with Array!T?

Hm, arrays? Kill? No, they are incredibly nice for prototyping + they 
are very useful even as just slices.

What I mean is to make it easy to use Phobos stuff with other containers 
in place of built-ins, basically no hard-codding behind the scenes.

Typical offender is Appender - it's rigid, has dangerous API and doesn't 
support building anything but T[]. For this particular case see: 
http://d.puremagic.com/issues/show_bug.cgi?id=11138

Another one:

It's trivially expendable to any other Array-like type, yet ATM it's 
hardwired.

O.T. I'd gladly kill built-in AA though just to save people a lot of 
time spent on debugging that crap. More precisely I'd keep AA _literals_ 
and give the user the means to construct any type of Key-->Value store 
out of it. It's too late probably.

-- 
Dmitry Olshansky

"Namespace" <rswhite4 googlemail.com> writes:

On Friday, 25 October 2013 at 19:03:14 UTC, Dmitry Olshansky 
wrote:
 25-Oct-2013 22:41, Namespace пишет:
 On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky 
 wrote:
 25-Oct-2013 16:52, Jacob Carlborg пишет:
 On 2013-10-25 02:01, Andrei Alexandrescu wrote:

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
   ...
 }

 setAllcator returns an rvalue that changes the global 
 allocator to the
 Mallocator in the constructor, and restores it to whatever 
 it was in
 the
 destructor.

 Wouldn't this be very unsafe? Say you call another function 
 inside the
 with-statement and that function assumes the standard GC for 
 allocating
 memory.

 Very true. To put it simply it's a disastrous idea that sadly 
 is too
 easy to be ignored.

 IMHO we'd better start with containers and refitting Phobos 
 from
 built-in AA/arrays to user-defined containers. One 
 interesting way
 there is to accept/adopt containers as OutputRange.

 Did you mean to get rid of built-in arrays / kill int[] and 
 replace it
 with Array!T?

 Hm, arrays? Kill? No, they are incredibly nice for prototyping 
 + they are very useful even as just slices.

 What I mean is to make it easy to use Phobos stuff with other 
 containers in place of built-ins, basically no hard-codding 
 behind the scenes.

 Typical offender is Appender - it's rigid, has dangerous API 
 and doesn't support building anything but T[]. For this 
 particular case see: 
 http://d.puremagic.com/issues/show_bug.cgi?id=11138

 Another one:

 It's trivially expendable to any other Array-like type, yet ATM 
 it's hardwired.

 O.T. I'd gladly kill built-in AA though just to save people a 
 lot of time spent on debugging that crap. More precisely I'd 
 keep AA _literals_ and give the user the means to construct any 
 type of Key-->Value store out of it. It's too late probably.

With which syntax? As far as e.g. int[string] (and not something 
ugly as Map!(string, int)) would stay what would be the problem 
to change the backend?

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

25-Oct-2013 23:26, Namespace пишет:
 On Friday, 25 October 2013 at 19:03:14 UTC, Dmitry Olshansky wrote:
 O.T. I'd gladly kill built-in AA though just to save people a lot of
 time spent on debugging that crap. More precisely I'd keep AA
 _literals_ and give the user the means to construct any type of
 Key-->Value store out of it. It's too late probably.

 With which syntax? As far as e.g. int[string] (and not something ugly as
 Map!(string, int)) would stay what would be the problem to change the
 backend?

Map!(string,int) is fine actually as I'm no syntax fan anyway.

It's not the problem with "backend" as much as with its state and the 
interface it imposes:

a) It's stuck in between compiler intrinsic and library artifact. Being 
neither it has weaknesses of both (well that's fixable).
b) Interface... I don't even know where to start. But first of all: it's 
a hash table and can't be anything else - see the rehash function. More 
interestingly it escapes pointers ('in' operator) that must stay valid 
until another insert. That's a tough constraint on hash table (HT) 
implementation. Being HT it doesn't expose nor allows to change the 
load-factor.
c) Built-in HTs are one size fits all. Needless to say - it doesn't 
always fit and in particular there are many variations that trade 
insert/remove and/or space-efficiency for faster lookup and vise-versa. 
Also to stay on topic - memory management / deterministic cleanup? 
Trivial in UDT but not in built-ins.
d) AA literals are of no use the moment one stops using built-in HTs 
least one constructs temporary built-in HTs on GC heap. In my opinion if 
that's left as is the AA literal feature doesn't quite pull its weight.

All in all built-in AA are aging badly since D1 days.

-- 
Dmitry Olshansky

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On 25/10/13 20:41, Namespace wrote:
 Did you mean to get rid of built-in arrays / kill int[] and replace it with
 Array!T?

Array!T is problematic as things stand -- e.g. you can't foreach over one.  So, 
forgetting syntax preferences, there needs to be some work on containers before 
they can "just work" like the builtins.

If it's possible, I'd rather see the converse -- that code that assumes the GC 
will "just work" with other allocation strategies, so one can use the builtins 
without worrying.  But am I being naively hopeful in seeking that? :-)

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

26-Oct-2013 20:01, Joseph Rushton Wakeling пишет:
 On 25/10/13 20:41, Namespace wrote:
 Did you mean to get rid of built-in arrays / kill int[] and replace it
 with
 Array!T?

 Array!T is problematic as things stand -- e.g. you can't foreach over
 one.

Sure you can. Try it and rejoice:

void main()
{
     import std.container, std.stdio;
     Array!int a = make!(Array!int)(1,2,3,4);
     //the rule is: if a can be sliced then slice it and use that slice
     foreach(v; a)
     {
         writeln(v);
     }
}

 So, forgetting syntax preferences, there needs to be some work on
 containers before they can "just work" like the builtins.

Depends on what's required for you to consider it "just works".
But in general they can't be complete replica of built-ins for many 
reasons, built-ins being designed with GC in mind is one.
Other problems include having no user-land analog of implicit tail-const 
of arrays.

 If it's possible, I'd rather see the converse -- that code that assumes
 the GC will "just work" with other allocation strategies, so one can use
 the builtins without worrying.

Only if you are switching to from one GC kind to another. There is no 
way out of automatic memory management.

 But am I being naively hopeful in
 seeking that? :-)

Yes.

-- 
Dmitry Olshansky

"Tourist" <gravatar gravatar.com> writes:

On Saturday, 26 October 2013 at 16:10:46 UTC, Dmitry Olshansky 
wrote:
 If it's possible, I'd rather see the converse -- that code 
 that assumes
 the GC will "just work" with other allocation strategies, so 
 one can use
 the builtins without worrying.

 Only if you are switching to from one GC kind to another. There 
 is no way out of automatic memory management.

Waiting for ARC :)

"Brad Anderson" <eco gnuk.net> writes:

On Saturday, 26 October 2013 at 16:10:46 UTC, Dmitry Olshansky 
wrote:
 26-Oct-2013 20:01, Joseph Rushton Wakeling пишет:
 On 25/10/13 20:41, Namespace wrote:
 Did you mean to get rid of built-in arrays / kill int[] and 
 replace it
 with
 Array!T?

 Array!T is problematic as things stand -- e.g. you can't 
 foreach over
 one.

 Sure you can. Try it and rejoice:

 void main()
 {
     import std.container, std.stdio;
     Array!int a = make!(Array!int)(1,2,3,4);
     //the rule is: if a can be sliced then slice it and use 
 that slice
     foreach(v; a)
     {
         writeln(v);
     }
 }

The fact that foreach with a ref item or foreach with an index 
doesn't work makes using them a lot more of a hassle than 
built-in arrays though.

     void main()
     {
         import std.container, std.stdio;
         Array!int a = make!(Array!int)(1,2,3,4);
         foreach(ref v; a)
         {
             v *= v; // no effect on a
         }

         writeln(a[]); // [1, 2, 3, 4]

         foreach(v, i; a) // Error: cannot infer argument types
         {
             a[i] *= v;
         }
     }

"Namespace" <rswhite4 googlemail.com> writes:

On Saturday, 26 October 2013 at 16:01:18 UTC, Joseph Rushton 
Wakeling wrote:
 On 25/10/13 20:41, Namespace wrote:
 Did you mean to get rid of built-in arrays / kill int[] and 
 replace it with
 Array!T?

 Array!T is problematic as things stand -- e.g. you can't 
 foreach over one.  So, forgetting syntax preferences, there 
 needs to be some work on containers before they can "just work" 
 like the builtins.

 If it's possible, I'd rather see the converse -- that code that 
 assumes the GC will "just work" with other allocation 
 strategies, so one can use the builtins without worrying.  But 
 am I being naively hopeful in seeking that? :-)

I would never vote to replace the built-in arrays with something 
ugly as Array!T.
If D would switch to Array!T and Map!(T, U) it would be the same 
hell as with C++. But I hope allocators enable the possibility 
that built-in arrays could use other memory mangaement besides 
the GC. That would be awesome.

"deadalnix" <deadalnix gmail.com> writes:

On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky 
wrote:
 Very true. To put it simply it's a disastrous idea that sadly 
 is too easy to be ignored.

 IMHO we'd better start with containers and refitting Phobos 
 from built-in AA/arrays to user-defined containers. One 
 interesting way there is to accept/adopt containers as 
 OutputRange.

Typed allocator on top of this should be defined before this is 
discussed. Right it doesn't make a lot of sense.

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

25-Oct-2013 23:50, deadalnix пишет:
 On Friday, 25 October 2013 at 17:57:23 UTC, Dmitry Olshansky wrote:
 Very true. To put it simply it's a disastrous idea that sadly is too
 easy to be ignored.

 IMHO we'd better start with containers and refitting Phobos from
 built-in AA/arrays to user-defined containers. One interesting way
 there is to accept/adopt containers as OutputRange.

 Typed allocator on top of this should be defined before this is
 discussed. Right it doesn't make a lot of sense.

Yeah, that's another step before that.

It's just I'm strongly opposed to "the strap this magical new allocator 
somewhere so that's it's global for everything, right now" movement and 
I had to plot the alternative course ;)

-- 
Dmitry Olshansky

"Adam D. Ruppe" <destructionator gmail.com> writes:

On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu 
wrote:
 with (setAllocator!Mallocator)

That'd be a mistake except perhaps in two circumstances:

1) if setAllocator was  system with warnings attached.

or

2) if the content inside the with block was all a pure function.



function cannot escape any reference, since it isn't allowed to 
access outside mutable data at all. When the pure function 
returns, the only possible places one of the other allocated 
values could be are in one of its parameters - which are set 
right here on the same (or the immediately following) line as the 
call to setAllocator, so it is no surprise, or the return value, 
and ditto.

Otherwise, by the pure rules, they must all be local and thus 
freeing them is fine.


I'm not quite sold on replacing the global allocator. I'd 
generally prefer to use a container type so it is clear who the 
owner is, and then if anything, just prohibit slice concat (and 
preferably, storing them too, scope storage class!!!!111!one) so 
you don't accidentally reallocate without informing the owner.

But if we did do it, signatures like

 system AllocatorResetStruct setAllocator(Allocator)() {}

and

 safe void setAllocator(Allocator)(pure  safe void function() 
code) {}

just might work out.

Michel Fortin <michel.fortin michelf.ca> writes:

On 2013-10-25 15:14:38 +0000, "Adam D. Ruppe" <destructionator gmail.com> said:

  safe void setAllocator(Allocator)(pure  safe void function() code) {}
 
 just might work out.

I wonder what kind of things can be done in a pure function that takes 
no parameter and returns void. Not much I guess.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca

Timon Gehr <timon.gehr gmx.ch> writes:

On 10/25/2013 08:50 PM, Michel Fortin wrote:
 On 2013-10-25 15:14:38 +0000, "Adam D. Ruppe"
 <destructionator gmail.com> said:

  safe void setAllocator(Allocator)(pure  safe void function() code) {}

 just might work out.

 I wonder what kind of things can be done in a pure function that takes
 no parameter and returns void. Not much I guess.

It can e.g. allocate data and embed it in an exception. (Which is also 
why the above function is still unsafe.)

"bitwise" <bitwise.pvt gmail.com> writes:

On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu 
wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?

 A dream of me, that will probably never come true, would be 
 also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global allocator 
 to the Mallocator in the constructor, and restores it to 
 whatever it was in the destructor.


 Andrei

Doesn't this race because the allocator instance is shared?

I couldn't find 'setAllocator' in the source code.

    Bit

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 09/11/2015 06:32 PM, bitwise wrote:
 On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable allocator?

 A dream of me, that will probably never come true, would be also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global allocator to the
 Mallocator in the constructor, and restores it to whatever it was in
 the destructor.


 Andrei

 Doesn't this race because the allocator instance is shared?

 I couldn't find 'setAllocator' in the source code.

Yah, that was a rumination, not something already implemented. -- Andrei

"bitwise" <bitwise.pvt gmail.com> writes:

On Friday, 11 September 2015 at 23:13:16 UTC, Andrei Alexandrescu 
wrote:
 On 09/11/2015 06:32 PM, bitwise wrote:
 On Friday, 25 October 2013 at 00:00:36 UTC, Andrei 
 Alexandrescu wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace 
 wrote:
 Awesome! Will Appender get an option to use a suitable 
 allocator?

 A dream of me, that will probably never come true, would be 
 also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global 
 allocator to the
 Mallocator in the constructor, and restores it to whatever it 
 was in
 the destructor.


 Andrei

 Doesn't this race because the allocator instance is shared?

 I couldn't find 'setAllocator' in the source code.

 Yah, that was a rumination, not something already implemented. 
 -- Andrei

Ok, thanks. I thought that may be the case.

One more question:

I'd like to integrate these into the containers I'm building, but 
I'm not clear about how to determine if GC.addRange() should be 
called.

I've thought through this, and I'm pretty sure that I can call 
GC.addRange and GC.removeRange indiscriminately on any memory my 
container gets from an allocator, as long as the container finds 
that (hasIndirections!T == true).

My reasoning is that if an allocator calls GC.addRange on it's 
own memory, then it should also reinitialize that memory when it 
gets deallocated, which would include calling GC.addRange again 
if it had to. Also, calling GC.addRange or GC.removeRange on GC 
allocated memory should have no effect.

Does this sound right, or am I crazy?

    Bit

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 09/11/2015 07:46 PM, bitwise wrote:
 On Friday, 11 September 2015 at 23:13:16 UTC, Andrei Alexandrescu wrote:
 On 09/11/2015 06:32 PM, bitwise wrote:
 On Friday, 25 October 2013 at 00:00:36 UTC, Andrei Alexandrescu wrote:
 On 10/24/13 2:38 PM, Namespace wrote:
 On Thursday, 24 October 2013 at 21:31:42 UTC, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable allocator?

 A dream of me, that will probably never come true, would be also
 something like this:
 ----
 with (Mallocator) {
      int[] arr;
      arr ~= 42; /// will use Mallocator.it.allocate internal
 }
 ----

 Oddly enough this can be actually done.

 with (setAllocator!Mallocator)
 {
    ...
 }

 setAllcator returns an rvalue that changes the global allocator to the
 Mallocator in the constructor, and restores it to whatever it was in
 the destructor.


 Andrei

 Doesn't this race because the allocator instance is shared?

 I couldn't find 'setAllocator' in the source code.

 Yah, that was a rumination, not something already implemented. -- Andrei

 Ok, thanks. I thought that may be the case.

 One more question:

 I'd like to integrate these into the containers I'm building, but I'm
 not clear about how to determine if GC.addRange() should be called.

 I've thought through this, and I'm pretty sure that I can call
 GC.addRange and GC.removeRange indiscriminately on any memory my
 container gets from an allocator, as long as the container finds that
 (hasIndirections!T == true).

 My reasoning is that if an allocator calls GC.addRange on it's own
 memory, then it should also reinitialize that memory when it gets
 deallocated, which would include calling GC.addRange again if it had to.
 Also, calling GC.addRange or GC.removeRange on GC allocated memory
 should have no effect.

 Does this sound right, or am I crazy?

Say you have a container that uses its own allocator inside, yet offers 
the user to store objects with indirections that use the GC. Then indeed 
the container would need to call addRange and removeRange on its own 
internal structures.

If, on the contrary, the container imposes that its own held objects use 
the container's allocator as well (a rare but not implausible design), 
it wouldn't need to do that.


Andrei

"bitwise" <bitwise.pvt gmail.com> writes:

On Saturday, 12 September 2015 at 00:31:27 UTC, Andrei 
Alexandrescu wrote:
 On 09/11/2015 07:46 PM, bitwise wrote:
 [...]

 Say you have a container that uses its own allocator inside, 
 yet offers the user to store objects with indirections that use 
 the GC. Then indeed the container would need to call addRange 
 and removeRange on its own internal structures.

This sounds like the default setup I'll be using. The containers 
will use a Mallocator by default, so I will have to add the 
ranges when the contained type is found to have indirections.

 If, on the contrary, the container imposes that its own held 
 objects use the container's allocator as well (a rare but not 
 implausible design), it wouldn't need to do that.

I don't understand exactly what you mean, but here is a more 
verbose version of my concern:

Lets say someone's allocator calls GC.addRange on the memory it 
allocates before a container gets it. The container would call 
GC.addRange when it gets it, but then call GC.removeRange before 
calling allocator.deallocate(). The allocator which think's the 
block has already been registered with the GC could possibly 
reuse it, thinking it's registered with the GC already....but 
that allocator may pass the memory to a container which chooses 
not to call GC.addRange, because it thinks it's using an 
allocator that does that for it. That second container may now 
store types with indirections, which would be unreachable, and 
get freed, and......chaos.

I suppose that as a general rule, a container could always add 
ranges it gets from an allocator to the GC, which would prevent 
the above problem...but only for containers that abide by the 
rule. It sounds like you're saying that although it may not be 
needed sometimes, that no harm would be done.

I think a better solution though would be for the GC to have 
something like GC.hasRange(). The container could check the state 
of the memory it receives and take appropriate action. I don't 
see anything in the current GC docs which would allow you to tell 
if a range had already been added to the GC.

     Bit

"bitwise" <bitwise.pvt gmail.com> writes:

On Saturday, 12 September 2015 at 06:30:42 UTC, bitwise wrote:
 On Saturday, 12 September 2015 at 00:31:27 UTC, Andrei 
 Alexandrescu wrote:
 [...]

 This sounds like the default setup I'll be using. The 
 containers will use a Mallocator by default, so I will have to 
 add the ranges when the contained type is found to have 
 indirections.

 [...]

Alternatively, GC.addRange() could return a value indicating 
whether or not the range had actually been added(for the first 
time) and should be removed.

    Bit

bitwise <bitwise.pvt gmail.com> writes:

On Saturday, 12 September 2015 at 06:45:16 UTC, bitwise wrote:
 [...]

 Alternatively, GC.addRange() could return a value indicating 
 whether or not the range had actually been added(for the first 
 time) and should be removed.

    Bit

Maybe the solution is as simple as specifying the state of memory 
that should be received from an untyped allocator.

It could be explicitly stated that an untyped allocator should 
give out raw memory, and should not initialize it's content or 
add any ranges to the GC. While it may seem obvious for a C++ 
allocator not to initialize it's contents, I think this makes 
sense in the presence of a GC.

I would appreciate some feedback on this.

    Bit

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 09/14/2015 10:46 AM, bitwise wrote:
 On Saturday, 12 September 2015 at 06:45:16 UTC, bitwise wrote:
 [...]

 Alternatively, GC.addRange() could return a value indicating whether
 or not the range had actually been added(for the first time) and
 should be removed.

    Bit

 Maybe the solution is as simple as specifying the state of memory that
 should be received from an untyped allocator.

 It could be explicitly stated that an untyped allocator should give out
 raw memory, and should not initialize it's content or add any ranges to
 the GC. While it may seem obvious for a C++ allocator not to initialize
 it's contents, I think this makes sense in the presence of a GC.

 I would appreciate some feedback on this.

     Bit

I think what we need here is a GCConnectedAllocator that can be inserted 
at the bottom of the allocation foodchain to insert calls to addRange 
and removeRange appropriately. -- Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 2:31 PM, Namespace wrote:
 Awesome! Will Appender get an option to use a suitable allocator?

Well there's the matter of defining typed allocators on top of these 
ones. But yes, once we have allocators in place, they will become part 
of the Phobos' APIs they can serve.

Andrei

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

Very, very nice!

One question that immediately jumps out for me:

As a primitive, will deallocateAll always call the parent's 
deallocate/deallocateAll?

Sometimes it's useful to tell a top-level allocator to forget it 
allocated anything, but not actually free memory, so that it can 
be reused - similarly to Appender's clear method. There doesn't 
seem to be an easy way to do it with Freelist, as it will always 
forward to the parent's deallocate or deallocateAll. Perhaps 
there could be a "clear" primitive too?

Spied two typos:
- s/AllocatorWithOptions/AllocatorWithStats/
- s/an the call/the call/

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Thursday, 24 October 2013 at 21:44:43 UTC, Vladimir Panteleev 
wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei 
 Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.


A nitpick:

FallbackAllocator!(InSituRegion!..., ...) can be faster by 
defining a specialized allocator as a Region, which is 
initialized with the slice of a static buffer, and once that is 
filled up, a new buffer. This avoids a conditional branch in 
FallbackAllocator (we'd be reusing the same branch that checks 
for a full buffer and returns null in Region). This combination 
will likely be used often so it might be worth optimizing this 
use case, but OTOH the FallbackAllocator conditional branch will 
likely be subject to good branch prediction, so at least on x86 
the gain might be minimal.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 3:08 PM, Vladimir Panteleev wrote:
 On Thursday, 24 October 2013 at 21:44:43 UTC, Vladimir Panteleev wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test
 drive.


 A nitpick:

 FallbackAllocator!(InSituRegion!..., ...) can be faster by defining a
 specialized allocator as a Region, which is initialized with the slice
 of a static buffer, and once that is filled up, a new buffer. This
 avoids a conditional branch in FallbackAllocator (we'd be reusing the
 same branch that checks for a full buffer and returns null in Region).
 This combination will likely be used often so it might be worth
 optimizing this use case, but OTOH the FallbackAllocator conditional
 branch will likely be subject to good branch prediction, so at least on
 x86 the gain might be minimal.

Yah, good idea. At some point I have no doubt that intensively-used 
allocators will be tuned a fair amount.

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 2:44 PM, Vladimir Panteleev wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Very, very nice!

 One question that immediately jumps out for me:

 As a primitive, will deallocateAll always call the parent's
 deallocate/deallocateAll?

That's a gray area for the time being. What I fear (for e.g. HeapBlock) 
is a pathological case in which deallocateAll frees parent's buffer and 
then a reallocation grabs another one etc.

Also I haven't forgotten your suggestion returnsZeroedMemory. I think we 
should implement it at a point.

 Sometimes it's useful to tell a top-level allocator to forget it
 allocated anything, but not actually free memory, so that it can be
 reused - similarly to Appender's clear method. There doesn't seem to be
 an easy way to do it with Freelist, as it will always forward to the
 parent's deallocate or deallocateAll. Perhaps there could be a "clear"
 primitive too?

I think that's a good idea.

 Spied two typos:
 - s/AllocatorWithOptions/AllocatorWithStats/
 - s/an the call/the call/

Fixed, thanks.


Andrei

Marco Leise <Marco.Leise gmx.de> writes:

Am Thu, 24 Oct 2013 12:54:41 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 Please destroy! I've literally sweat as I'm sending this :o).
=20
 Andrei

1) =C2=BBFallbackAllocator is an the allocator equivalent of an "or"
operator in algebra.=C2=AB

Remove the "the" there.


2) FallbackAllocator!(InSituRegion!16384, GCAllocator) a;

Awesome, I actually use stuff like this.


3) // Same as above, except the freelist only applies to the
reap.

What is a reap? Is my English just bad or did 'h' get replaced
with 'r' here a few times?


4) =C2=BBFor example, (D HeapBlock!(Allocator, 4096) (described in
detail below) =E2=80=A6=C2=AB

Broken DDoc macro.


The class wrapper at the end is a nice addition to the set of
allocators. It seems like a good pattern to have in D with the
constant mix of class hierarchies and value types.

--=20
Marco

"qznc" <qznc web.de> writes:

On Thursday, 24 October 2013 at 21:57:10 UTC, Marco Leise wrote:
 3) // Same as above, except the freelist only applies to the
 reap.

 What is a reap? Is my English just bad or did 'h' get replaced
 with 'r' here a few times?

A reap mixes region and heap.

Previous discussion:
http://forum.dlang.org/post/aikhlqoffaznpkugesev forum.dlang.org

Paper:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.7.6505&rep=rep1&type=pdf

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Thursday, 24 October 2013 at 21:57:10 UTC, Marco Leise wrote:
 3) // Same as above, except the freelist only applies to the
 reap.

 What is a reap? Is my English just bad or did 'h' get replaced
 with 'r' here a few times?

Reap = region + heap. I believe the term originates from this 
2002 paper:
http://people.cs.umass.edu/~emery/pubs/berger-oopsla2002.pdf

Andrei mentioned them in this 2008 talk:
http://accu.org/content/conf2008/Alexandrescu-memory-allocation.screen.pdf

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 2:56 PM, Marco Leise wrote:
 Am Thu, 24 Oct 2013 12:54:41 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 Please destroy! I've literally sweat as I'm sending this :o).

 Andrei

 1) »FallbackAllocator is an the allocator equivalent of an "or"
 operator in algebra.«

 Remove the "the" there.

Fixed, thanks. I see "reap" has been discussed.

Andrei

"John Colvin" <john.loughran.colvin gmail.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there 
 are no benchmarks. Both would be appreciated, particularly 
 benchmarks to validate the gains (which I speculate can be very 
 sizable) of custom-built, special-purpose allocators compared 
 to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this 
 design. I have been knocking around it for long enough to have 
 no idea how easy it is to get into it from the outside, or how 
 good it is. By all signs I could gather this feels like good 
 design, and one of the best I've ever put together. The 
 allocators defined have an archetypal feeling, are flexible 
 both statically and dynamically, and morph and combine in 
 infinite ways.

 CAllocator and CAllocatorImpl make the link between the static 
 and dynamic worlds. Once an allocator is assembled out of 
 pieces and finely tuned, wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei

I'm nowhere near experienced enough to comment on how good this 
really is, but it sure looks like an impressive design. 
Congratulations :)

"Kapps" <opantm2+spam gmail.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

I admit, I don't use custom allocators as much as I should and am 
not as experienced in this topic as others, yet for my purposes 
this design seems excellent. The docs are solid, and I really 
like the idea of just being able to composite something like 
AllocatorWithStats. The use cases I had in mind (which are 
admittedly quite simple) are covered, and it's very clear how to 
add a custom allocator if desired.

A couple of questions:
Shouldn't most of these implementations be marked nothrow? I 
could be missing something here.
For CAllocator, I notice setting alignment is a property yet does 
not return void. Does this not go against the idea of  property? 
Perhaps it should be setAlignment instead?

"Kapps" <opantm2+spam gmail.com> writes:

On Thursday, 24 October 2013 at 22:19:10 UTC, Kapps wrote:
 For CAllocator, I notice setting alignment is a property yet 
 does not return void. Does this not go against the idea of 
  property? Perhaps it should be setAlignment instead?

Actually, this doesn't make sense because then it would not be 
possible to use CAllocator instead of an Allocator in a template 
as the type would be wrong. It still feels a bit odd that it 
returns bool to me, but I definitely see the logic now.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 3:19 PM, Kapps wrote:
 A couple of questions:
 Shouldn't most of these implementations be marked nothrow? I could be
 missing something here.

Oh yah, that's part of the whole "alpha" thing. Then, since most 
allocators are templated, I'm hoping language definition will catch up 
with me and just infer them most of the time.

 For CAllocator, I notice setting alignment is a property yet does not
 return void. Does this not go against the idea of  property? Perhaps it
 should be setAlignment instead?

Yah, I'll fix that. Thanks!


Andrei

Brad Roberts <braddr puremagic.com> writes:

There was a comment in an earlier reply that's very relevant to mine.  A good
demonstration of the 
utility of these classes is how much of the current (and proposed) garbage
collector can be replaced 
by using this module.  For that to happen, the code needs to actually live in
the runtime (at least 
as things are currently layered).

On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha release of
untyped allocators is
 ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there are no
benchmarks. Both would be
 appreciated, particularly benchmarks to validate the gains (which I speculate
can be very sizable)
 of custom-built, special-purpose allocators compared to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this design. I have been
knocking around it for
 long enough to have no idea how easy it is to get into it from the outside, or
how good it is. By
 all signs I could gather this feels like good design, and one of the best I've
ever put together.
 The allocators defined have an archetypal feeling, are flexible both
statically and dynamically, and
 morph and combine in infinite ways.

 CAllocator and CAllocatorImpl make the link between the static and dynamic
worlds. Once an allocator
 is assembled out of pieces and finely tuned, wrapping it in a dynamic API is a
snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

25-Oct-2013 02:29, Brad Roberts пишет:
 There was a comment in an earlier reply that's very relevant to mine.  A
 good demonstration of the utility of these classes is how much of the
 current (and proposed) garbage collector can be replaced by using this
 module.  For that to happen, the code needs to actually live in the
 runtime (at least as things are currently layered).

 On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is
 ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there are no
 benchmarks. Both would be
 appreciated, particularly benchmarks to validate the gains (which I
 speculate can be very sizable)
 of custom-built, special-purpose allocators compared to traditional
 allocators.

 I acknowledge I'm clearly in no position to evaluate this design. I
 have been knocking around it for
 long enough to have no idea how easy it is to get into it from the
 outside, or how good it is. By
 all signs I could gather this feels like good design, and one of the
 best I've ever put together.
 The allocators defined have an archetypal feeling, are flexible both
 statically and dynamically, and
 morph and combine in infinite ways.

 CAllocator and CAllocatorImpl make the link between the static and
 dynamic worlds. Once an allocator
 is assembled out of pieces and finely tuned, wrapping it in a dynamic
 API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei


Looks incredibly cool so far :)

Some thoughts:

- I don't like the name CAllocator. Whatever that C stands for it's 
ambiguous (Class, Cee, Caramel?).

- It may be only me but I _think_ it could be more useful to have a few 
specific interfaces then 1 fat CAllocator. I've come to dislike 
fat-interfaces with isSupported kludges but that's IMHO.

- I see that shrink primitive didn't make it... Well, on the upside the 
primitives count is really low.

- In description of chooseAtRuntime:
HeapBlock!chooseAtRuntime  --> HeapBlock!(Allocator, chooseAtRuntime)


-- 
Dmitry Olshansky

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 12:19 PM, Dmitry Olshansky wrote:
 - I don't like the name CAllocator. Whatever that C stands for it's
 ambiguous (Class, Cee, Caramel?).

Class. Better names welcome.

 - It may be only me but I _think_ it could be more useful to have a few
 specific interfaces then 1 fat CAllocator. I've come to dislike
 fat-interfaces with isSupported kludges but that's IMHO.

Yah it's a good idea. I'll think of it.

 - I see that shrink primitive didn't make it... Well, on the upside the
 primitives count is really low.

Yah. "Getting away with as little as you can" is my mantra :o).

 - In description of chooseAtRuntime:
 HeapBlock!chooseAtRuntime  --> HeapBlock!(Allocator, chooseAtRuntime)

Fixed and pushed.

https://github.com/andralex/phobos/blob/allocator/std/allocator.d


Andrei

"Jacob Carlborg" <doob me.com> writes:

On Sunday, 27 October 2013 at 06:15:52 UTC, Andrei Alexandrescu 
wrote:

 Class. Better names welcome.

Allocator. Or we could call CAllocator AllocatorBase and call 
CAllocatorImpl Allocator. It depends on which of these two 
classes is most likely to be used in API's. I think the one most 
likely to be used should be called Allocator.

Other names could be DynamicAllocator or RuntimeAllocator.

--
/Jacob Carlborg

"Jakob Ovrum" <jakobovrum gmail.com> writes:

On Sunday, 27 October 2013 at 06:15:52 UTC, Andrei Alexandrescu 
wrote:
 On 10/25/13 12:19 PM, Dmitry Olshansky wrote:
 - I don't like the name CAllocator. Whatever that C stands for 
 it's
 ambiguous (Class, Cee, Caramel?).

 Class. Better names welcome.

I think DynamicAllocator would be a better name than CAllocator, 
but regardless, I think they're both pretty bad. However, I think 
it's indicative of a deeper problem with the interface chosen 
here: CAllocator is probably a kind of god object.

Like Dmitry, I think using `interface`s (maybe `wrap` can be 
leveraged for the adaptation role[1]) is an approach worth 
investigating. It lets the library statically enforce that it 
gets the functionality it needs as opposed to having to raise a 
runtime error, while still allowing the caller to implement that 
functionality dynamically (urgh, reaching for words!).

[1] Assuming `wrap` works on structs.

"deadalnix" <deadalnix gmail.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there 
 are no benchmarks. Both would be appreciated, particularly 
 benchmarks to validate the gains (which I speculate can be very 
 sizable) of custom-built, special-purpose allocators compared 
 to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this 
 design. I have been knocking around it for long enough to have 
 no idea how easy it is to get into it from the outside, or how 
 good it is. By all signs I could gather this feels like good 
 design, and one of the best I've ever put together. The 
 allocators defined have an archetypal feeling, are flexible 
 both statically and dynamically, and morph and combine in 
 infinite ways.

 CAllocator and CAllocatorImpl make the link between the static 
 and dynamic worlds. Once an allocator is assembled out of 
 pieces and finely tuned, wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei

First, I love it !

Now destroying :D

The first thing I notice is that this module is dead unsafe. I'm 
not sure it belong to std. We should probably reserve that for a 
"guru" part of the standard lib. Putting some system function in 
the standard lib is OK, as they are packaged around concepts 
(range, array, algorithms, etc . . .) but an entire module 
dedicated to  system stuff seems like it belong to another place.

I see that you did update the whole stuff according to previous 
discussion, and I'm not sure I have something against the design 
anymore :D

Now, typed allocators, generic tail const and we can get awesome 
collections!

"Meta" <jared771 gmail.com> writes:

On Thursday, 24 October 2013 at 23:22:19 UTC, deadalnix wrote:
 The first thing I notice is that this module is dead unsafe. 
 I'm not sure it belong to std. We should probably reserve that 
 for a "guru" part of the standard lib. Putting some system 
 function in the standard lib is OK, as they are packaged around 
 concepts (range, array, algorithms, etc . . .) but an entire 
 module dedicated to  system stuff seems like it belong to 
 another place.

std.unsafe.*?

"deadalnix" <deadalnix gmail.com> writes:

On Friday, 25 October 2013 at 01:10:24 UTC, Meta wrote:
 On Thursday, 24 October 2013 at 23:22:19 UTC, deadalnix wrote:
 The first thing I notice is that this module is dead unsafe. 
 I'm not sure it belong to std. We should probably reserve that 
 for a "guru" part of the standard lib. Putting some system 
 function in the standard lib is OK, as they are packaged 
 around concepts (range, array, algorithms, etc . . .) but an 
 entire module dedicated to  system stuff seems like it belong 
 to another place.

 std.unsafe.*?

uac.*

Many bonus points for who get this.

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

25-Oct-2013 03:22, deadalnix пишет:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html


[snip]
 First, I love it !

 Now destroying :D

 The first thing I notice is that this module is dead unsafe. I'm not
 sure it belong to std. We should probably reserve that for a "guru" part
 of the standard lib. Putting some system function in the standard lib is
 OK, as they are packaged around concepts (range, array, algorithms, etc
 . . .) but an entire module dedicated to  system stuff seems like it
 belong to another place.

+1 on both counts. It's a  system module, incredibly convenient but it 
belongs to core.allocator;


-- 
Dmitry Olshansky

Michel Fortin <michel.fortin michelf.ca> writes:

On 2013-10-24 19:54:41 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Please destroy! I've literally sweat as I'm sending this :o).

Seems good. Although I'm no expert on allocators I find it very easy to 
grasp. One remark though.

While it generally makes sense that you know the size of an allocation 
when you want to deallocate, in some case it might be suboptimal to 
have to provide the size. For instance, you could use an allocator to 
allocate an object (with a virtual table and all). When comes the time 
to deallocate, to get the size you might have to dereference two 
pointers to extract the value from the classinfo, then create the 
proper void[] range to feed deallocate(). If your allocator is a 
wrapper for malloc/free, the size bit is ignored when calling free and 
all that work for retrieving the actual size of the object is wasted.

I don't know if this is something worth addressing.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 5:13 PM, Michel Fortin wrote:
 On 2013-10-24 19:54:41 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:

 Please destroy! I've literally sweat as I'm sending this :o).

 Seems good. Although I'm no expert on allocators I find it very easy to
 grasp. One remark though.

 While it generally makes sense that you know the size of an allocation
 when you want to deallocate, in some case it might be suboptimal to have
 to provide the size. For instance, you could use an allocator to
 allocate an object (with a virtual table and all). When comes the time
 to deallocate, to get the size you might have to dereference two
 pointers to extract the value from the classinfo, then create the proper
 void[] range to feed deallocate(). If your allocator is a wrapper for
 malloc/free, the size bit is ignored when calling free and all that work
 for retrieving the actual size of the object is wasted.

 I don't know if this is something worth addressing.

If that happens, perhaps AffixAllocator!(A, size_t) could be of use by 
sticking the allocated size just before the allocation. But then you 
have a different problem - tapping into possibly cold memory when 
deallocating.

I have plans to define a type that allows user-defined extra data 
grouped together (e.g. in an array or a list). Due to packing, subword 
per-allocation data can be stored efficiently. Such a design would allow 
storing all object sizes together.


Andrei

Michel Fortin <michel.fortin michelf.ca> writes:

On 2013-10-25 00:20:41 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 If that happens, perhaps AffixAllocator!(A, size_t) could be of use by 
 sticking the allocated size just before the allocation. But then you 
 have a different problem - tapping into possibly cold memory when 
 deallocating.

The size is already available in the classinfo, but the underlying 
allocator doesn't need it when deallocating (the size part will get 
ignored). The goal is to not have to give the size to deallocate when 
it doesn't need it (to save you from retrieving it). All you need is to 
know that deallocate() ignores the size part of the given array telling 
you whether or not it's safe to call deallocate(pointer[0..0]). You 
could add an optional "deallocateIgnoresSize" property for that.

By the way, I think the idea of adding a boolean property for this 
offers less room for misuse than adding an optional deallocate(void*b) 
overload. With an overload you're duplicating the API and it won't be 
immediately clear whether or not it's best to provide the size.

But I'm still not sure it's worth the trouble. I'll leave others be the 
judge of that.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca

"deadalnix" <deadalnix gmail.com> writes:

On Friday, 25 October 2013 at 01:00:04 UTC, Michel Fortin wrote:
 On 2013-10-25 00:20:41 +0000, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:

 If that happens, perhaps AffixAllocator!(A, size_t) could be 
 of use by sticking the allocated size just before the 
 allocation. But then you have a different problem - tapping 
 into possibly cold memory when deallocating.

 The size is already available in the classinfo, but the 
 underlying allocator doesn't need it when deallocating (the 
 size part will get ignored). The goal is to not have to give 
 the size to deallocate when it doesn't need it (to save you 
 from retrieving it). All you need is to know that deallocate() 
 ignores the size part of the given array telling you whether or 
 not it's safe to call deallocate(pointer[0..0]). You could add 
 an optional "deallocateIgnoresSize" property for that.

 By the way, I think the idea of adding a boolean property for 
 this offers less room for misuse than adding an optional 
 deallocate(void*b) overload. With an overload you're 
 duplicating the API and it won't be immediately clear whether 
 or not it's best to provide the size.

 But I'm still not sure it's worth the trouble. I'll leave 
 others be the judge of that.

Many allocators have different policies for different size. So, 
if you don't pass the size, you ends up have to recover it, which 
can be a tedious constraint.

If the length is not used, I hope that The compiler may be able 
to optimize it way if the allocator is simple (so the deallocate 
function is inlined). If the deallocate function is quite 
complex, then I guess this extra calculation is required anyway, 
and won't matter that much.

"John Colvin" <john.loughran.colvin gmail.com> writes:

On Friday, 25 October 2013 at 01:00:04 UTC, Michel Fortin wrote:
 On 2013-10-25 00:20:41 +0000, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:

 If that happens, perhaps AffixAllocator!(A, size_t) could be 
 of use by sticking the allocated size just before the 
 allocation. But then you have a different problem - tapping 
 into possibly cold memory when deallocating.

 The size is already available in the classinfo, but the 
 underlying allocator doesn't need it when deallocating (the 
 size part will get ignored). The goal is to not have to give 
 the size to deallocate when it doesn't need it (to save you 
 from retrieving it). All you need is to know that deallocate() 
 ignores the size part of the given array telling you whether or 
 not it's safe to call deallocate(pointer[0..0]). You could add 
 an optional "deallocateIgnoresSize" property for that.

 By the way, I think the idea of adding a boolean property for 
 this offers less room for misuse than adding an optional 
 deallocate(void*b) overload. With an overload you're 
 duplicating the API and it won't be immediately clear whether 
 or not it's best to provide the size.

 But I'm still not sure it's worth the trouble. I'll leave 
 others be the judge of that.

A void deallocate(void* b) overload might be good in these cases.

Walter Bright <newshound2 digitalmars.com> writes:

On 10/24/2013 12:54 PM, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha release of
 untyped allocators is ready for tire-kicking and a test drive.

This is not a comment on the allocator design, but the module layout.

Would it be possible that this use the "package" idea with one allocator per 
file instead of the all-in-one-file setup?

Jacob Carlborg <doob me.com> writes:

On 2013-10-25 03:08, Walter Bright wrote:

 This is not a comment on the allocator design, but the module layout.

 Would it be possible that this use the "package" idea with one allocator
 per file instead of the all-in-one-file setup?

Yes, please.

-- 
/Jacob Carlborg

Martin Nowak <code dawg.eu> writes:

On 10/25/2013 03:08 AM, Walter Bright wrote:
 Would it be possible that this use the "package" idea with one allocator
 per file instead of the all-in-one-file setup?

I'd like to see that as well.

"inout" <inout gmail.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there 
 are no benchmarks. Both would be appreciated, particularly 
 benchmarks to validate the gains (which I speculate can be very 
 sizable) of custom-built, special-purpose allocators compared 
 to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this 
 design. I have been knocking around it for long enough to have 
 no idea how easy it is to get into it from the outside, or how 
 good it is. By all signs I could gather this feels like good 
 design, and one of the best I've ever put together. The 
 allocators defined have an archetypal feeling, are flexible 
 both statically and dynamically, and morph and combine in 
 infinite ways.

 CAllocator and CAllocatorImpl make the link between the static 
 and dynamic worlds. Once an allocator is assembled out of 
 pieces and finely tuned, wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei

Looks good at first glance. A few notes:

- GC.realloc() is nothrow, but you are still catching OOM 
exception in GCAllocator.reallocate(). You don't in other methods 
(e.g. allocate)
- methods aren't marked as nothrow, they probably should be
- I don't understand how UntypedAllocator.collect() works - since 
it is not allowed to store any pointers in it, calling collect() 
on it should just free all allocated memory, right? In this case, 
it might be best to rename it. It will also be beneficial to pass 
proper bitmask attribute to GC.allocate() (e.g. BlkAttr.NO_SCAN).
- Many constants are global lowercase (i.e. platformAlignment, 
unbounded etc). It confused me for a bit when I was reading code 
("where the hell is this variable coming from?").

Will read the rest later.

Jacob Carlborg <doob me.com> writes:

On 2013-10-24 21:54, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

As someone else mentioned. Shouldn't this possibly be place in druntime 
instead. It might take advantage of some allocator where it cannot use 
the GC.

I think we can come up with better names for "it" and "goodAllocSize".

Typo:

In the table, the description of goodAllocSize:

"This module defines a default mplementation"

Missing an "i" in "implementation".

-- 
/Jacob Carlborg

"simendsjo" <simendsjo gmail.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

Does anyone has good resources on allocators in general? I've 
mostly used malloc/free and GCs. None of the books I've read has 
gone into allocators..

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 1:21 AM, simendsjo wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Does anyone has good resources on allocators in general? I've mostly
 used malloc/free and GCs. None of the books I've read has gone into
 allocators..

I don't know of any literature in book format. There are a few papers 
though on HeapLayers, Hoard, jemalloc, and probably more. Fetch a couple 
and see what they cite and who cites them.

Andrei

Timon Gehr <timon.gehr gmx.ch> writes:

On 10/24/2013 09:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha release of
untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

Looks good.

 ...

 Please destroy! I've literally sweat as I'm sending this :o).
...

- Possible null dereference on line 3128. :o)
   (In general, CascadingAllocator is not yet very composable. It should 
return null allocations instead of failing in undocumented ways. It 
should probably also allow the 'make' function to return null.)

- "CAllocator". The name is non-descriptive/misleading.

- Preconditions should be in the corresponding section.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 5:50 AM, Timon Gehr wrote:
 - Possible null dereference on line 3128. :o)
    (In general, CascadingAllocator is not yet very composable. It should
 return null allocations instead of failing in undocumented ways.

Good point. Fixed and pushed.

 It
 should probably also allow the 'make' function to return null.)

make() returns Allocator objects by value, not pointers.

 - "CAllocator". The name is non-descriptive/misleading.

"Class Allocator" :o)

 - Preconditions should be in the corresponding section.

Will do...


Andrei

Timon Gehr <timon.gehr gmx.ch> writes:

On 10/25/2013 05:21 PM, Andrei Alexandrescu wrote:
 On 10/25/13 5:50 AM, Timon Gehr wrote:
 - Possible null dereference on line 3128. :o)
    (In general, CascadingAllocator is not yet very composable. It should
 return null allocations instead of failing in undocumented ways.

 Good point. Fixed and pushed.
 ...

Thanks.

 It
 should probably also allow the 'make' function to return null.)

 make() returns Allocator objects by value, not pointers.
 ...

'Allocator' could in general be a reference type.

 - "CAllocator". The name is non-descriptive/misleading.

 "Class Allocator" :o)
 ...

Then its purpose must be allocating classes? :o)

Marco Leise <Marco.Leise gmx.de> writes:

Am Sat, 26 Oct 2013 00:11:13 +0200
schrieb Timon Gehr <timon.gehr gmx.ch>:

 - "CAllocator". The name is non-descriptive/misleading.

 "Class Allocator" :o)
 ...

 
 Then its purpose must be allocating classes? :o)

I'm actually fine with that name, probably because I tend to
prefix my D reference types in that manner, too. 'I' for
interfaces, 'C' for classes and maybe 'A' for abstract classes.

-- 
Marco

Jacob Carlborg <doob me.com> writes:

On 2013-10-26 11:03, Marco Leise wrote:

 I'm actually fine with that name, probably because I tend to
 prefix my D reference types in that manner, too. 'I' for
 interfaces, 'C' for classes and maybe 'A' for abstract classes.

That's just horrible.

-- 
/Jacob Carlborg

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/26/13 4:02 AM, Jacob Carlborg wrote:
 On 2013-10-26 11:03, Marco Leise wrote:

 I'm actually fine with that name, probably because I tend to
 prefix my D reference types in that manner, too. 'I' for
 interfaces, 'C' for classes and maybe 'A' for abstract classes.

 That's just horrible.

I think it's a commonly-used convention.

Andrei

"Jacob Carlborg" <doob me.com> writes:

On Saturday, 26 October 2013 at 15:00:12 UTC, Andrei Alexandrescu 
wrote:

 I think it's a commonly-used convention.

Is this something that comes from C++ which doesn't have explicit 
abstract classes and interfaces in the same sense that D, Java 


I dislike this as much as I dislike adding or appending m_ or _ 
for instance variables.

--
/Jacob Carlborg

Jacob Carlborg <doob me.com> writes:

On 2013-10-26 17:01, Andrei Alexandrescu wrote:

 I think it's a commonly-used convention.

Any book, covering the topic, will say that mangling types in names is 
bad practice.

We don't use this naming convention anywhere in Phobos.

-- 
/Jacob Carlborg

"ponce" <contact gam3sfrommars.fr> writes:

 Please destroy! I've literally sweat as I'm sending this :o).

This looks fantastic. Congratulations!

Manu <turkeyman gmail.com> writes:

On 25 October 2013 05:54, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org
 wrote:

 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha release
 of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/**phobos/blob/allocator/std/**
 allocator.d<https://github.com/andralex/phobos/blob/allocator/std/allocator.d>

 Dox: http://erdani.com/d/phobos-**prerelease/std_allocator.html<http://erdani.com/d/phobos-prerelease/std_allocator.html>

 Warning: this is alpha quality. Unit tests are thin, and there are no
 benchmarks. Both would be appreciated, particularly benchmarks to validate
 the gains (which I speculate can be very sizable) of custom-built,
 special-purpose allocators compared to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this design. I have
 been knocking around it for long enough to have no idea how easy it is to
 get into it from the outside, or how good it is. By all signs I could
 gather this feels like good design, and one of the best I've ever put
 together. The allocators defined have an archetypal feeling, are flexible
 both statically and dynamically, and morph and combine in infinite ways.

 CAllocator and CAllocatorImpl make the link between the static and dynamic
 worlds. Once an allocator is assembled out of pieces and finely tuned,
 wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


I like it. It seems like a solid foundation.
I'm still very curious to see how the next phase (deep language
integration) will look though.


My immediate reactions:

1. I'm still sad there are no allocAligned() functions or something of that
type to request explicit alignment with allocations. I'm not sure there is
sufficient support for requesting alignment with allocations. The set-able
alignment property approach seems a little weird (and only seemed to be
supported on one allocator?). I guess experience will tell if this is
sufficient and/or convenient.
I'd still like to see an allocWithAlignment() method or something, which
may be implemented efficiently by allocators that can support it.

2. I see some lines like this:
 assert(parent.alignment >= X.alignof);
What if parent.alignment < X.alignof? If 'parent' is something with an
inflexible alignment, like malloc or the GC, what is the proper
(convenient) way to reconcile the requirement?

3. FreeList has some options; minSize, maxSize, maxNodes. When I'm using a
freelist, the most important option to me is to be able to allocate new
nodes in batches. I'd like an option added to control the batch size, so
multiple new nodes are allocated in contiguous blocks when the pool grows.
Perhaps add a 4th parameter; minBatchSize = 1? (to retain support for your
existing logic batching small allocations into larger allocated blocks)
The main reasons for this are cache/spatial locality of small allocations,
and minimising overhead burden on the upstream allocator.

4. OT: Working with D struct's, you often encounter, for instance:

struct Region(uint minAlign = platformAlignment)
{
private BasicRegion!(minAlign) base;
...

Doesn't this make you sad? It makes me sad quite regularly.
Especially when the very next line is (often): alias base this;

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 7:23 AM, Manu wrote:
 My immediate reactions:

 1. I'm still sad there are no allocAligned() functions or something of
 that type to request explicit alignment with allocations. I'm not sure
 there is sufficient support for requesting alignment with allocations.
 The set-able alignment property approach seems a little weird (and only
 seemed to be supported on one allocator?). I guess experience will tell
 if this is sufficient and/or convenient.
 I'd still like to see an allocWithAlignment() method or something, which
 may be implemented efficiently by allocators that can support it.

Per-allocation alignment requests are currently allowed (by setting the 
property transitorily) but indeed not really nice. I can see how 
HeapBlock could implement a nice alignedAllocate call but the others 
would kind of boringly pass it along.

Don't forget that it's always possible to define extra primitives for a 
given allocator (see e.g. relinquish or available). They should be 
migrated to official API status only if they could help composition in 
one way or another. In fact that's how I defined the API - I started 
with allocate()/deallocate() and a list of allocators I wanted to 
implement, and tried to get away with as few primitives as possible. For 
example FallbackAllocator makes owns() necessary etc.

 2. I see some lines like this:
 assert(parent.alignment>=X.alignof);
 What if parent.alignment < X.alignof? If 'parent' is something with an
 inflexible alignment, like malloc or the GC, what is the proper
 (convenient) way to reconcile the requirement?

That assert is in Freelist and requires that the parent allocator 
returns memory aligned to at least pointer alignment, so as to write 
pointers at the front of the allocation. That's a really low bar, I 
think there's no need to worry about it. (If you do, defining a 
UnalignedFreelist is always an option.)

 3. FreeList has some options; minSize, maxSize, maxNodes. When I'm using
 a freelist, the most important option to me is to be able to allocate
 new nodes in batches. I'd like an option added to control the batch
 size, so multiple new nodes are allocated in contiguous blocks when the
 pool grows. Perhaps add a 4th parameter; minBatchSize = 1? (to retain
 support for your existing logic batching small allocations into larger
 allocated blocks)
 The main reasons for this are cache/spatial locality of small
 allocations, and minimising overhead burden on the upstream allocator.

Good idea; it's what people often do anyway. Before sending this out I'd 
added at least the option to allocate several nodes at a time to fill 
memory more efficiently, see allocateFresh at 
https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L1988. 
Allowing the user to choose the number of nodes in a batch is a good 
extension of that. (I think the default should be unbounded, i.e. let 
the freelist allocator decide depending on goodMallocSize of the parent.)

 4. OT: Working with D struct's, you often encounter, for instance:

 structRegion(uintminAlign=platformAlignment)
 {
 privateBasicRegion!(minAlign)base;
 ...

 Doesn't this make you sad? It makes me sad quite regularly.
 Especially when the very next line is (often): alias base this;

I'm content with that and I think you should too.


Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 7:23 AM, Manu wrote:
 1. I'm still sad there are no allocAligned() functions or something of
 that type to request explicit alignment with allocations. I'm not sure
 there is sufficient support for requesting alignment with allocations.
 The set-able alignment property approach seems a little weird (and only
 seemed to be supported on one allocator?).

Woke up this morning with the following idea.

1. We add two optional API functions:

void[] alignedAllocate(size_t, uint);
bool alignedReallocate(ref void[], size_t, uint);

These would satisfy aligned allocation requests. The pointer thus 
allocated should be deallocated the usual way (there is no 
alignedDeallocate).

2. We make alignment a statically-known enum and delete all code that 
allows it to be get and set at runtime.

Consequences:

1. The API gets larger due to the two new APIs. However, that is offset 
(probably more than enough) by the removal the option to set the 
alignment at runtime, which currently is causing severe ripple effects. 
I think the result will be a simpler overall design and implementation.

2. HeapBlock should be able to implement the two functions. All regions 
should be able to implement the first.

3. Mallocator could also implement these on Posix: 
http://man7.org/linux/man-pages/man3/posix_memalign.3.html. However, 
Windows requires a specific call for deallocating aligned memory. To 
accommodate both portably, we leave Mallocator as is and create 
AlignedMallocator that uses the _aligned_* family on Windows and the 
respective functions on Posix. On Windows, allocate() requests would 
pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc.

Is this satisfactory for everyone?


Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 3. Mallocator could also implement these on Posix:
 http://man7.org/linux/man-pages/man3/posix_memalign.3.html. However,
 Windows requires a specific call for deallocating aligned memory. To
 accommodate both portably, we leave Mallocator as is and create
 AlignedMallocator that uses the _aligned_* family on Windows and the
 respective functions on Posix. On Windows, allocate() requests would
 pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc.

Just implemented AlignedMallocator and pushed.



Untested on Windows.


Andrei

"Tourist" <gravatar gravatar.com> writes:

On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu 
wrote:
 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 3. Mallocator could also implement these on Posix:
 http://man7.org/linux/man-pages/man3/posix_memalign.3.html. 
 However,
 Windows requires a specific call for deallocating aligned 
 memory. To
 accommodate both portably, we leave Mallocator as is and create
 AlignedMallocator that uses the _aligned_* family on Windows 
 and the
 respective functions on Posix. On Windows, allocate() requests 
 would
 pass a default of platformSize (i.e. 16 I suspect) to 
 _aligned_malloc.

 Just implemented AlignedMallocator and pushed.



 Untested on Windows.


 Andrei

In the following line:
----------
On Posix, forwards to realloc. On Windows, calls 
_aligned_realloc(b.ptr, newSize, platformAlignment).
----------

Link is incorrect (http//, colon is missing).

"Tourist" <gravatar gravatar.com> writes:

On Sunday, 27 October 2013 at 10:52:06 UTC, Tourist wrote:
 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu 
 wrote:
 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 3. Mallocator could also implement these on Posix:
 http://man7.org/linux/man-pages/man3/posix_memalign.3.html. 
 However,
 Windows requires a specific call for deallocating aligned 
 memory. To
 accommodate both portably, we leave Mallocator as is and 
 create
 AlignedMallocator that uses the _aligned_* family on Windows 
 and the
 respective functions on Posix. On Windows, allocate() 
 requests would
 pass a default of platformSize (i.e. 16 I suspect) to 
 _aligned_malloc.

 Just implemented AlignedMallocator and pushed.



 Untested on Windows.


 Andrei

 In the following line:
 ----------
 On Posix, forwards to realloc. On Windows, calls 
 _aligned_realloc(b.ptr, newSize, platformAlignment).
 ----------

 Link is incorrect (http//, colon is missing).

That's how Firefox interprets it.
In source code it's:
http://http//msdn.microsoft.com/en-US/library/y69db7sx(v=vs.80).aspx

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/27/13 3:52 AM, Tourist wrote:
 In the following line:
 ----------
 On Posix, forwards to realloc. On Windows, calls _aligned_realloc(b.ptr,
 newSize, platformAlignment).
 ----------

 Link is incorrect (http//, colon is missing).

Thanks, ouch, there were a few more bugs around there as well in the 
untested code.

Andrei

"safety0ff" <safety0ff.dev gmail.com> writes:

On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu 
wrote:
 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 AlignedMallocator that uses the _aligned_* family on Windows 
 and the
 respective functions on Posix. On Windows, allocate() requests 
 would
 pass a default of platformSize (i.e. 16 I suspect) to 
 _aligned_malloc.

 Just implemented AlignedMallocator and pushed.



The constraints on the alignment parameter are neither documented 
nor checked.

i.e. Alignment must be a power of two, and for posix_memalign it 
must additionally be greater than sizeof(void*).

I can only think of one use case for needing runtime specified 
alignment: allocating operating system page sized chunks.
Are there any other use cases?

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/27/13 11:26 AM, safety0ff wrote:
 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote:
 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:
 AlignedMallocator that uses the _aligned_* family on Windows and the
 respective functions on Posix. On Windows, allocate() requests would
 pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc.

 Just implemented AlignedMallocator and pushed.



 The constraints on the alignment parameter are neither documented nor
 checked.

 i.e. Alignment must be a power of two, and for posix_memalign it must
 additionally be greater than sizeof(void*).

Fixed, will push soon.

 I can only think of one use case for needing runtime specified
 alignment: allocating operating system page sized chunks.
 Are there any other use cases?

Cache-line aligned.

http://stackoverflow.com/questions/794632/programmatically-get-the-cache-line-size/1900464

Also some I/O requires aligned buffers.

http://goo.gl/ni860U


Andrei

Manu <turkeyman gmail.com> writes:

On 28 October 2013 04:36, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org
 wrote:

 On 10/27/13 11:26 AM, safety0ff wrote:

 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote:

 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:

 AlignedMallocator that uses the _aligned_* family on Windows and the
 respective functions on Posix. On Windows, allocate() requests would
 pass a default of platformSize (i.e. 16 I suspect) to _aligned_malloc.

 Just implemented AlignedMallocator and pushed.




  The constraints on the alignment parameter are neither documented nor

 checked.

 i.e. Alignment must be a power of two, and for posix_memalign it must
 additionally be greater than sizeof(void*).

 Fixed, will push soon.


  I can only think of one use case for needing runtime specified
 alignment: allocating operating system page sized chunks.
 Are there any other use cases?

 Cache-line aligned.

 http://stackoverflow.com/**questions/794632/**programmatically-get-the-**
 cache-line-size/1900464<http://stackoverflow.com/questions/794632/programmatically-get-the-cache-line-size/1900464>

 Also some I/O requires aligned buffers.

 http://goo.gl/ni860U


GPU's also tend to deal with strict alignments of various buffers.

SIMD requires >= 16byte alignment, which is higher than the default 4-8
byte alignment of many allocators.

"safety0ff" <safety0ff.dev gmail.com> writes:

On Monday, 28 October 2013 at 02:30:53 UTC, Manu wrote:
 GPU's also tend to deal with strict alignments of various 
 buffers.

 SIMD requires >= 16byte alignment, which is higher than the 
 default 4-8
 byte alignment of many allocators.

AFAIK the alignments for those cases are known a priori, the 
question was regarding alignments that are only known at runtime.

Manu <turkeyman gmail.com> writes:

On 28 October 2013 12:47, safety0ff <safety0ff.dev gmail.com> wrote:

 On Monday, 28 October 2013 at 02:30:53 UTC, Manu wrote:

 GPU's also tend to deal with strict alignments of various buffers.

 SIMD requires >= 16byte alignment, which is higher than the default 4-8
 byte alignment of many allocators.

 AFAIK the alignments for those cases are known a priori, the question was
 regarding alignments that are only known at runtime.

You don't know the runtime GPU at compile time.

"jerro" <a a.com> writes:

 i.e. Alignment must be a power of two, and for posix_memalign 
 it must additionally be greater than sizeof(void*).

It must be a power of two and a multiple of sizeof(void*), which 
means that it must be either zero or greater or equal to 
sizeof(void*).

"Lars T. Kyllingstad" <public kyllingen.net> writes:

On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu 
wrote:
 Just implemented AlignedMallocator and pushed.



 Untested on Windows.

It seems inconsistent that allocate() forwards to 
alignedAllocate(), while reallocate() does not forward to 
alignedReallocate().  Why is this?

Lars

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/28/13 1:03 AM, Lars T. Kyllingstad wrote:
 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote:
 Just implemented AlignedMallocator and pushed.




 Untested on Windows.

 It seems inconsistent that allocate() forwards to alignedAllocate(),
 while reallocate() does not forward to alignedReallocate().  Why is this?

 Lars

Two reasons: (1) Posix does not support aligned reallocation. That would 
make forwarding a pessimization on that OS; (2) On Windows, realloc() 
cannot be applied to memory gotten with _aligned_malloc and vice versa.

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/28/13 8:15 AM, Andrei Alexandrescu wrote:
 On 10/28/13 1:03 AM, Lars T. Kyllingstad wrote:
 On Sunday, 27 October 2013 at 10:45:31 UTC, Andrei Alexandrescu wrote:
 Just implemented AlignedMallocator and pushed.





 Untested on Windows.

 It seems inconsistent that allocate() forwards to alignedAllocate(),
 while reallocate() does not forward to alignedReallocate().  Why is this?

 Lars

 Two reasons: (1) Posix does not support aligned reallocation. That would
 make forwarding a pessimization on that OS; (2) On Windows, realloc()
 cannot be applied to memory gotten with _aligned_malloc and vice versa.

 Andrei

Oh, my mistake. On Windows reallocate() should forward to alignedReallocate.

Andrei

Jerry <jlquinn optonline.net> writes:

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

 On 10/26/13 8:00 AM, Andrei Alexandrescu wrote:

 Just implemented AlignedMallocator and pushed.



I'm having a vision of a perverse version called MalignedAllocator...

Jerry <jlquinn optonline.net> writes:

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

 On 10/25/13 7:23 AM, Manu wrote:
 Woke up this morning with the following idea.

 1. We add two optional API functions:

 void[] alignedAllocate(size_t, uint);
 bool alignedReallocate(ref void[], size_t, uint);

A bit of bikeshedding:

Since you have 

allocate()
allocateAll()

I would use 

allocateAligned()
reallocateAligned()

to be more consistent.

Jerry

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 7:23 AM, Manu wrote:
 3. FreeList has some options; minSize, maxSize, maxNodes. When I'm using
 a freelist, the most important option to me is to be able to allocate
 new nodes in batches.

Pushed new version with batchSize parameter, defaulted to 8.

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 7:23 AM, Manu wrote:
 1. I'm still sad there are no allocAligned() functions or something of
 that type to request explicit alignment with allocations. I'm not sure
 there is sufficient support for requesting alignment with allocations.
 The set-able alignment property approach seems a little weird (and only
 seemed to be supported on one allocator?). I guess experience will tell
 if this is sufficient and/or convenient.
 I'd still like to see an allocWithAlignment() method or something, which
 may be implemented efficiently by allocators that can support it.

Code: http://erdani.com/d/phobos-prerelease/std_allocator.html

Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

I've made alignedAllocate() and alignedReallocate() parts of the 
official API, added alignedAlloc to all regions, added AlignedMallocator 
which taps into the aligned system APIs, and eliminated all that dynamic 
alignment setting stuff. I think we're in better shape now. Thanks for 
the suggestion!

I still need to:

1. add stats for aligned calls to AllocatorWithStats

2. define HeapBlock.alignedXxx (which should be interesting)

3. separate IOwns, IAlignedAllocate, IDeallocate from CAllocator and 
have CAllocatorImpl!alloc conditionally implement them depending on 
whether alloc implements the respective primitives.


Andrei

"goughy" <andrew goughy.org> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

Not really qualified to comment, but looks cool.

Looking at AllocatorWithStats,

https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2913

it looks like an error, should be

... static if (flags & Options.callerTime)

Goughy

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/25/13 6:17 PM, goughy wrote:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Not really qualified to comment, but looks cool.

 Looking at AllocatorWithStats,

 https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2913

 it looks like an error, should be

 ... static if (flags & Options.callerTime)

 Goughy

Famous criminals Cut & Paste strike again. Thanks!

Andrei

"Chad Joan" <chadjoan gmail.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 ... awesome stuff ...

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei

I like it a lot so far.

I was really worried about being able to dynamically dispatch to 
an allocator determined at a previous place in the call stack, 
and it seems you're all over it with CAllocator.  Hell yeah!

I have an editing suggestion for the CAllocator comment:
Instead of
"Implementation of CAllocator using Allocator. [...]"
I suggest
"Implements CAllocator using the given Allocator. [...]"
The current one read strangely to me at first, and I had to 
re-read it several times and notice that "Allocator" referred to 
the template parameter.

I agree with others that say that the name CAllocator is too 
ambiguous or vague.  When I scanned through the allocators, I 
initially dismissed it because I though it was a proxy for the 
system's underlying C allocator, with Mallocator being a D-based 
optimized reimplementation of the C allocator.  Reading further 
clarified this, but it does probably harm skimming and searching.

I suggest an alternative name for CAllocator: 
DispatchingAllocator.  I believe this may represent what it does: 
dispatch allocation to another allocator that is behind a 
curtain.  Something like AbstractAllocator might work too, but 
still seems slightly ambiguous to me (i.e. abstract in what 
sense?).

I just hope that the future "top" allocator that handles 
language-builtin allocations will be one that can maintain a 
stack of allocators and push/pop the current default allocator, 
as well as prevent or redirect allocator choice made within calls 
to 3rd party libraries (assuming the libraries are written in D, 
of course).

Johannes Pfau <nospam example.com> writes:

Am Thu, 24 Oct 2013 12:54:41 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 Hello,
 
 
 I know it's been a long wait. Hopefully it was worth it. The alpha 
 release of untyped allocators is ready for tire-kicking and a test
 drive.
 
 Code:
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d
 
 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html
 
 Warning: this is alpha quality. Unit tests are thin, and there are no 
 benchmarks. Both would be appreciated, particularly benchmarks to 
 validate the gains (which I speculate can be very sizable) of 
 custom-built, special-purpose allocators compared to traditional
 allocators.
 
 I acknowledge I'm clearly in no position to evaluate this design. I
 have been knocking around it for long enough to have no idea how easy
 it is to get into it from the outside, or how good it is. By all
 signs I could gather this feels like good design, and one of the best
 I've ever put together. The allocators defined have an archetypal
 feeling, are flexible both statically and dynamically, and morph and
 combine in infinite ways.
 
 CAllocator and CAllocatorImpl make the link between the static and 
 dynamic worlds. Once an allocator is assembled out of pieces and
 finely tuned, wrapping it in a dynamic API is a snap.
 
 Please destroy! I've literally sweat as I'm sending this :o).
 
 
 Andrei

The overall design looks great!


Some small nitpicks:
* byAllocation is not accessible from outside std.allocator
* Is it intentional that AllocatorWithStats can't log the
  __FUNCTION__ / __PRETTY_FUNCTION__ ?
* The documentation for byAllocation should probably state that it only
  lists 'alive' allocations and that this is especially useful for
  finding memory leaks.


And one bigger problem: CAllocatorImpl doesn't work well with
AllocatorWithStats.

AllocatorWithStats special members (byAllocation, etc) are not
directly accessible. Although we can access them by using .impl it'd be
nice to have a special AllocatorWithStats class which directly exposes
those members.

The second problem is that the logged file/line is always in
std.allocator.d. It's probably not easy to get this working in all
cases (especially regarding polymorphism).

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/26/13 5:23 AM, Johannes Pfau wrote:
 Some small nitpicks:
 * byAllocation is not accessible from outside std.allocator

Fixed.

 * Is it intentional that AllocatorWithStats can't log the
    __FUNCTION__ / __PRETTY_FUNCTION__ ?

Just an omission, good idea! Fixed. Let's just record __FUNCTION__.

 * The documentation for byAllocation should probably state that it only
    lists 'alive' allocations and that this is especially useful for
    finding memory leaks.

Fixed.

 And one bigger problem: CAllocatorImpl doesn't work well with
 AllocatorWithStats.

 AllocatorWithStats special members (byAllocation, etc) are not
 directly accessible. Although we can access them by using .impl it'd be
 nice to have a special AllocatorWithStats class which directly exposes
 those members.

That shouldn't be a problem - just downcast to 
CAllocatorImpl!AllocatorWithStats and you got access.

 The second problem is that the logged file/line is always in
 std.allocator.d. It's probably not easy to get this working in all
 cases (especially regarding polymorphism).

Uhm, that /is/ a problem.


Andrei

Lionello Lunesu <lionello lunesu.remove.com> writes:

On 10/27/13, 7:10, Andrei Alexandrescu wrote:
 The second problem is that the logged file/line is always in
 std.allocator.d. It's probably not easy to get this working in all
 cases (especially regarding polymorphism).

 Uhm, that /is/ a problem.

This is supported by using default values:

1:import std.stdio;
2:void w(int l = __LINE__){writeln(l);}
3:void main(){w();}

outputs:
3

By design :)

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/31/13 8:57 AM, Lionello Lunesu wrote:
 On 10/27/13, 7:10, Andrei Alexandrescu wrote:
 The second problem is that the logged file/line is always in
 std.allocator.d. It's probably not easy to get this working in all
 cases (especially regarding polymorphism).

 Uhm, that /is/ a problem.

 This is supported by using default values:

 1:import std.stdio;
 2:void w(int l = __LINE__){writeln(l);}
 3:void main(){w();}

 outputs:
 3

 By design :)

The problem is only in CAllocator.

Andrei

Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:

On 24/10/13 21:54, Andrei Alexandrescu wrote:
 I know it's been a long wait. Hopefully it was worth it. The alpha release of
 untyped allocators is ready for tire-kicking and a test drive.

  ...

 Please destroy! I've literally sweat as I'm sending this :o).

Hmm, seems like I'm coming late to quite a party ... :-)

Like John Colvin, I feel that much of this is over my head -- I don't have the 
experience/expertise to judge requirements or solutions.  But it looks like
very 
nice code indeed and I'm encouraged by the fact that most "Can it do ...?" 
questions seem to have a ready answer.

My own interests have always been less in allocators per se and more in the 
functionality they unlock -- stuff like containers, for example.  It may be 
unfamiliarity -- my own code has either used new/delete (the latter only in
C++, 
not D...) or malloc/dealloc -- but looking at this module as it stands I don't 
really see how to practically make use of it; which makes me wonder whether the 
people calling for it to be in core rather than std have a point.

Maybe it would help to have code examples that really show these allocators 
really being used to do something -- the existing examples are comprehensive
and 
show what to do, but not really what to do _with_ the constructs thus created.

Or, maybe I should just educate myself more ... ;-)

Suffice to say that my overwhelming feeling is of gratitude and admiration for 
all the obvious thought and hard work put into creating this module.  Thanks
and 
congratulations, Andrei! :-)

Best wishes,

     -- Joe

"Lars T. Kyllingstad" <public kyllingen.net> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

About time! ;)  But it was definitely worth waiting for.  I don't
have that much experience with using custom allocators, but
API-wise, this looks great.  Not overburdened, but not 
underpowered either.

I would suggest a different name for goodAllocSize(), though.
Maybe actualAllocSize() or just allocSize().

This may be a stupid question, but is expand() useful outside the
allocator infrastructure itself?  Would you ever use that instead
of reallocate() in a container, for instance?

Lars

"Joseph Cassman" <jc7919 outlook.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei

Going through the code today after reading the thread for the 
past few days I was blown away by the depth of the concepts 
involved. Now the information in the article "Memory Allocation: 
Either Love It or Hate It (or just think it’s okay)" makes a lot 
more sense. The composability and customizability of the design 
look good. This is not an area of coding that I am that familiar 
with but I am interested in trying out a couple of ideas using 
the code in the coming weeks.

I had a good laugh when I saw the ASCII art before the definition 
of NullAllocator. Pretty cool!

Thanks for this work.

Joseph

Martin Nowak <code dawg.eu> writes:

On 10/24/2013 09:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

This looks really promising.
There are a lot of building blocks and the way different capabilities 
are modelled by optional methods nicely solves the biggest difficulty 
with allocators.
I think it's important to put this in it's own github repo and add a dub 
package for it on code.dlang.org so that it's easy to test the 
implementation and to contribute improvements.

That said it failed my litmus test.
I previously used David Simcha's RegionAllocator
https://github.com/dsimcha/TempAlloc/blob/master/std/allocators/region.d.

The pattern is to allocate some metadata followed by allocating many 
fixed size tree nodes. When the tree is constructed it is used to render 
an image which is the result of that operation.
The tree and all metadata is freed and the region allocator is reused 
for the next method invocation (it keeps the memory).

I think the closest would be to use CascadingAllocator with Region but 
there are two issues.

CascadingAllocator successively tries all allocators and if that fails 
creates a new region. So this runs in O(N) complexity even though most 
of the time only the last allocator will have memory available.

There is no simple way to deallocateAll without freeing the regions.
What I need is something similar to clear in appender.
I also can't relinquish the memory from the inner regions because they 
are private.

So for my use-case the only way that I found to use this module
is to compute the upper bound of memory needed when the renderer is 
invoked. Then I have to relinquish the buffer from a region, reallocate 
it using Mallocator.it and construct a new region with the reallocated 
buffer.

This works only because I can cheaply compute the upper bound of 
required memory. This wouldn't work in other scenarios.
I think this a very important use-case, e.g. using an auto-growing 
thread local region is what I would use to serve HTTP requests.
But for this one might also want to use nested regions.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/30/13 1:02 PM, Martin Nowak wrote:
 This looks really promising.
 There are a lot of building blocks and the way different capabilities
 are modelled by optional methods nicely solves the biggest difficulty
 with allocators.
 I think it's important to put this in it's own github repo and add a dub
 package for it on code.dlang.org so that it's easy to test the
 implementation and to contribute improvements.

Tried to register github package andralex/phobos, and:

============================
500 - Internal Server Error

Internal Server Error

Internal error information:
object.Exception source/dubregistry/repositories/repository.d(43): 
Failed to read JSON from 
https://raw.github.com/andralex/phobos/master/package.json: Unexpected 
reply for 'https://raw.github.com/andralex/phobos/master/package.json': 
Not Found
----------------
./dub-registry(dubregistry.repositories.repository.PackageVersionInfo 
dubregistry.repositories.github.GithubRepository.getVersionInfo(immut
ble(char)[])+0x3b5) 
[0x6e8ec1]
./dub-registry(void 
dubregistry.registry.DubRegistry.addPackage(vibe.data.json.Json, 
vibe.data.bson.BsonObjectID)+0xa6) [0x62d372]
./dub-registry(void 
dubregistry.web.DubRegistryWebFrontend.addPackage(vibe.http.serve
.HTTPServerRequest, 
vibe.http.server.HTTPServerResponse, userman.controller.User)+0x222) 
[0x6e4406]
./dub-registry(void delegate(vibe.http.server.HTTPServerRequest, 
vibe.http.server.HTTPServerResponse) 
userman.web.UserManWebInterface.auth(void 
delegate(vibe.http.server.HTTPServerRequest, 
vibe.http.server.HTTPServerResponse, userman.controller.User)).void 
requestHandler(vibe.http.server.HTTPServerRequest, 
vibe.http.server.HTTPServerResponse)+0x10d) [0x800181]
./dub-registry(void 
vibe.http.router.URLRouter.handleRequest(vibe.http.server.HTTPServerRequest, 
vibe.http.server.HTTPServerResponse)+0x179) [0x6fb8b5]
./dub-registry(bool 
vibe.http.server.handleRequest(vibe.core.stream.Stream, 
immutable(char)[], vibe.http.server.HTTPServerListener, ref 
vibe.http.server.HTTPServerSettings, ref bool)+0x16c8) [0x6f0344]
./dub-registry(void 
vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, 
vibe.http.server.HTTPServerListener)+0x143) [0x6eebb7]
./dub-registry(void 
vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings, 
void delegate(vibe.http.server.HTTPServerRequest, 
vibe.http.server.HTTPServerResponse)).void 
doListen(vibe.http.server.HTTPServerSettings, 
vibe.http.server.HTTPServerListener, immutable(char)[]).void 
__lambda54(vibe.core.net.TCPConnection)+0x2c) [0x6eb160]
./dub-registry(extern (C) nothrow void 
vibe.core.drivers.libevent2_tcp.onConnect(int, short, void*).void 
ClientTask.execute()+0x2d6) [0x70939a]
./dub-registry(void vibe.core.core.CoreTask.run()+0xf2) [0x7172fe]
./dub-registry(void core.thread.Fiber.run()+0x2a) [0x83eae2]
./dub-registry(fiber_entryPoint+0x61) [0x83e9ed]
[(nil)]
============================

This is obviously because package.json is absent from the repo, but I'd 
say it shouldn't cause such an error.

That makes me think probably Phobos should have a package.json so people 
can install updates via code.dlang.org.

 That said it failed my litmus test.
 I previously used David Simcha's RegionAllocator
 https://github.com/dsimcha/TempAlloc/blob/master/std/allocators/region.d.

Let's see!

 The pattern is to allocate some metadata followed by allocating many
 fixed size tree nodes. When the tree is constructed it is used to render
 an image which is the result of that operation.
 The tree and all metadata is freed and the region allocator is reused
 for the next method invocation (it keeps the memory).

 I think the closest would be to use CascadingAllocator with Region but
 there are two issues.

 CascadingAllocator successively tries all allocators and if that fails
 creates a new region. So this runs in O(N) complexity even though most
 of the time only the last allocator will have memory available.

Yah, I'd left a TODO in there when I first wrote the code:

https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L3723

The newly-added allocator should come to the front of the list.

My suspicion, however, is that you won't be able to measure a 
difference. A well-dimensioned cascade of regions will have a high 
ration of allocations within a regions to number of regions.

 There is no simple way to deallocateAll without freeing the regions.
 What I need is something similar to clear in appender.

Hm, interesting. But then again - do you think it makes a difference? 
Allocation of regions is a very small fraction of the work done on using 
the regions.

 I also can't relinquish the memory from the inner regions because they
 are private.

How do we formalize that?

 So for my use-case the only way that I found to use this module
 is to compute the upper bound of memory needed when the renderer is
 invoked. Then I have to relinquish the buffer from a region, reallocate
 it using Mallocator.it and construct a new region with the reallocated
 buffer.

I'd say just plow ahead with a straight region as implemented. If you 
measure any difference, let's talk.


Andrei

Martin Nowak <code dawg.eu> writes:

On 11/01/2013 10:00 PM, Andrei Alexandrescu wrote:
 On 10/30/13 1:02 PM, Martin Nowak wrote:
 This looks really promising.
 There are a lot of building blocks and the way different capabilities
 are modelled by optional methods nicely solves the biggest difficulty
 with allocators.
 I think it's important to put this in it's own github repo and add a dub
 package for it on code.dlang.org so that it's easy to test the
 implementation and to contribute improvements.

 Tried to register github package andralex/phobos, and:

I transferred you a separate repo with all std.allocator commits.
https://github.com/dawgfoto/std_allocator
You can build the library with "dub" and run unittests with "dub 
-build=unittest -config=unittest".
Creating docs is a problem, maybe some expert can chime in.

 This is obviously because package.json is absent from the repo, but I'd
 say it shouldn't cause such an error.

https://github.com/rejectedsoftware/dub-registry/issues/31

 https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L3723

 The newly-added allocator should come to the front of the list.

 My suspicion, however, is that you won't be able to measure a
 difference. A well-dimensioned cascade of regions will have a high
 ration of allocations within a regions to number of regions.

You're probably right.

"Martin Nowak" <code dawg.eu> writes:

 Tried to register github package andralex/phobos, and:

 ============================
 500 - Internal Server Error

 Internal Server Error

Now fixed 
https://github.com/rejectedsoftware/dub-registry/issues/31.

"safety0ff" <safety0ff.dev gmail.com> writes:

I noticed that the GCAllocator provides no way of controlling the 
memory block attributes 

allocations get the default (no attributes.) This is a leaky 
abstraction, a data structure or composed allocators may desire 
to control the attributes to reduce GC pressure.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/31/13 7:26 PM, safety0ff wrote:
 I noticed that the GCAllocator provides no way of controlling the memory

 ,) all allocations get the default (no attributes.) This is a leaky
 abstraction, a data structure or composed allocators may desire to
 control the attributes to reduce GC pressure.

These attributes seem to be informed by the types stored, which would be 
above the charter of untyped allocator.

Andrei

"deadalnix" <deadalnix gmail.com> writes:

On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu 
wrote:
 On 10/31/13 7:26 PM, safety0ff wrote:
 I noticed that the GCAllocator provides no way of controlling 
 the memory
 block attributes 

 ,) all allocations get the default (no attributes.) This is a 
 leaky
 abstraction, a data structure or composed allocators may 
 desire to
 control the attributes to reduce GC pressure.

 These attributes seem to be informed by the types stored, which 
 would be above the charter of untyped allocator.

 Andrei

The choice is for the typed allocator. That mean that untyped 
allocator should either provide the choice, or assume the worse 
(it may contain pointer).

"safety0ff" <safety0ff.dev gmail.com> writes:

On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu 
wrote:
 On 10/31/13 7:26 PM, safety0ff wrote:
 I noticed that the GCAllocator provides no way of controlling 
 the memory
 block attributes 

 ,) all allocations get the default (no attributes.) This is a 
 leaky
 abstraction, a data structure or composed allocators may 
 desire to
 control the attributes to reduce GC pressure.

 These attributes seem to be informed by the types stored, which 
 would be above the charter of untyped allocator.

 Andrei

The attributes are informed by whatever code is calling the GC, 
the GC interface deals in void*'s.

Consider an AA implementation that wishes to use FancyAllocator 
with fallback GCAllocator with block attributes NO_INTERIOR and 
NO_SCAN.
With your proposed GCAllocator you either need to rewrite 
GCAllocator, or you need to add some nasty code to set the 
attributes depending on whether the primary allocator or 
secondary allocator own the memory.

By fixing the leaky abstraction this use case can be coded as 
follows:
FallbackAllocator!(FancyAllocator, 
GCAllocator!(GC.BLkAttr.NO_INTERIOR | GC.BLkAttr.NO_SCAN)) a;

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/31/13 8:37 PM, safety0ff wrote:
 On Friday, 1 November 2013 at 02:43:00 UTC, Andrei Alexandrescu wrote:
 On 10/31/13 7:26 PM, safety0ff wrote:
 I noticed that the GCAllocator provides no way of controlling the memory

 ,) all allocations get the default (no attributes.) This is a leaky
 abstraction, a data structure or composed allocators may desire to
 control the attributes to reduce GC pressure.

 These attributes seem to be informed by the types stored, which would
 be above the charter of untyped allocator.

 Andrei

 The attributes are informed by whatever code is calling the GC, the GC
 interface deals in void*'s.

 Consider an AA implementation that wishes to use FancyAllocator with
 fallback GCAllocator with block attributes NO_INTERIOR and NO_SCAN.
 With your proposed GCAllocator you either need to rewrite GCAllocator,
 or you need to add some nasty code to set the attributes depending on
 whether the primary allocator or secondary allocator own the memory.

 By fixing the leaky abstraction this use case can be coded as follows:
 FallbackAllocator!(FancyAllocator, GCAllocator!(GC.BLkAttr.NO_INTERIOR |
 GC.BLkAttr.NO_SCAN)) a;

Migrating the flags into the type is a possibility but maybe it's 
easiest to add flags as runtime parameters.

Allocators can always define additional nonstandard routines. The 
standard routines concern mostly composition.

Of course, it is also possible to make such flags standard (it may be 
the case that typed allocators require such).


Andrei

"safety0ff" <safety0ff.dev gmail.com> writes:

On Friday, 1 November 2013 at 15:15:10 UTC, Andrei Alexandrescu 
wrote:
 Migrating the flags into the type is a possibility but maybe 
 it's easiest to add flags as runtime parameters.

I was using that as an example to argue for its inclusion because 
it was concise.

I'm not sure what the best interface would be, I'd have to think 
about it for a while.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

Added SharedFreelist, a lock-free freelist.




Andrei

Martin Nowak <code dawg.eu> writes:

On 11/01/2013 03:34 AM, Andrei Alexandrescu wrote:
 Added SharedFreelist, a lock-free freelist.




 Andrei

Nice, that reminds me that I still want to polish my implementation of a 
lock free doubly-linked list in D.
It would be great to collect these efforts in a library.
I remember the request of Adam Wilson for a 
System.Collections.Concurrent in D.
I put my stuff on github https://github.com/dawgfoto/lock-free.

The doubly-linked list is based on this paper. IIRC the paper had a bug.
http://dx.doi.org/10.1016/j.jpdc.2008.03.001

Recently I also found a C++ implementation. It's much simpler in D due 
to the GC.
https://github.com/Kometes/Honeycomb/blob/master/src/common/Honey/Thread/LockFree/List.h

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/31/13 7:34 PM, Andrei Alexandrescu wrote:
 On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Added SharedFreelist, a lock-free freelist.



Added documentation clarifying the allocators' approach to shared 
memory. See section "Allocating memory for sharing across threads" near 
the top of

http://erdani.com/d/phobos-prerelease/std_allocator.html



Andrei

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

01-Nov-2013 06:34, Andrei Alexandrescu пишет:
 On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Added SharedFreelist, a lock-free freelist.



Neat. However it's missing from "Synopsis of predefined allocator 
building blocks" table.

-- 
Dmitry Olshansky

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/2/13 1:15 PM, Dmitry Olshansky wrote:
 01-Nov-2013 06:34, Andrei Alexandrescu пишет:
 On 10/24/13 12:54 PM, Andrei Alexandrescu wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The alpha
 release of untyped allocators is ready for tire-kicking and a test
 drive.

 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Added SharedFreelist, a lock-free freelist.



 Neat. However it's missing from "Synopsis of predefined allocator
 building blocks" table.

Thanks, fixed and pushed.

Andrei

"Meta" <jared771 gmail.com> writes:

 From the "Allocating memory for sharing across threads" section, 
this paragraph:

"All allocators in this module accept and return void[] **(as 
opposed to )**. This is because at the time of allocation, 
deallocation, or reallocation, the memory is effectively not 
shared (**it** it were, it would be a bug at the application 
level)."

Starred items appear to be typos or missing a piece.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/2/13 1:39 PM, Meta wrote:
  From the "Allocating memory for sharing across threads" section, this
 paragraph:

 "All allocators in this module accept and return void[] **(as opposed to
 )**. This is because at the time of allocation, deallocation, or
 reallocation, the memory is effectively not shared (**it** it were, it
 would be a bug at the application level)."

 Starred items appear to be typos or missing a piece.

Ew. Thanks, fixed and pushed.

Andrei

"Temtaime" <temtaime gmail.com> writes:

Hi, Andrei.

I cannot compile the allocator.

allocator.d(799): Error: undefined identifier returned

At 499 line:
returned alignedReallocate(b, newSize, alignment);

Ur mistake?

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/3/13 4:14 AM, Temtaime wrote:
 Hi, Andrei.

 I cannot compile the allocator.

 allocator.d(799): Error: undefined identifier returned

 At 499 line:
 returned alignedReallocate(b, newSize, alignment);

 Ur mistake?

Yes, mine. I don't have access to Windows so I expect a bunch of typos 
in the Windows-specific code. Fixed and pushed this one.

Thanks,

Andrei

"Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:

On Friday, 1 November 2013 at 02:33:57 UTC, Andrei Alexandrescu 
wrote:
 Added SharedFreelist, a lock-free freelist.




 Andrei

Hi Andrei,

Please check this bug fix for SharedFreelist 
https://github.com/andralex/phobos/pull/21 .

I have found that source code for bounded `maxNodes` version of 
SharedFreelist is commented out. I understand that `maxNodes` can 
be only approximate bound for _shared_ free list. However, 
approximate `maxNodes` bound is very useful too. Can I create PR 
for this feature?

Best Regards, --Ilya

Martin Nowak <code dawg.eu> writes:

I have another request despite putting this in it's own repo.

Often one want an exponentially (power of 2) growing step size for 
Bucketizer. Currently only equally spaced buckets are possible which 
isn't very practical to scale from 16b to 2Mb.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/1/13 1:36 PM, Martin Nowak wrote:
 I have another request despite putting this in it's own repo.

I assume s/despite/in addition to/ :o).

 Often one want an exponentially (power of 2) growing step size for
 Bucketizer. Currently only equally spaced buckets are possible which
 isn't very practical to scale from 16b to 2Mb.


I considered the growth strategy as a policy. My personal favorite is 
"choose an approximate exponential growth strategy that keeps maximum 
internal fragmentation less than x%." That's how jemalloc is dimensioned.

I decided to stick with linear at least for now, for a simple reason: 
it's easy enough to simply enumerate the strategy by hand by using 
Segregator. Exponentials quickly grow to span a bunch of memory, so 
there aren't a lot of terms involved. Nevertheless it would be a nice 
illustration of D's generative powers.


Andrei

"Dicebot" <public dicebot.lv> writes:

 That makes me think probably Phobos should have a package.json 
 so people

can install updates via code.dlang.org.

How would you imagine that? Not-yet-in-Phobos packages are 
expected to be submitted as standalone ones, not as whole Phobos 
repo.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/1/13 2:12 PM, Dicebot wrote:
 That makes me think probably Phobos should have a package.json so people

 can install updates via code.dlang.org.

 How would you imagine that? Not-yet-in-Phobos packages are expected to
 be submitted as standalone ones, not as whole Phobos repo.

I meant people who'd want to just install Phobos from master. Probably 
not a good idea because Phobos changes often are in sync with compiler 
changes.

Andrei

"Namespace" <rswhite4 googlemail.com> writes:

How about a stack allocator like this:
----
enum StackSize = 8192;

struct Stack {
	static Stack it;
	
	void[StackSize] _buffer = void;
	size_t _bufUsage;

	void[] take(size_t N) {
		if ((this._bufUsage + N) <= StackSize) {
			scope(exit) this._bufUsage += N;

			return _buffer[this._bufUsage .. this._bufUsage + N];
		}

		return null;
	}
	
	void reset() {
		this._bufUsage = 0;
	}
}
----
Would that fit in std.allocator?

=?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:

Am 07.11.2013 11:32, schrieb Namespace:
 How about a stack allocator like this:
 ----
 enum StackSize = 8192;

 struct Stack {
      static Stack it;

      void[StackSize] _buffer = void;
      size_t _bufUsage;

      void[] take(size_t N) {
          if ((this._bufUsage + N) <= StackSize) {
              scope(exit) this._bufUsage += N;

              return _buffer[this._bufUsage .. this._bufUsage + N];
          }

          return null;
      }

      void reset() {
          this._bufUsage = 0;
      }
 }
 ----
 Would that fit in std.allocator?

That's std.allocator.InSituRegion, just that is misses the reset() method.

"Namespace" <rswhite4 googlemail.com> writes:

On Thursday, 7 November 2013 at 13:15:03 UTC, Sönke Ludwig wrote:
 Am 07.11.2013 11:32, schrieb Namespace:
 How about a stack allocator like this:
 ----
 enum StackSize = 8192;

 struct Stack {
     static Stack it;

     void[StackSize] _buffer = void;
     size_t _bufUsage;

     void[] take(size_t N) {
         if ((this._bufUsage + N) <= StackSize) {
             scope(exit) this._bufUsage += N;

             return _buffer[this._bufUsage .. this._bufUsage + 
 N];
         }

         return null;
     }

     void reset() {
         this._bufUsage = 0;
     }
 }
 ----
 Would that fit in std.allocator?

 That's std.allocator.InSituRegion, just that is misses the 
 reset() method.

Nice! But I suggest two things:
  1. Add a reset function for reusing the same storage
  2. Should that: 
https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907 
not be replaced with ubyte[size] _store = void; ?

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

07-Nov-2013 17:30, Namespace пишет:
 On Thursday, 7 November 2013 at 13:15:03 UTC, Sönke Ludwig wrote:
 Am 07.11.2013 11:32, schrieb Namespace:
 How about a stack allocator like this:
 ----
 enum StackSize = 8192;

 struct Stack {
     static Stack it;

     void[StackSize] _buffer = void;
     size_t _bufUsage;

     void[] take(size_t N) {
         if ((this._bufUsage + N) <= StackSize) {
             scope(exit) this._bufUsage += N;

             return _buffer[this._bufUsage .. this._bufUsage + N];
         }

         return null;
     }

     void reset() {
         this._bufUsage = 0;
     }
 }
 ----
 Would that fit in std.allocator?

 That's std.allocator.InSituRegion, just that is misses the reset()
 method.

 Nice! But I suggest two things:
   1. Add a reset function for reusing the same storage
   2. Should that:
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907
 not be replaced with ubyte[size] _store = void; ?

Hm... Andrei has just been caught using internal pointers in structs :o)

-- 
Dmitry Olshansky

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/7/13 5:30 AM, Namespace wrote:
 Nice! But I suggest two things:
   1. Add a reset function for reusing the same storage

freeAll().

   2. Should that:
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d#L2907
 not be replaced with ubyte[size] _store = void; ?

Made the change and pushed, thanks. Unfortunately there's a performance 
bug in the front-end (if at least one field has initialization, all 
fields are initialized by bitblasting .init over the object).

https://d.puremagic.com/issues/show_bug.cgi?id=11331


Andrei

"Namespace" <rswhite4 googlemail.com> writes:

On Thursday, 7 November 2013 at 16:28:42 UTC, Andrei Alexandrescu 
wrote:
 On 11/7/13 5:30 AM, Namespace wrote:
 Nice! But I suggest two things:
  1. Add a reset function for reusing the same storage

 freeAll().

There is no freeAll. Did you mean deallocateAll?
----
void deallocateAll()
{
    _crt = null;
}
----
_crt is null and the same storage can't be used again.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/7/13 9:02 AM, Namespace wrote:
 On Thursday, 7 November 2013 at 16:28:42 UTC, Andrei Alexandrescu wrote:
 On 11/7/13 5:30 AM, Namespace wrote:
 Nice! But I suggest two things:
  1. Add a reset function for reusing the same storage

 freeAll().

 There is no freeAll. Did you mean deallocateAll?
 ----
 void deallocateAll()
 {
     _crt = null;
 }
 ----
 _crt is null and the same storage can't be used again.

Sorry that's indeed the name and there's a bug in it!

Andrei

"Jack Applegame" <japplegame gmail.com> writes:

Passing -g flag to the linker causes a linker error on win32.
Very simple program:

import std.allocator;
void main() {}

Compilation commands:

dmd.exe -g -debug -c std/allocator.d -ofallocator.obj
dmd.exe -g -debug -c main.d -ofmain.obj
dmd.exe out.exe -g allocator.obj main.obj

Output:

dmd.exe out.exe -g allocator.obj main.obj
OPTLINK (R) for Win32  Release 8.00.13
Copyright (C) Digital Mars 1989-2010  All rights reserved.
http://www.digitalmars.com/ctg/optlink.html
allocator.obj(allocator)
  Error 42: Symbol Undefined __aligned_malloc
allocator.obj(allocator)
  Error 42: Symbol Undefined __aligned_free
allocator.obj(allocator)
  Error 42: Symbol Undefined __aligned_realloc
--- errorlevel 3

Without -g flag on the last command (dmd.exe out.exe 
allocator.obj main.obj) no linker error appears.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/16/13 2:50 AM, Jack Applegame wrote:
 Passing -g flag to the linker causes a linker error on win32.

[snip]

Maybe this would help? 
http://community.rti.com/kb/why-does-visual-studio-complain-about-alignedmalloc-and-alignedfree

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/7/13 5:15 AM, Sönke Ludwig wrote:
 That's std.allocator.InSituRegion, just that is misses the reset() method.

freeAll should take care of it.

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 11/7/13 2:32 AM, Namespace wrote:
 How about a stack allocator like this:
 ----
 enum StackSize = 8192;

 struct Stack {
      static Stack it;

      void[StackSize] _buffer = void;
      size_t _bufUsage;

      void[] take(size_t N) {
          if ((this._bufUsage + N) <= StackSize) {
              scope(exit) this._bufUsage += N;

              return _buffer[this._bufUsage .. this._bufUsage + N];
          }

          return null;
      }

      void reset() {
          this._bufUsage = 0;
      }
 }
 ----
 Would that fit in std.allocator?

It's there!

Andrei

"Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:

Awesome!

Since we have SIMD instructions in core it will be nice to have 
AlignedGCallocator.

Best Regards,
Ilya

"Dejan Lekic" <dejan.lekic gmail.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 I know it's been a long wait. Hopefully it was worth it. The 
 alpha release of untyped allocators is ready for tire-kicking 
 and a test drive.

 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Warning: this is alpha quality. Unit tests are thin, and there 
 are no benchmarks. Both would be appreciated, particularly 
 benchmarks to validate the gains (which I speculate can be very 
 sizable) of custom-built, special-purpose allocators compared 
 to traditional allocators.

 I acknowledge I'm clearly in no position to evaluate this 
 design. I have been knocking around it for long enough to have 
 no idea how easy it is to get into it from the outside, or how 
 good it is. By all signs I could gather this feels like good 
 design, and one of the best I've ever put together. The 
 allocators defined have an archetypal feeling, are flexible 
 both statically and dynamically, and morph and combine in 
 infinite ways.

 CAllocator and CAllocatorImpl make the link between the static 
 and dynamic worlds. Once an allocator is assembled out of 
 pieces and finely tuned, wrapping it in a dynamic API is a snap.

 Please destroy! I've literally sweat as I'm sending this :o).


 Andrei

All I can say is - I can't wait to see it in the run-time 
library! I read about the jmalloc last year, and your work makes 
it possible to have something similar in D. Kudos! System 
developers will definitely like std.allocator, no doubt! :)

"Dicebot" <public dicebot.lv> writes:

Andrei, what is current state of std.allocator? I am asking this 
in context of recent Walter Phobos proposal (ScopeBuffer) to 
evaluate how feasible is to define any relations between two at 
current stage.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking this in
 context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how
 feasible is to define any relations between two at current stage.

Progress on std.allocator is slow but I do have a plan. I don't think we 
should pull ScopeBuffer yet.

Andrei

"ANtlord" <antlord92 gmail.com> writes:

On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu 
wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking 
 this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.

 Progress on std.allocator is slow but I do have a plan. I don't 
 think we should pull ScopeBuffer yet.

 Andrei

Mr. Alexandrescu. What about std allocator? Is feature valuable? 
I've took a look in Wish List in wiki and Allocator still there.
If this feature is valuable. What is left in currently 
implementation of your project? Can you tell about state, maybe 
show To Do list?

Sorry, if my english is not clear. Thank you.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking this in
 context of recent Walter Phobos proposal (ScopeBuffer) to evaluate how
 feasible is to define any relations between two at current stage.

 Progress on std.allocator is slow but I do have a plan. I don't think
 we should pull ScopeBuffer yet.

 Andrei

 Mr. Alexandrescu. What about std allocator? Is feature valuable? I've
 took a look in Wish List in wiki and Allocator still there.
 If this feature is valuable. What is left in currently implementation of
 your project? Can you tell about state, maybe show To Do list?

 Sorry, if my english is not clear. Thank you.

Interface with garbage collector is to be done. The rest is usable right 
now. -- Andrei

"John Colvin" <john.loughran.colvin gmail.com> writes:

On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei 
Alexandrescu wrote:
 On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking 
 this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.

 Progress on std.allocator is slow but I do have a plan. I 
 don't think
 we should pull ScopeBuffer yet.

 Andrei

 Mr. Alexandrescu. What about std allocator? Is feature 
 valuable? I've
 took a look in Wish List in wiki and Allocator still there.
 If this feature is valuable. What is left in currently 
 implementation of
 your project? Can you tell about state, maybe show To Do list?

 Sorry, if my english is not clear. Thank you.

 Interface with garbage collector is to be done. The rest is 
 usable right now. -- Andrei

Should it be in std.experimental? Or at least on code.dlang.org?

"MrSmith" <mrsmith33 yandex.ru> writes:

On Wednesday, 11 February 2015 at 15:57:26 UTC, John Colvin wrote:
 Should it be in std.experimental? Or at least on code.dlang.org?

Yeah, dub package would be really nice!

"ANtlord" <antlord92 gmail.com> writes:

On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei 
Alexandrescu wrote:
 On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking 
 this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.

 Progress on std.allocator is slow but I do have a plan. I 
 don't think
 we should pull ScopeBuffer yet.

 Andrei

 Mr. Alexandrescu. What about std allocator? Is feature 
 valuable? I've
 took a look in Wish List in wiki and Allocator still there.
 If this feature is valuable. What is left in currently 
 implementation of
 your project? Can you tell about state, maybe show To Do list?

 Sorry, if my english is not clear. Thank you.

 Interface with garbage collector is to be done. The rest is 
 usable right now. -- Andrei

I thought, that general purpose of allocator it is memory 
management without garbage collector. Because allocator will be 
used in containers, and they must work without garbage collector. 
If no, what is difference between containers and arrays in 
current implementation of D? I assumed, that containers are need 
for increased performance within saving safety code. But a using 
GC will not increase performance.

Or Am I dead wrong?

"Paulo Pinto" <pjmlp progtools.org> writes:

On Thursday, 12 February 2015 at 08:56:53 UTC, ANtlord wrote:
 On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei 
 Alexandrescu wrote:
 On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am asking 
 this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.

 Progress on std.allocator is slow but I do have a plan. I 
 don't think
 we should pull ScopeBuffer yet.

 Andrei

 Mr. Alexandrescu. What about std allocator? Is feature 
 valuable? I've
 took a look in Wish List in wiki and Allocator still there.
 If this feature is valuable. What is left in currently 
 implementation of
 your project? Can you tell about state, maybe show To Do list?

 Sorry, if my english is not clear. Thank you.

 Interface with garbage collector is to be done. The rest is 
 usable right now. -- Andrei

 I thought, that general purpose of allocator it is memory 
 management without garbage collector. Because allocator will be 
 used in containers, and they must work without garbage 
 collector. If no, what is difference between containers and 
 arrays in current implementation of D? I assumed, that 
 containers are need for increased performance within saving 
 safety code. But a using GC will not increase performance.

 Or Am I dead wrong?

You still need to cooperate with the GC and let it know which 
memory should be hands off, for example.

Or just use it for the initial allocation and then ask it 
politely to release ownership of it.

--
Paulo

"ANtlord" <antlord92 gmail.com> writes:

On Thursday, 12 February 2015 at 11:10:28 UTC, Paulo  Pinto wrote:
 On Thursday, 12 February 2015 at 08:56:53 UTC, ANtlord wrote:
 On Wednesday, 11 February 2015 at 15:56:07 UTC, Andrei 
 Alexandrescu wrote:
 On 2/11/15 4:55 AM, ANtlord wrote:
 On Monday, 17 February 2014 at 15:49:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/17/14, 5:55 AM, Dicebot wrote:
 Andrei, what is current state of std.allocator? I am 
 asking this in
 context of recent Walter Phobos proposal (ScopeBuffer) to 
 evaluate how
 feasible is to define any relations between two at current 
 stage.

 Progress on std.allocator is slow but I do have a plan. I 
 don't think
 we should pull ScopeBuffer yet.

 Andrei

 Mr. Alexandrescu. What about std allocator? Is feature 
 valuable? I've
 took a look in Wish List in wiki and Allocator still there.
 If this feature is valuable. What is left in currently 
 implementation of
 your project? Can you tell about state, maybe show To Do 
 list?

 Sorry, if my english is not clear. Thank you.

 Interface with garbage collector is to be done. The rest is 
 usable right now. -- Andrei

 I thought, that general purpose of allocator it is memory 
 management without garbage collector. Because allocator will 
 be used in containers, and they must work without garbage 
 collector. If no, what is difference between containers and 
 arrays in current implementation of D? I assumed, that 
 containers are need for increased performance within saving 
 safety code. But a using GC will not increase performance.

 Or Am I dead wrong?

 You still need to cooperate with the GC and let it know which 
 memory should be hands off, for example.

 Or just use it for the initial allocation and then ask it 
 politely to release ownership of it.

 --
 Paulo

Aaa, thank you for explanation. I've got it. If the rest is 
usable right now and interface with GC. Why is not this module in 
std? Is it in queue for review?

"Piotrek" <no_data no_data.pl> writes:

Hi,

Sorry for putting it here but I don't know where to file a bug 
report for the allocator project.

On 32-bit linux with the latest dmd beta I get errors for ulong 
-> uint (size_t) conversions.

dmd -main -unittest allocator.d

allocator.d(2015): Error: cannot implicitly convert expression (i 
* 4096LU) of type ulong to uint
allocator.d(2015): Error: cannot implicitly convert expression 
((i + cast(ulong)blocks) * 4096LU) of type ulong to uint
allocator.d(1743): Error: template instance 
std.allocator.HeapBlock!(4096u, 4u) <cut the long line>
allocator.d(331):        instantiated from here: 
HeapBlock!(4096u, 4u)
allocator.d(334): Error: template instance Segregator! <cut the 
long line>
allocator.d(2015): Error: cannot implicitly convert expression (i 
* 128LU) of type ulong to uint
allocator.d(2015): Error: cannot implicitly convert expression 
((i + cast(ulong)blocks) * 128LU) of type ulong to uint
allocator.d(1743): Error: template instance 
std.allocator.HeapBlock!(128u, 4u) <cut the long line>
  , __ctmp2303).this(m)) error instantiating
allocator.d(1342):        instantiated from here: 
HeapBlock!(128u, 4u)
allocator.d(1493): Error: cannot implicitly convert expression (x 
/ 64LU) of type ulong to immutable(uint)
allocator.d(1495): Error: cannot implicitly convert expression (y 
/ 64LU) of type ulong to immutable(uint)
allocator.d(1520): Error: cannot implicitly convert expression (x 
/ 64LU) of type ulong to uint
allocator.d(1526): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1527): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1544): Error: cannot implicitly convert expression 
(w) of type ulong to uint
allocator.d(1553): Error: cannot implicitly convert expression 
(w) of type ulong to uint
allocator.d(1572): Error: cannot implicitly convert expression 
(w) of type ulong to uint
allocator.d(1582): Error: cannot implicitly convert expression 
(w) of type ulong to uint
allocator.d(1607): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1615): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1627): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(1633): Error: cannot implicitly convert expression 
(i) of type ulong to uint
allocator.d(4143): Error: function 
std.allocator.roundUpToMultipleOf (uint s, uint base) is not 
callable using argument types (ulong, uint)


Is it a known issue? Or are there currently only 64-bit OSes 
supported by the allocator project?

Piotrek

"ANtlord" <antlord92 gmail.com> writes:

On Thursday, 26 February 2015 at 21:01:27 UTC, Piotrek wrote:
 Hi,

 Sorry for putting it here but I don't know where to file a bug 
 report for the allocator project.

 On 32-bit linux with the latest dmd beta I get errors for ulong 
 -> uint (size_t) conversions.

 dmd -main -unittest allocator.d

 allocator.d(2015): Error: cannot implicitly convert expression 
 (i * 4096LU) of type ulong to uint
 allocator.d(2015): Error: cannot implicitly convert expression 
 ((i + cast(ulong)blocks) * 4096LU) of type ulong to uint
 allocator.d(1743): Error: template instance 
 std.allocator.HeapBlock!(4096u, 4u) <cut the long line>
 allocator.d(331):        instantiated from here: 
 HeapBlock!(4096u, 4u)
 allocator.d(334): Error: template instance Segregator! <cut the 
 long line>
 allocator.d(2015): Error: cannot implicitly convert expression 
 (i * 128LU) of type ulong to uint
 allocator.d(2015): Error: cannot implicitly convert expression 
 ((i + cast(ulong)blocks) * 128LU) of type ulong to uint
 allocator.d(1743): Error: template instance 
 std.allocator.HeapBlock!(128u, 4u) <cut the long line>
  , __ctmp2303).this(m)) error instantiating
 allocator.d(1342):        instantiated from here: 
 HeapBlock!(128u, 4u)
 allocator.d(1493): Error: cannot implicitly convert expression 
 (x / 64LU) of type ulong to immutable(uint)
 allocator.d(1495): Error: cannot implicitly convert expression 
 (y / 64LU) of type ulong to immutable(uint)
 allocator.d(1520): Error: cannot implicitly convert expression 
 (x / 64LU) of type ulong to uint
 allocator.d(1526): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1527): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1544): Error: cannot implicitly convert expression 
 (w) of type ulong to uint
 allocator.d(1553): Error: cannot implicitly convert expression 
 (w) of type ulong to uint
 allocator.d(1572): Error: cannot implicitly convert expression 
 (w) of type ulong to uint
 allocator.d(1582): Error: cannot implicitly convert expression 
 (w) of type ulong to uint
 allocator.d(1607): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1615): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1627): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(1633): Error: cannot implicitly convert expression 
 (i) of type ulong to uint
 allocator.d(4143): Error: function 
 std.allocator.roundUpToMultipleOf (uint s, uint base) is not 
 callable using argument types (ulong, uint)


 Is it a known issue? Or are there currently only 64-bit OSes 
 supported by the allocator project?

 Piotrek

I think, that if use this project 
https://github.com/andralex/std_allocator/, than you can post the 
issue to related issue tracker. And I see, that types in 
traceback are different from source 
https://github.com/andralex/std_allocator/blob/master/source/s
d/allocator.d#L857. 
Maybe you need to upgrade package.

Sorry, if my english isn't clear.

"Piotrek" <no_data no_data.pl> writes:

On Friday, 27 February 2015 at 08:18:53 UTC, ANtlord wrote:
 I think, that if use this project 
 https://github.com/andralex/std_allocator/, than you can post 
 the issue to related issue tracker.

Oh, I must be blind. I thought the issue tracker was disables on 
the repository in the same way as for Phobos. Thanks for 
checking. I submitted the issue.

 And I see, that types in traceback are different from source 
 https://github.com/andralex/std_allocator/blob/master/source/s
d/allocator.d#L857. 
 Maybe you need to upgrade package.

If you mean size_t and uint difference it's because size_t is an 
alias for uint on 32-bit os.

  roundUpToMultipleOf(size_t s, uint base)
becomes
  roundUpToMultipleOf (uint s, uint base)

Piotrek

"bitwise" <bitwise.pvt gmail.com> writes:

On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu 
wrote:
 Code: 
 https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

Am I the only one seeing dead links?

=?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:

Am 11.09.2015 um 04:35 schrieb bitwise:
 On Thursday, 24 October 2013 at 19:53:56 UTC, Andrei Alexandrescu wrote:
 Code: https://github.com/andralex/phobos/blob/allocator/std/allocator.d

 Dox: http://erdani.com/d/phobos-prerelease/std_allocator.html

 Am I the only one seeing dead links?

The code has moved to std.experimental:
https://github.com/andralex/phobos/tree/allocator/std/experimental/allocator
http://erdani.com/d/phobos-prerelease/std_experimental_allocator.html