www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - tanya library 0.2.0

reply Eugene Wissner <belka caraus.de> writes:
It isn't really a release announce, maybe a pre-release.

tanya is a general purpose library, used mostly for networking by 
me. It is an attempt to develop an alternative memory model for 
D; 100% of the library are usable in  nogc code.

I previously announced that I want to merge my code with another 
project and that I would stop the development. But it didn't 
happen. I'm sorry that I wanted to abandon it. It won't happen 
anymore. I'll continue the development.

tanya contains an event loop, containers, an URL parsing routine 
and alternative TCP sockets implementation, multiple precision 
integer. The library is cross plattform, but not thread-safe yet.

The memory management is based on allocators. The allocators are 
one-way compatible with std.experimental.allocator, it means my 
allocators can be used with std.exeperimental.allocator but the 
std.experimental.allocator isn't usable with my lbirary (though 
it is pretty straightforward to write a wrapper for phobos 
allocators). tanya's allocators follow Bloomberg Allocator Model, 
see discussions on BDE Allocator Model for pro and contra.

And sorry once again for a long post...

What's new:
- Bug fixes and performance improvements in the main allocator: 
MmapPool.
- Container redesign: There is a new container, Vector (similar 
to std.container.array) with the support of ranges and all the 
cool stuff. I also started to experiment with the "move" 
semantics in D based on the D conf talk of Ali Çehreli. There are 
also a singly linked list, circular buffer and a queue. SList 
will get range support soon, and some time later the buffers.
- Bug fixes in the epoll event loop and a few new features.
- RefCounted: more similar to C++ shared_ptr than to phobos' 
RefCounted, but more primitive.

Future plans:
I still need a few more containers: UTF-8 string and a hash 
table. The string won't be the same as RCString. I'm thinking of 
a not templated string with UTF-8 support only (with a 
possibility to initialize it from a string, dstring and wstring).
Big integer bug fixing and optimization has a priority aswell for 
me.
I'm also planning to add streams but it won't happen very soon 
since it is a lot of work.

I plan to start a small blog in April based on the library 
(university project). It will be a SCGI-server, serving only 
static pages, behind Apache. It will be the first try-out :). 
 From there I can move to a http-server.

The library is still under permanent development. I begin to care 
about the API stability and proper deprecations, but it isn't 
always possible.
I intend to make releases regularly, maybe all the 6 weeks and to 
prepare some new features for each of the releases.
The API docs for the latest version can be found on 
https://docs.caraus.io/tanya/
If you have questions, need some examples, let me know. I'll 
write something up and publish it somewhere.

https://github.com/caraus-ecms/tanya
http://code.dlang.org/packages/tanya
https://docs.caraus.io/tanya/
Feb 18 2017
parent reply Seb <seb wilzba.ch> writes:
On Saturday, 18 February 2017 at 15:51:59 UTC, Eugene Wissner 
wrote:

 It isn't really a release announce, maybe a pre-release.

 tanya is a general purpose library, used mostly for networking 
 by me. It is an attempt to develop an alternative memory model 
 for D; 100% of the library are usable in  nogc code.


Sounds really cool!



 tanya contains an event loop, containers, an URL parsing 
 routine and alternative TCP sockets implementation, multiple 
 precision integer. The library is cross plattform, but not 
 thread-safe yet.


Do you know about eventcore [1] (event loop abstraction)?
I hope that this will soon be submitted to Phobos as everyone 
seems to be brewing their own solution and I am _not_ looking 
forward to manage to find a way to run five event loops when I 
want to use fix different libraries.

[1] https://github.com/vibe-d/eventcore


 The memory management is based on allocators. The allocators 
 are one-way compatible with std.experimental.allocator, it 
 means my allocators can be used with 
 std.exeperimental.allocator but the std.experimental.allocator 
 isn't usable with my lbirary (though it is pretty 
 straightforward to write a wrapper for phobos allocators). 
 tanya's allocators follow Bloomberg Allocator Model, see 
 discussions on BDE Allocator Model for pro and contra.


 From [2] and your API I get that the BDE API is:

     virtual void* allocate(size_type size)
     virtual void deallocate(void *address)

whereas you extended this by:

    int alignment()
    void[] allocate(in size_t size)
    bool deallocate(void[] p)
    bool reallocate(ref void[] p, in size_t size)
    bool reallocateInPlace(ref void[] p, in size_t size)

Well obviously the std.experimental.allocator protocol is more 
complicated:

     uint alignment()
     void[] allocate(size_t, TypeInfo ti = null)
     void[] alignedAllocate(size_t n, uint a)
     void[] allocateAll()
     bool expand(ref void[], size_t);
     bool reallocate(ref void[], size_t);
     bool alignedReallocate(ref void[] b, size_t size, uint 
alignment);
     Ternary owns(void[] b);
     Ternary resolveInternalPointer(void* p, ref void[] result);
     bool deallocate(void[] b);
     bool deallocateAll();
     Ternary empty();

However I would be very interested in hearing your reasons for 
not using it.
Keep in mind that std.experimental.allocator is supposed to be a 
general purpose library exactly to avoid everyone writing their 
own, incompatible Allocators ;-)
So please raise your voice as long as it's still in experimental!

[2] https://github.com/bloomberg/bde/wiki/BDE-Allocator-model
Feb 18 2017
parent reply Eugene Wissner <belka caraus.de> writes:
On Saturday, 18 February 2017 at 23:08:52 UTC, Seb wrote:

 Do you know about eventcore [1] (event loop abstraction)?
 I hope that this will soon be submitted to Phobos as everyone 
 seems to be brewing their own solution and I am _not_ looking 
 forward to manage to find a way to run five event loops when I 
 want to use fix different libraries.

 [1] https://github.com/vibe-d/eventcore


Yes. The event loop is the oldest part of tanya. At the time I 
began to work on it, eventcore was very young and got very seldom 
updates. I only heard it is the new fast and native event loop 
for the new generation of vibe.d. There was libasync aswell which 
was written as a native replacement for libevent, but was never 
adopted more than just as an option; instead eventcore was 
written from the ground up. Botan suffered an even worse fate. So 
I wasn't sure what direction eventcore would take. No of the 
event loops, I was aware of, had IOCP support on Windows. For me 
windows wasn't that important, but I implemented it to ensure 
that the same API can work with both, Posix-style events and the 
asynchronous Windows completion ports; and I could really fix 
some design flaws implementing IOCP. I don't know if I really 
want to have such an abstraction included in phobos, since there 
are bindings for C event loops and a few native ones, and maybe 
they will continue to exist, regardless whether there is 
something in Phobos or not. But yes, if Phobos would have 
something for asynchronous programming, I would take a look into 
it again.


 However I would be very interested in hearing your reasons for 
 not using it.
 Keep in mind that std.experimental.allocator is supposed to be 
 a general purpose library exactly to avoid everyone writing 
 their own, incompatible Allocators ;-)
 So please raise your voice as long as it's still in 
 experimental!

 [2] https://github.com/bloomberg/bde/wiki/BDE-Allocator-model


Thanks for the question. I had to describe it from the beginning 
or in the README, but didn't want to blow my message up.

BDE allocators implement an interface, other allocators normally 
not. It is the main point. It is not about a concrete interface. 
Phobos allocators are structs, they can't implement anything and 
there is a CAllocatorImpl, that implements IAllocator and wraps 
other allocators, so structs can behave as if they would 
implement IAllocator too. But these structs can be used directly 
without CallocatorImpl in the templated code.
So if you have a container, you can do:
auto arr = Array!(int, Mallocator)(Mallocator.instance);

EMSIContainers do that and it is the way Phobos allocators and 
containers will go. 
(https://github.com/economicmodeling/containers)

If you don't want to template your code or you use phobos default 
allocator, you do the following (because theAllocator should have 
some type):
theAllocator = allocatorObject(Mallocator.instance);


Contras of my allocators:
1) they are slower (because of virtual function calls)
2) If the allocator is passed as template parameter, functions 
that rely on the allocator can infer the attributes like  safe, 
 nogc, pure (yes, there was a discussion to make malloc pure).

Pros:
1) "is-a" relation. Mallocator "is-a" allocator, MmapAllocator 
"is-a" allocator. In your code if you use allocators as a 
template parameter (the faster way), you can't rely that the 
allocator implements some method. So your code will look like:

static if (allocator has method "expand")
{
     if (allocator.expand(arr, 8))
     {
     }
}

2) Less code bloat. With phobos allocators you would make as much 
as possible to templates.

3) I see absolute no reason, why Array!(int, Mallocator) and 
Array!(int, GCAllocator) has different types. Do you want a 
function that accepts an array:

void myFunc(Array!int arr);

you have to make a template function from it (with a long "if 
check" that ensures that an Array!int is passed). Code bloat.

4) It just works everywhere. Phobos allocators are structs but 
there is still IAllocator interface, because the model isn't 
universal enough.

5) I started by implementing IAllocator and the reason I dropped 
it then, is IAllocator.expand. If you have an array of type A 
where A is a struct with a postblit constructor, you can't just 
call realloc() to resize the array. realloc() can move memory and 
if an object of type A has references to other objects in the 
array, the objects will be corrupted. "A" should be a POD-type. 
Otherwise you have to allocate new memory, initialize it, copy 
the objects by one and deallocate the old memory. Now there is 
IAllocator.expand(). If IAllocator.expand() can expand the memory 
in place, everything is fine, you haven't to copy the objects 
because the addresses don't change. But libc realloc for example 
doesn't guarantee that the address doesn't change if you shrink 
the memory. But IAllocator doesn't have a shrink() method.
And anyway I find this separation expand/shrink or 
expandArray/shrinkArray inconvenient; I had to check everywhere 
if the new size is larger or smaller than the old one, then call 
expandArray or shrinkArray with a delta size (maybe it was only 
my use case). Therefore I replaced it reallocateInPlace.
Feb 19 2017
parent reply Kagamin <spam here.lot> writes:
On Sunday, 19 February 2017 at 11:41:44 UTC, Eugene Wissner wrote:

 realloc() can move memory and if an object of type A has 
 references to other objects in the array, the objects will be 
 corrupted. "A" should be a POD-type. Otherwise you have to 
 allocate new memory, initialize it, copy the objects by one and 
 deallocate the old memory. Now there is IAllocator.expand().


What's the difference between realloc+postblit and copy one by 
one? Postblit is called only after copy, it's not a constructor.
Feb 22 2017
parent Eugene Wissner <belka caraus.de> writes:
On Wednesday, 22 February 2017 at 13:00:19 UTC, Kagamin wrote:

 On Sunday, 19 February 2017 at 11:41:44 UTC, Eugene Wissner 
 wrote:

 realloc() can move memory and if an object of type A has 
 references to other objects in the array, the objects will be 
 corrupted. "A" should be a POD-type. Otherwise you have to 
 allocate new memory, initialize it, copy the objects by one 
 and deallocate the old memory. Now there is 
 IAllocator.expand().


 What's the difference between realloc+postblit and copy one by 
 one? Postblit is called only after copy, it's not a constructor.


The difference is that realloc will deallocate the old memory 
before postblit.
For the sake of example:

struct Sample
{
	Sample* sample;

	this(this)
	{
		sample.sample = &this;
	}
}

void main()
{
	auto s = new Sample[2];

	s[0].sample = &s[1];
	s[1].sample = &s[0];
}

Now suppose s[0] has the address 7fc8d2717000, s[1]: 7fc8d2717008.
Realloc cannot reallocate in place and after the call to realloc 
s[0] has the address 7f27771e0000 and s[1]: 7f27771e0000. It is 
too late to call the postblit here, Sample.sample refers to 
invalid memory.
Feb 22 2017