www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Reference counting for resource management

reply LMB <lmbarros gmail.com> writes:
Hello,

I've been reading the forums for some time, but this is my first post here :-)

I am trying to create yet another D2 wrapper for SQLite. As usual with many C
libraries, SQLite provides us some "objects" and functions to create and
destroy them. So, I decided to encapsulate these SQLite objects in a struct,
and use reference counting to destroys the objects as soon as I can safely do
so.

The problem is that I can't make it work correctly in all situations. It guess
that I am not increasing the reference count on every situation in which my
object is copied, because I got "negative" reference counts in some tests. But
this is just a guess, I am not sure at all.

So, is there any complete example on how to implement reference counting in D2?

I found some discussions about ref counting in D, like Bartoz's nice post
(http://bartoszmilewski.wordpress.com/2009/08/19/the-anatomy-of-r
ference-counting/), but not a complete working example.

If there is no example around, I'd be pleased if someone could give me any
advice. This is what one of my wrappers roughly looks like (for brevity, I
removed all code not related to reference counting):

struct Database
{
   public this(in string fileName)
   {
      sqlite3_open(toStringz(fileName), &db_);

      refCount_ = cast(uint*)(malloc(uint.sizeof));
      *refCount_ = 1;
   }

   this(this)
   body
   {
      ++(*refCount_);
   }

   Database opAssign(Database other)
   {
      db_ = other.db_;
      refCount_ = other.refCount_;
      ++(*refCount_);
      return this;
   }

   public ~this()
   {
      if (refCount_ !is null)
      {
         --(*refCount_);

         if (*refCount_ == 0)
         {
            free(refCount_);
            refCount_ = null;
            immutable status = sqlite3_close(db_);
         }
      }
   }

   private sqlite3* db_;
   private uint* refCount_;
}


Thanks!

LMB
Nov 27 2009
next sibling parent LMB <lmbarros gmail.com> writes:
LMB Wrote:


 Hello,
 
 [...]


Ouch! I thought I was posting to the "learn" group. My bad, sorry!
Nov 27 2009
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Sat, 28 Nov 2009 01:27:41 +0300, LMB <lmbarros gmail.com> wrote:


 Hello,

 I've been reading the forums for some time, but this is my first post  
 here :-)

 I am trying to create yet another D2 wrapper for SQLite. As usual with  
 many C libraries, SQLite provides us some "objects" and functions to  
 create and destroy them. So, I decided to encapsulate these SQLite  
 objects in a struct, and use reference counting to destroys the objects  
 as soon as I can safely do so.

 The problem is that I can't make it work correctly in all situations. It  
 guess that I am not increasing the reference count on every situation in  
 which my object is copied, because I got "negative" reference counts in  
 some tests. But this is just a guess, I am not sure at all.

 So, is there any complete example on how to implement reference counting  
 in D2?

 I found some discussions about ref counting in D, like Bartoz's nice  
 post  
 (http://bartoszmilewski.wordpress.com/2009/08/19/the-anatomy-of-r
ference-counting/),  
 but not a complete working example.

 If there is no example around, I'd be pleased if someone could give me  
 any advice. This is what one of my wrappers roughly looks like (for  
 brevity, I removed all code not related to reference counting):

 struct Database
 {
    public this(in string fileName)
    {
       sqlite3_open(toStringz(fileName), &db_);

       refCount_ = cast(uint*)(malloc(uint.sizeof));
       *refCount_ = 1;
    }

    this(this)
    body
    {
       ++(*refCount_);
    }

    Database opAssign(Database other)
    {
       db_ = other.db_;
       refCount_ = other.refCount_;
       ++(*refCount_);
       return this;
    }

    public ~this()
    {
       if (refCount_ !is null)
       {
          --(*refCount_);

          if (*refCount_ == 0)
          {
             free(refCount_);
             refCount_ = null;
             immutable status = sqlite3_close(db_);
          }
       }
    }

    private sqlite3* db_;
    private uint* refCount_;
 }


 Thanks!

 LMB


I attached my implementation of RefCounted pattern. Feel free to use and  
improve it.

Here is an excerpt from a letter I wrote about RefCounted. It covers its  
design, implementation and usage.


There are two templates:
RefCounted and RefObject (see implementation in
net/core/RefCounted.d). They are not classes/structs but rather
templates that are used for being mixed in other classes so that you
don't have to inherit from it. RefObject defines the following
methods:

final public uint referenceCounter() const;
final public void addRef();
final public void releaseRef();
(I'll probably also add final public void destroy(); methods that
would do nothing).

Once you mixin this template into your class, your class instances
become reference-counted (explicitly at this moment), so it could be
used without the other one (see below for details).

Mixins also have useful ability to override default implementation.
For example, destroy() would do nothing by default, and rely on
garbage-collector to recycle memory instead of calling "delete this;".
User may override it and free some resources immediately (e.g.
disconnect from remote host) before memory is recycled.

Automatic reference-counting is achieved using RefCounted template.
It's use is as follows:

struct DriverPtr
{
    mixin RefCountedT!(Driver, DriverPtr);
}

struct LinkPtr
{
    mixin RefCountedT!(Link, LinkPtr);
}

The reason is, again, flexibility. It is possible to override certain
methods when needed. I'll probably also add a default RefCounted
class, something like this:

struct RefCounted!(T)
{
    mixin RefCountedT!(T, RefCounted!(T));
}

Users will be able to use it like this:

alias RefCounted!(Link) LinkPtr;

There are other also other approaches to implement ref-counting
mechanism, but I found this one to be the most flexible one.

The two classes are best used together, but you may use one without
other, two. RefCountedT relies on the following 3 methods:

uint referenceCounter();
void addRef();
void releaseRef();

and it doesn't really care the way they are implemented. The most
convenient way is to use mixin RefObject, though.
Nov 27 2009
prev sibling parent reply Bartosz Milewski <bartosz-nospam relisoft.com> writes:
I don't think you need opAssign. The compiler will automatically use bitblit
and postblit. 

Here's my implementation of a reference counted thread id structure for
comparison:

struct Tid
{
    this(HANDLE h) 
    {
        _h = h;
        Counter * cnt = cast(Counter *) core.stdc.stdlib.malloc(Counter.sizeof);
        cnt._n = 1;
        _cnt = cast(shared Counter *) cnt;
    }
    ~this()
    {
        release();
    }
    this(this)
    {
        _cnt.inc;
    }
    // invalidates current Tid object
    Tid transfer()
    {
        Tid tid = { _h, _cnt };
        _cnt = null;
        return tid;
    }
    void release()
    {
        if (_cnt !is null)
        {
            uint newCount = _cnt.dec;
            if (newCount == 0)
            {
                CloseHandle(_h);
                core.stdc.stdlib.free(cast(Counter *)_cnt);
                _cnt = null;
            }
        }
    }
    void start()
    {
        assert(_h != INVALID_HANDLE_VALUE);
        if (ResumeThread(_h) == -1)
            throw new ThreadException("Error resuming thread");
    }
    void join(bool rethrow = true) 
    {
        if (_h != INVALID_HANDLE_VALUE)
            if (WaitForSingleObject(_h, INFINITE) != WAIT_OBJECT_0)
                throw new ThreadException("Join failed");
    }
    void setPriority() // dummy for now
    {
    }
private:
    // Revisit: implement using atomic add
    struct Counter
    {
        uint inc() shared { return ++_n; }
        uint dec() shared { return --_n; }

        uint _n = 1;
    }
private:
    HANDLE             _h = INVALID_HANDLE_VALUE;
    shared Counter * _cnt;
}

LMB Wrote:


 Hello,
 
 I've been reading the forums for some time, but this is my first post here :-)
 
 I am trying to create yet another D2 wrapper for SQLite. As usual with many C
libraries, SQLite provides us some "objects" and functions to create and
destroy them. So, I decided to encapsulate these SQLite objects in a struct,
and use reference counting to destroys the objects as soon as I can safely do
so.
 
 The problem is that I can't make it work correctly in all situations. It guess
that I am not increasing the reference count on every situation in which my
object is copied, because I got "negative" reference counts in some tests. But
this is just a guess, I am not sure at all.
 
 So, is there any complete example on how to implement reference counting in D2?
 
 I found some discussions about ref counting in D, like Bartoz's nice post
(http://bartoszmilewski.wordpress.com/2009/08/19/the-anatomy-of-r
ference-counting/), but not a complete working example.
 
 If there is no example around, I'd be pleased if someone could give me any
advice. This is what one of my wrappers roughly looks like (for brevity, I
removed all code not related to reference counting):
 
 struct Database
 {
    public this(in string fileName)
    {
       sqlite3_open(toStringz(fileName), &db_);
 
       refCount_ = cast(uint*)(malloc(uint.sizeof));
       *refCount_ = 1;
    }
 
    this(this)
    body
    {
       ++(*refCount_);
    }
 
    Database opAssign(Database other)
    {
       db_ = other.db_;
       refCount_ = other.refCount_;
       ++(*refCount_);
       return this;
    }
 
    public ~this()
    {
       if (refCount_ !is null)
       {
          --(*refCount_);
 
          if (*refCount_ == 0)
          {
             free(refCount_);
             refCount_ = null;
             immutable status = sqlite3_close(db_);
          }
       }
    }
 
    private sqlite3* db_;
    private uint* refCount_;
 }
 
 
 Thanks!
 
 LMB
Nov 29 2009
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Mon, 30 Nov 2009 03:19:33 +0300, Bartosz Milewski  
<bartosz-nospam relisoft.com> wrote:


 I don't think you need opAssign. The compiler will automatically use  
 bitblit and postblit.

 Here's my implementation of a reference counted thread id structure for  
 comparison:

 struct Tid
 {
     this(HANDLE h)
     {
         _h = h;
         Counter * cnt = cast(Counter *)  
 core.stdc.stdlib.malloc(Counter.sizeof);
         cnt._n = 1;
         _cnt = cast(shared Counter *) cnt;
     }
     ~this()
     {
         release();
     }
     this(this)
     {
         _cnt.inc;
     }
     // invalidates current Tid object
     Tid transfer()
     {
         Tid tid = { _h, _cnt };
         _cnt = null;
         return tid;
     }
     void release()
     {
         if (_cnt !is null)
         {
             uint newCount = _cnt.dec;
             if (newCount == 0)
             {
                 CloseHandle(_h);
                 core.stdc.stdlib.free(cast(Counter *)_cnt);
                 _cnt = null;
             }
         }
     }
     void start()
     {
         assert(_h != INVALID_HANDLE_VALUE);
         if (ResumeThread(_h) == -1)
             throw new ThreadException("Error resuming thread");
     }
     void join(bool rethrow = true)
     {
         if (_h != INVALID_HANDLE_VALUE)
             if (WaitForSingleObject(_h, INFINITE) != WAIT_OBJECT_0)
                 throw new ThreadException("Join failed");
     }
     void setPriority() // dummy for now
     {
     }
 private:
     // Revisit: implement using atomic add
     struct Counter
     {
         uint inc() shared { return ++_n; }
         uint dec() shared { return --_n; }

         uint _n = 1;
     }
 private:
     HANDLE             _h = INVALID_HANDLE_VALUE;
     shared Counter * _cnt;
 }

 LMB Wrote:


 Hello,

 I've been reading the forums for some time, but this is my first post  
 here :-)

 I am trying to create yet another D2 wrapper for SQLite. As usual with  
 many C libraries, SQLite provides us some "objects" and functions to  
 create and destroy them. So, I decided to encapsulate these SQLite  
 objects in a struct, and use reference counting to destroys the objects  
 as soon as I can safely do so.

 The problem is that I can't make it work correctly in all situations.  
 It guess that I am not increasing the reference count on every  
 situation in which my object is copied, because I got "negative"  
 reference counts in some tests. But this is just a guess, I am not sure  
 at all.

 So, is there any complete example on how to implement reference  
 counting in D2?

 I found some discussions about ref counting in D, like Bartoz's nice  
 post  
 (http://bartoszmilewski.wordpress.com/2009/08/19/the-anatomy-of-r
ference-counting/),  
 but not a complete working example.

 If there is no example around, I'd be pleased if someone could give me  
 any advice. This is what one of my wrappers roughly looks like (for  
 brevity, I removed all code not related to reference counting):

 struct Database
 {
    public this(in string fileName)
    {
       sqlite3_open(toStringz(fileName), &db_);

       refCount_ = cast(uint*)(malloc(uint.sizeof));
       *refCount_ = 1;
    }

    this(this)
    body
    {
       ++(*refCount_);
    }

    Database opAssign(Database other)
    {
       db_ = other.db_;
       refCount_ = other.refCount_;
       ++(*refCount_);
       return this;
    }

    public ~this()
    {
       if (refCount_ !is null)
       {
          --(*refCount_);

          if (*refCount_ == 0)
          {
             free(refCount_);
             refCount_ = null;
             immutable status = sqlite3_close(db_);
          }
       }
    }

    private sqlite3* db_;
    private uint* refCount_;
 }


 Thanks!

 LMB




I think RefCounted should be flexible enough to re-usable for other data  
structures, too (e.g. so that it could be part of Phobos).

By the way, your code has a bug:

Tid tid; // or release() and exisiting Tid
Tid tid2 = tid; // segfault since you don't check if _cnt is null in  
postblit
Nov 29 2009
parent Bartosz Milewski <bartosz-nospam relisoft.com> writes:
Denis Koroskin Wrote:


 I think RefCounted should be flexible enough to re-usable for other data  
 structures, too (e.g. so that it could be part of Phobos).
 


I agree. However this code goes into core, so it can't use Phobos.


 By the way, your code has a bug:
 
 Tid tid; // or release() and exisiting Tid
 Tid tid2 = tid; // segfault since you don't check if _cnt is null in  
 postblit


I realize that, but I'm not sure if this would be a library bug or a user bug.
I wish one could just prevent such usage, e.g., by nulling the default
constructor:

struct Tid {
    this() = null;
    ...
}

This is a more general problem of the absence of user-defined default
constructors for structs. This makes perfect sense for PODS, but not for
structs that have a destructor (and postblit). Of course, if the default
constructor is blocked, you wouldn't be able to have arrays of such structs,
which, as a side effect, would eliminate the problem of exceptions thrown from
default constructors and array unwinding. It's a bit of a minefield there.
Nov 30 2009