www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Knowledge of managed memory pointers

reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
It occurs to me that a central issue regarding the memory management
debate, and a major limiting factor with respect to options, is the fact
that, currently, it's impossible to tell a raw pointer apart from a gc
pointer.

Is this is a problem worth solving? And would it be as big an enabler to
address some tricky problems as it seems to be at face value?

What are some options? Without turning to fat pointers or convoluted
changes in the type system, are there any clever mechanisms that could be
applied to distinguish managed from unmanaged pointers. If an API could be
provided in druntime, it may be used by GC's, ARC, allocators, or systems
that operate at the barrier between languages.

Obviously it needs to be super trivial to gather this information from the
pointer...
On embedded systems with fixed/limited memory it's easy, just make the gc
allocate pages in course physically aligned blocks and check a bit indexed
by a couple of bits in pointers whether that page is owned by the GC or not.

On large scale OS's with unknown quantity (perhaps lots) of memory, it's
not so simple, but they have other advantages, like virtual memory
managers. Can virtual pages be attributed with a bit of data somehow that's
easy to look up?

What about 'hacks' like an unlikely sentinel value at ptr[-1]?
Apr 16 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
You can do anything, what fits your task, see RefCounted and 
Unique for an example on how to write smart pointers.
Apr 17 2014
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 17 April 2014 18:20, Kagamin via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 You can do anything, what fits your task, see RefCounted and Unique for an
 example on how to write smart pointers.


... what?

I don't think you understood my post.

void f(void* ptr)
{
  // was ptr allocated with malloc, or new?
}

If we knew this, we may be able to address some problems with designing
better GC's, or cross-language API's.
Apr 17 2014
parent reply "Kagamin" <spam here.lot> writes:
On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via 
Digitalmars-d wrote:

 void f(void* ptr)
 {
   // was ptr allocated with malloc, or new?


Then what?
Apr 17 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 18 April 2014 04:10, Kagamin via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via Digitalmars-d wrote:


 void f(void* ptr)
 {
   // was ptr allocated with malloc, or new?


 Then what?


Whatever. inc/dec ref, or not. core.memory.addRoot/Range, or not. That sort
of thing.

There's a persistent problem  when dealing with memory systems in D that it
must remain backward compatible with C and raw pointers, and that creates
complications.
Apr 17 2014
parent reply "Tove" <tove fransson.se> writes:
On Friday, 18 April 2014 at 00:01:25 UTC, Manu via Digitalmars-d 
wrote:

 On 18 April 2014 04:10, Kagamin via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:


 On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via 
 Digitalmars-d wrote:


 void f(void* ptr)
 {
   // was ptr allocated with malloc, or new?


 Then what?


 Whatever. inc/dec ref, or not. core.memory.addRoot/Range, or 
 not. That sort
 of thing.

 There's a persistent problem  when dealing with memory systems 
 in D that it
 must remain backward compatible with C and raw pointers, and 
 that creates
 complications.


Both NaCl and go-lang uses a trick to reserve a huge amount of 
continuos virtual memory...
Apr 18 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 18 April 2014 20:10, Tove via Digitalmars-d
<digitalmars-d puremagic.com>wrote:


 On Friday, 18 April 2014 at 00:01:25 UTC, Manu via Digitalmars-d wrote:


 On 18 April 2014 04:10, Kagamin via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:

  On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via Digitalmars-d wrote:

  void f(void* ptr)

 {
   // was ptr allocated with malloc, or new?


 Then what?


 Whatever. inc/dec ref, or not. core.memory.addRoot/Range, or not. That
 sort
 of thing.

 There's a persistent problem  when dealing with memory systems in D that
 it
 must remain backward compatible with C and raw pointers, and that creates
 complications.


 Both NaCl and go-lang uses a trick to reserve a huge amount of continuos
 virtual memory...


Yeah, that's what I imagined too... but does that work in 32bit apps?
Apr 18 2014
prev sibling next sibling parent Orvid King via Digitalmars-d <digitalmars-d puremagic.com> writes:
I think the biggest advantage to this distinction would really be the
cross-language API's, the GC can determine which pointers it owns,
although I don't believe it currently exposes this capability.

On 4/17/14, Manu via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On 17 April 2014 18:20, Kagamin via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:


 You can do anything, what fits your task, see RefCounted and Unique for
 an
 example on how to write smart pointers.


 ... what?

 I don't think you understood my post.

 void f(void* ptr)
 {
   // was ptr allocated with malloc, or new?
 }

 If we knew this, we may be able to address some problems with designing
 better GC's, or cross-language API's.
Apr 17 2014
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 17 April 2014 23:14, Orvid King via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 I think the biggest advantage to this distinction would really be the
 cross-language API's, the GC can determine which pointers it owns,
 although I don't believe it currently exposes this capability.


But in a lightning fast way? Let's imagine ARC refcounts were stored at
ptr[-1], how can we know if this is a managed pointer or not?
I think the major hurdle in an ARC implementation is distinguishing a
managed pointer from a malloc pointer without making (breaking?) changes to
the type system.

I would also find this useful in language boundary interaction though. I
have had numerous issues identifying proper handling of cross-language
memory.
Apr 17 2014
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/17/2014 08:55 AM, Manu via Digitalmars-d wrote:

 It occurs to me that a central issue regarding the memory management
 debate, and a major limiting factor with respect to options, is the fact
 that, currently, it's impossible to tell a raw pointer apart from a gc
 pointer.

 Is this is a problem worth solving? And would it be as big an enabler to
 address some tricky problems as it seems to be at face value?

 What are some options? Without turning to fat pointers or convoluted
 changes in the type system, are there any clever mechanisms that could
 be applied to distinguish managed from unmanaged pointers.


It does not matter if changes to the type system are 'convoluted'. (They 
don't need to be.)


 If an API
 could be provided in druntime, it may be used by GC's, ARC, allocators,
 or systems that operate at the barrier between languages.


There already is.

bool isGCPointer(void* ptr){
     import core.memory;
     return !!GC.addrOf(ptr);
}

void main(){
     import std.c.stdlib;
     auto x=cast(int*)malloc(int.sizeof);
     auto y=new int;
     assert(!x.isGCPointer() && y.isGCPointer());
}
Apr 17 2014
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 17 Apr 2014 10:52:19 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:


 On 04/17/2014 08:55 AM, Manu via Digitalmars-d wrote:



 If an API
 could be provided in druntime, it may be used by GC's, ARC, allocators,
 or systems that operate at the barrier between languages.


 There already is.

 bool isGCPointer(void* ptr){
      import core.memory;
      return !!GC.addrOf(ptr);
 }


I don't think this is a viable mechanism to check pointers. It's too slow.

-Steve
Apr 17 2014
next sibling parent "Kagamin" <spam here.lot> writes:
On Thursday, 17 April 2014 at 14:59:14 UTC, Steven Schveighoffer 
wrote:

 I don't think this is a viable mechanism to check pointers. 
 It's too slow.


I suggested to write a smart pointer. It could provide 
compile-time checks and whatever developer feels like.
Apr 17 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-04-17 16:59, Steven Schveighoffer wrote:


 I don't think this is a viable mechanism to check pointers. It's too slow.


Couldn't a single bit be used to indicate if it's a GC pointer or not?

-- 
/Jacob Carlborg
Apr 19 2014
parent "Yota" <yotaxp thatGoogleMailThing.com> writes:
On Saturday, 19 April 2014 at 10:30:42 UTC, Jacob Carlborg wrote:

 On 2014-04-17 16:59, Steven Schveighoffer wrote:


 I don't think this is a viable mechanism to check pointers. 
 It's too slow.


 Couldn't a single bit be used to indicate if it's a GC pointer 
 or not?


I think he's hoping more for the static type system to
provide the information.  I think it would be nifty if
the fancy layered allocators that were being designed
would return pointers with types describing the allocator
itself.  Could this sort of info be useful in shared libs?
Apr 19 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/16/2014 11:55 PM, Manu via Digitalmars-d wrote:

 It occurs to me that a central issue regarding the memory management debate,
and
 a major limiting factor with respect to options, is the fact that, currently,
 it's impossible to tell a raw pointer apart from a gc pointer.


No it isn't, in fact, the gc has to be able to tell the difference or it cannot 
do mark/sweep. It tells the difference with "does the pointer point into the gc 
memory pool."



 What about 'hacks' like an unlikely sentinel value at ptr[-1]?


That doesn't work as soon as one increments a pointer.
Apr 18 2014