www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Interfacing to C - Storage Allocation - eek?

reply Mike Capp <mike.capp gmail.com> writes:
From the current page:

"If pointers to D garbage collector allocated memory are passed to C functions,
it's critical to ensure that that memory will not be collected by the garbage
collector before the C function is done with it."

Various suggestions on how to prevent this are offered. Well and good, for now.
What happens if and when D gets a copying GC implementation?

The first suggestion - copying to non-GCed memory - is not appealing for large
data blocks. The others would presumably throw up hard-to-diagnose wild pointer
errors in rare and hard-to-reproduce circumstances.
Aug 02 2005
next sibling parent reply llothar <llothar_member pathlink.com> writes:
In article <dcorea$2pov$1 digitaldaemon.com>, Mike Capp says...

Various suggestions on how to prevent this are offered. Well and good, for now.
What happens if and when D gets a copying GC implementation?


If the GC changes in such a way, all non D code must be completely rewritten. 

The usual way for a copying garbage collector is to offer indirect pointers to
the referrenced value, while non copying collectors simply pass a direct
pointer.
Aug 02 2005
parent Mike Capp <mike.capp gmail.com> writes:
In article <dcosu3$2r13$1 digitaldaemon.com>, llothar says...

If the GC changes in such a way, all non D code must be completely rewritten. 


Hence the "eek". Completely rewriting win32, for example, is not a viable
option. And yet the docs and many previous postings have indicated that a
copying GC at some point is possible.

cheers
Mike
Aug 02 2005
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <dcorea$2pov$1 digitaldaemon.com>, Mike Capp says...

From the current page:

"If pointers to D garbage collector allocated memory are passed to C functions,
it's critical to ensure that that memory will not be collected by the garbage
collector before the C function is done with it."

Various suggestions on how to prevent this are offered. Well and good, for now.
What happens if and when D gets a copying GC implementation?


The GC would offer a method to "pin" memory that should not be moved.


Sean
Aug 02 2005
parent reply Mike Capp <mike.capp gmail.com> writes:
In article <dcou45$2s21$1 digitaldaemon.com>, Sean Kelly says...

What happens if and when D gets a copying GC implementation?


The GC would offer a method to "pin" memory that should not be moved.


Exactly. Existing D code rebuilt with such an implementation won't ever call
that method and will _silently_ become horribly unsafe. That's going to be
popular.

Plus, of course, many if not most calls to C APIs will have to be wrapped with
pin/unpin calls. Verbose, ugly and easy to forget. Even worse with asynchronous
operations or userdata pointers for callbacks.

cheers
Mike
Aug 02 2005
parent reply Sean Kelly <sean f4.ca> writes:
In article <dcovni$2svo$1 digitaldaemon.com>, Mike Capp says...

In article <dcou45$2s21$1 digitaldaemon.com>, Sean Kelly says...

What happens if and when D gets a copying GC implementation?


The GC would offer a method to "pin" memory that should not be moved.


Exactly. Existing D code rebuilt with such an implementation won't ever call
that method and will _silently_ become horribly unsafe. That's going to be
popular.


Once I get threading finished for Ares I'm planning to look at the GC interface.
I'll add 'pin' functions in the process, even if they do nothing with the
current GC.  I'll try and remember to ask that the same be done for Phobos.


Plus, of course, many if not most calls to C APIs will have to be wrapped with
pin/unpin calls. Verbose, ugly and easy to forget. Even worse with asynchronous
operations or userdata pointers for callbacks.


So what would you suggest?  Auto classes might help to streamline the process,
but I can't think of any way completely automate it.


Sean
Aug 02 2005
parent reply Mike Capp <mike.capp gmail.com> writes:
In article <dcp03v$2t8k$1 digitaldaemon.com>, Sean Kelly says...

So what would you suggest? 


Shrug. I'd suggest that GC is a lousy official memory management strategy for a
language like D, but I've already done that and seem to be in a minority of one.
(Cue 'Hearts and Flowers'...)


but I can't think of any way completely automate it.


Not at the library level, no. It might be partially possible at the compiler
level - "whenever you see a pointer argument being passed to a function declared
as extern (C), insert pin/unpin calls for that pointer around the function
invocation" - but that doesn't help for the async and userdata-callback cases.
Still, those are maybe rare enough that requiring a manual pin is acceptable.

cheers
Mike
Aug 02 2005
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <dcp2vg$2v6p$1 digitaldaemon.com>, Mike Capp says...

In article <dcp03v$2t8k$1 digitaldaemon.com>, Sean Kelly says...

but I can't think of any way completely automate it.


Not at the library level, no. It might be partially possible at the compiler
level - "whenever you see a pointer argument being passed to a function declared
as extern (C), insert pin/unpin calls for that pointer around the function
invocation" - but that doesn't help for the async and userdata-callback cases.


There are two cases where a pointer may be passed to a C function:

- it is used for the function call, and no references persist after return
- the function stores this pointer in a static location and continues using it
after return (callbacks and such)

For the latter case, auto-unpinning is incorrect behavior.  Also, pin/unpin is
currently only necessary in the first case if the application is multithreaded,
as the current D GC will only cleanup during a call to 'new'.


Still, those are maybe rare enough that requiring a manual pin is acceptable.


Perhaps an auto class would facilitate this a tad.  Not ideal, but it's
something.


Sean
Aug 02 2005
parent reply Mike Capp <mike.capp gmail.com> writes:
In article <dcp6pf$b5$1 digitaldaemon.com>, Sean Kelly says...

There are two cases where a pointer may be passed to a C function:

- it is used for the function call, and no references persist after return
- the function stores this pointer in a static location and continues using it
after return (callbacks and such)

For the latter case, auto-unpinning is incorrect behavior.


Is it? I don't how pinning is implemented in garbage collectors; I was imagining
that you could stick multiple pins in a chunk of memory. Auto-unpinning would
then be pointless but harmless as long as the user had also done a manual pin.


Also, pin/unpin is
currently only necessary in the first case if the application is multithreaded,
as the current D GC will only cleanup during a call to 'new'.


Not _strictly_ true. If you call a C function (fc) which takes both a data
pointer (pd) and a function pointer (pf), and you supply a function in your D
module (fd) as the value of pf, and fc calls pf before accessing pd, and fd
happens to call 'new'...


Perhaps an auto class would facilitate this [async case] a tad.  
Not ideal, but it's something.


The more I look at auto the less I'm convinced it facilitates anything. As
currently implemented it appears to be a variant syntax for try-finally, only
with added limitations and extra cost. I don't see how it could help here, since
autos are limited to local scope and an asynchronous function guaranteed to
complete by the end of a local scope doesn't sound very asynchronous to me.

cheers
Mike
Aug 02 2005
parent Brad Beveridge <brad somewhere.net> writes:
I think that it depends on the pin/unpin syntax.  Obviously you should 
be able to pin/unpin a chunk of memory at any time, but how about also 
being able to pin the memory at variable assign time.

pinned char myBuffer[];

which tells the GC to treat myBuffer in exactly the same way that the 
current GC treats it.  Ie, it can move if you reallocated it, but the 
copying collector will not move it.

Brad
Aug 03 2005
prev sibling parent reply Shammah Chancellor <Shammah_member pathlink.com> writes:
In article <dcp2vg$2v6p$1 digitaldaemon.com>, Mike Capp says...

In article <dcp03v$2t8k$1 digitaldaemon.com>, Sean Kelly says...

So what would you suggest? 


Shrug. I'd suggest that GC is a lousy official memory management strategy for a
language like D, but I've already done that and seem to be in a minority of one.
(Cue 'Hearts and Flowers'...)


You earlier stated that you found pin / unpin as being ugly:

    Plus, of course, many if not most calls to C APIs will have to be wrapped
with
    pin/unpin calls. Verbose, ugly and easy to forget. Even worse with
asynchronous
    operations or userdata pointers for callbacks.

Yet you, in the msg I quoted, say that you'd rather there be no GC.  So you'd
rather be doing ugly verbose memory management everywhere than a few ugly
verbose pin/unpin calls when you occasionally need to call into a C library?

-Sha
Aug 03 2005
parent reply Mike Capp <mike.capp gmail.com> writes:
In article <dcr6l7$1l46$1 digitaldaemon.com>, Shammah Chancellor says...

You earlier stated that you found pin / unpin as being ugly:


If manually wrapped around almost every C API call, yes. Particularly since D's
easy integration with C means that it's not necessarily blindingly obvious which
calls are to C APIs.


Yet you, in the msg I quoted, say that you'd rather there be no GC. 


No, I said I thought it was a lousy official memory management strategy. My
problem isn't with D having GC; it's with D not having much else. GC is a
reasonable approach to managing memory, but it's a completely worthless approach
to managing most other resources. When non-memory resources are tightly coupled
to objects, e.g. handles in an SWT-like GUI library that uses native peers, that
inadequacy spills over.


So you'd
rather be doing ugly verbose memory management everywhere than a few ugly
verbose pin/unpin calls when you occasionally need to call into a C library?


Resource management (and not just memory) in a language with deterministic dtors
is far from ugly or verbose. (My current C++ project contains a grand total of
one 'delete' and one 'delete[]'.) And it's consistent; the same mechanism works
for everything. You don't have to worry about whether class Foo needs to be
Dispose()d or Close()d or whatever, and when, and what happens to the 'zombie'
object afterwards. You just let it go. The class knows how to clean itself up,
and does so predictably as soon as possible and no sooner. Yes, there are costs.
Yes, it takes a bit of discipline. IMHO it's worth it.

cheers
Mike
Aug 03 2005
parent Shammah Chancellor <Shammah_member pathlink.com> writes:
In article <dcrcql$1qpi$1 digitaldaemon.com>, Mike Capp says...

In article <dcr6l7$1l46$1 digitaldaemon.com>, Shammah Chancellor says...

You earlier stated that you found pin / unpin as being ugly:


If manually wrapped around almost every C API call, yes. Particularly since D's
easy integration with C means that it's not necessarily blindingly obvious which
calls are to C APIs.


Eh?  D Requires imports, just like perl or any other language.  With the proper
imports any calls to C 
from perl, php, qbasic, or whatever else look the same as a normal call.  If
you're talking about the 
imports already in phobos.  I should hope you know which functions are in
std.c.* that you're 
importing.  If not, you can easily check by removing that import and looking at
the slew of errors.


Yet you, in the msg I quoted, say that you'd rather there be no GC. 


No, I said I thought it was a lousy official memory management strategy. My
problem isn't with D having GC; it's with D not having much else. GC is a
reasonable approach to managing memory, but it's a completely worthless approach
to managing most other resources. When non-memory resources are tightly coupled
to objects, e.g. handles in an SWT-like GUI library that uses native peers, that
inadequacy spills over.


So you'd
rather be doing ugly verbose memory management everywhere than a few ugly
verbose pin/unpin calls when you occasionally need to call into a C library?


Resource management (and not just memory) in a language with deterministic dtors
is far from ugly or verbose. (My current C++ project contains a grand total of
one 'delete' and one 'delete[]'.) And it's consistent; the same mechanism works
for everything. You don't have to worry about whether class Foo needs to be
Dispose()d or Close()d or whatever, and when, and what happens to the 'zombie'
object afterwards. You just let it go. The class knows how to clean itself up,
and does so predictably as soon as possible and no sooner. Yes, there are costs.
Yes, it takes a bit of discipline. IMHO it's worth it.


Congradulations, you spent a helluva alot of time getting your stuff setup
perfectly and have no 
memory leaks.  (Which is a big problem with C++ programs...  Exactly the reason
there is a huge 
market for memory debuggers.  Have you used one of these on your program? you
might be surprised.)

Being required to pin a variable which is getting passed to a C function is not
that big of a deal IMO.  If 
if becomes such a big problem, it might not be hard to auto-pin references
passed to stuff which is 
extern(C)  I don't know.
Aug 03 2005