www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Garbage collected pointers?

reply John Burton <john.burton jbmail.com> writes:
In the language spec here :-
https://dlang.org/spec/garbage.html#pointers_and_gc

It refers to a distinction between pointers to garbage collected 
memory and pointers that are not. In particular it says that with 
a non garbage collected pointer you can do anything  that is 
legal in C but with a garbage collected pointer there are a lot 
of undefined behaviors if you don't follow some restrictions.

My question is how do I tell if a pointer is "garbage collected" 
or not?

For example :-
* Do not store magic values into pointers, other than null.

So how do I tell if it's safe to do this for any individual 
pointer? What makes a pointer "garbage collected"?
Mar 01 2018
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 01/03/2018 11:10 PM, John Burton wrote:

 In the language spec here :-
 https://dlang.org/spec/garbage.html#pointers_and_gc
 
 It refers to a distinction between pointers to garbage collected memory 
 and pointers that are not. In particular it says that with a non garbage 
 collected pointer you can do anything  that is legal in C but with a 
 garbage collected pointer there are a lot of undefined behaviors if you 
 don't follow some restrictions.
 
 My question is how do I tell if a pointer is "garbage collected" or not?
 
 For example :-
 * Do not store magic values into pointers, other than null.
 
 So how do I tell if it's safe to do this for any individual pointer? 
 What makes a pointer "garbage collected"?


You cannot tell if a random pointer is owned by the GC or not.

If a piece of memory is allocated not by the GC in your own function, 
that's fairly easy. You'll know about it thanks to calling e.g. malloc 
versus new or .length.
Mar 01 2018
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Thursday, 1 March 2018 at 10:10:27 UTC, John Burton wrote:

 In the language spec here :-
 https://dlang.org/spec/garbage.html#pointers_and_gc

 It refers to a distinction between pointers to garbage 
 collected memory and pointers that are not. In particular it 
 says that with a non garbage collected pointer you can do 
 anything  that is legal in C but with a garbage collected 
 pointer there are a lot of undefined behaviors if you don't 
 follow some restrictions.


Basically you can do whatever you want with pointers, you may 
twart ability of GC to free an object b/c you constructed some 
value that points to GC heap. That’s pretty much it.

Second side is loosing pointers to GC owned object, this happens 
when you put pointers into eg malloced chunk of memory.

Calling GC.addRange/removeRange you can let GC know that you 
stored pointers to GC heap in some place other then stack or GC 
heap.
Mar 01 2018
prev sibling parent reply Gary Willoughby <dev nomad.so> writes:
On Thursday, 1 March 2018 at 10:10:27 UTC, John Burton wrote:

 My question is how do I tell if a pointer is "garbage 
 collected" or not?


You could try `GC.addrOf()` or `GC.query()` in core.memory.
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 7:05 AM, Gary Willoughby wrote:

 On Thursday, 1 March 2018 at 10:10:27 UTC, John Burton wrote:

 My question is how do I tell if a pointer is "garbage collected" or not?


 
 You could try `GC.addrOf()` or `GC.query()` in core.memory.


I was going to say this, but then I realized, it's not that the pointer 
points at GC-allocated memory, but that the pointer itself is stored in 
a location that is scanned by the GC. That's not as easy to figure out, 
as you have to look at stacks, added ranges, etc.

-Steve
Mar 01 2018
next sibling parent reply John Burton <john.burton jbmail.com> writes:
On Thursday, 1 March 2018 at 12:20:08 UTC, Steven Schveighoffer 
wrote:

 On 3/1/18 7:05 AM, Gary Willoughby wrote:

 On Thursday, 1 March 2018 at 10:10:27 UTC, John Burton wrote:

 My question is how do I tell if a pointer is "garbage 
 collected" or not?


 
 You could try `GC.addrOf()` or `GC.query()` in core.memory.


 I was going to say this, but then I realized, it's not that the 
 pointer points at GC-allocated memory, but that the pointer 
 itself is stored in a location that is scanned by the GC. 
 That's not as easy to figure out, as you have to look at 
 stacks, added ranges, etc.

 -Steve


Ah.... it's where the pointer is stored. That makes some sense.
Thanks
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 10:35 AM, John Burton wrote:

 On Thursday, 1 March 2018 at 12:20:08 UTC, Steven Schveighoffer wrote:

 On 3/1/18 7:05 AM, Gary Willoughby wrote:

 On Thursday, 1 March 2018 at 10:10:27 UTC, John Burton wrote:

 My question is how do I tell if a pointer is "garbage collected" or 
 not?


 You could try `GC.addrOf()` or `GC.query()` in core.memory.


 I was going to say this, but then I realized, it's not that the 
 pointer points at GC-allocated memory, but that the pointer itself is 
 stored in a location that is scanned by the GC. That's not as easy to 
 figure out, as you have to look at stacks, added ranges, etc.


 
 Ah.... it's where the pointer is stored. That makes some sense.
 Thanks


Yes, the text isn't exactly clear. For instance, C rules apply for a 
pointer stored in a block allocated by C malloc, even if it points at a 
GC piece of memory -- as long as the block hasn't been added as a root 
or range to the GC.

Also note that realistically, even though the document talks about a 
"copying GC", this probably will never happen. Many things that are 
identified as "undefined behavior" in that doc, are probably going to be 
fine. I really wish it didn't say UB for a lot of those things. It 
should really say that it's up to the GC implementation whether it's UB 
or not.

-Steve
Mar 01 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 01, 2018 10:55:34 Steven Schveighoffer via Digitalmars-d-
learn wrote:

 It should really say that it's up to the GC implementation whether it's UB
 or not.


Well, that arguably makes it UB in general then, because it can't be relied
on. By putting restrictions on the GC in general based on what types of GCs
theoretically could be used, it makes it so that D code in general could
theoretically work with any GC that fits the bill, whereas if the rules of
what was allowed changed depending on the GC being used, what was valid D
would effectively change depending on the GC.

- Jonathan M Davis
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 2:04 PM, Jonathan M Davis wrote:

 On Thursday, March 01, 2018 10:55:34 Steven Schveighoffer via Digitalmars-d-
 learn wrote:

 It should really say that it's up to the GC implementation whether it's UB
 or not.


 
 Well, that arguably makes it UB in general then, because it can't be relied
 on. By putting restrictions on the GC in general based on what types of GCs
 theoretically could be used, it makes it so that D code in general could
 theoretically work with any GC that fits the bill, whereas if the rules of
 what was allowed changed depending on the GC being used, what was valid D
 would effectively change depending on the GC.


There are a few in there, which I think are over-the-top. Such as "don't 
cast a pointer to a non-pointer", or "Do not take advantage of alignment 
of pointers to store bit flags in the low order bits". I can't conceive 
of any GC that would have fits with either of these things without 
breaking pretty much everything. While I can't conceive of it, it 
doesn't mean there isn't a reason for it. But clearly the reason isn't 
currently implemented.

-Steve
Mar 01 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 01, 2018 14:52:26 Steven Schveighoffer via Digitalmars-d-
learn wrote:

 On 3/1/18 2:04 PM, Jonathan M Davis wrote:

 On Thursday, March 01, 2018 10:55:34 Steven Schveighoffer via
 Digitalmars-d->
 learn wrote:

 It should really say that it's up to the GC implementation whether it's
 UB or not.


 Well, that arguably makes it UB in general then, because it can't be
 relied on. By putting restrictions on the GC in general based on what
 types of GCs theoretically could be used, it makes it so that D code in
 general could theoretically work with any GC that fits the bill,
 whereas if the rules of what was allowed changed depending on the GC
 being used, what was valid D would effectively change depending on the
 GC.


 There are a few in there, which I think are over-the-top. Such as "don't
 cast a pointer to a non-pointer", or "Do not take advantage of alignment
 of pointers to store bit flags in the low order bits". I can't conceive
 of any GC that would have fits with either of these things without
 breaking pretty much everything. While I can't conceive of it, it
 doesn't mean there isn't a reason for it. But clearly the reason isn't
 currently implemented.


It would not surprise me in the least if the list of what's allowed and
isn't allowed in there needs an overhaul. I just think that saying that
something is okay with one GC and not with another such that whether the
code is valid or not depends on which GC is being used is begging for
trouble. The approach of saying what's allowed or not in general makes more
sense, even if the exact list needs work. And D and the plans for D have
changed enough over time that the current list could be making assumptions
about what's reasonable that actually aren't at all reasonable at this
point, particularly as it's become clearer over time that certain types of
GCs probably aren't reasonable, and if officially disallowing some of them
simplified things, then that may be a good idea. It hasn't mattered much in
practice though, because the work on alternative GCs has been minimal. There
have been some, but thus far, we haven't really ended up in a situation
where choosing which GC you're doing to use is much of a thing.

- Jonathan M Davis
Mar 01 2018
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 01, 2018 at 02:52:26PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
[...]

 There are a few in there, which I think are over-the-top. Such as
 "don't cast a pointer to a non-pointer",


[...]

Isn't that necessary for a precise GC?

Also, AIUI the current GC already does not scan size_t[] blocks for
pointers, so if you cast a pointer to size_t and put it in such an array
with no other references to your object, you stand the chance of the GC
collecting your object while it is still live.



 or "Do not take advantage of alignment of pointers to store bit flags
 in the low order bits".


[...]

Won't a precise GC scanning for pointers to aligned objects want to skip
values that can't be an aligned pointer?  Though in D's case, being
required to be conservative would negate this.


T

-- 
There are three kinds of people in the world: those who can count, and those
who can't.
Mar 01 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/1/18 3:33 PM, H. S. Teoh wrote:

 On Thu, Mar 01, 2018 at 02:52:26PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:
 [...]

 There are a few in there, which I think are over-the-top. Such as
 "don't cast a pointer to a non-pointer",


 [...]
 
 Isn't that necessary for a precise GC?


It's not strictly necessary for a precise GC. It's necessary for a 
copying GC. In that case, if the GC decides to move your object and 
update all the references, it would miss that one. And even there, it's 
not necessary that you refrain from casting, it's just necessary to 
refrain from using that as an actual pointer in the future. I can't see 
how you might print a pointer value without casting it to an integer ;)

What is more important is that you don't use that as the ONLY reference 
to the object.


 
 Also, AIUI the current GC already does not scan size_t[] blocks for
 pointers, so if you cast a pointer to size_t and put it in such an array
 with no other references to your object, you stand the chance of the GC
 collecting your object while it is still live.


Again, the key is "no other references". The document referenced above 
just says you can't cast.


 or "Do not take advantage of alignment of pointers to store bit flags
 in the low order bits".


 [...]
 
 Won't a precise GC scanning for pointers to aligned objects want to skip
 values that can't be an aligned pointer?  Though in D's case, being
 required to be conservative would negate this.


int[] x = [1,2,3];
void[] y = x;
y = y[1 .. $]; // misaligned pointer.

It's easy to do, and it's done all over the place (buffers anyone?).

-Steve
Mar 01 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 01, 2018 15:53:08 Steven Schveighoffer via Digitalmars-d-
learn wrote:

 On 3/1/18 3:33 PM, H. S. Teoh wrote:

 Won't a precise GC scanning for pointers to aligned objects want to skip
 values that can't be an aligned pointer?  Though in D's case, being
 required to be conservative would negate this.


 int[] x = [1,2,3];
 void[] y = x;
 y = y[1 .. $]; // misaligned pointer.

 It's easy to do, and it's done all over the place (buffers anyone?).


Yeah, until recently readText could kill your program because of that, which
I found out the hard way when I had to muck around with some UTF-16 files.
It's an easy mistake to make.

- Jonathan M Davis
Mar 01 2018
prev sibling parent reply Gary Willoughby <dev nomad.so> writes:
On Thursday, 1 March 2018 at 12:20:08 UTC, Steven Schveighoffer 
wrote:

 On 3/1/18 7:05 AM, Gary Willoughby wrote:

 On Thursday, 1 March 2018 at 10:10:27 UTC, John Burton wrote:

 My question is how do I tell if a pointer is "garbage 
 collected" or not?


 
 You could try `GC.addrOf()` or `GC.query()` in core.memory.


 I was going to say this, but then I realized, it's not that the 
 pointer points at GC-allocated memory, but that the pointer 
 itself is stored in a location that is scanned by the GC. 
 That's not as easy to figure out, as you have to look at 
 stacks, added ranges, etc.

 -Steve


It would be interesting to test whether those methods handle 
these scenarios.
Mar 02 2018
parent Gary Willoughby <dev nomad.so> writes:
On Friday, 2 March 2018 at 08:44:53 UTC, Gary Willoughby wrote:

 It would be interesting to test whether those methods handle 
 these scenarios.


Yeah, it doesn't work.

https://dpaste.dzfl.pl/55116efd0c9c
Mar 02 2018