www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Relocatable objects and internal pointers

reply Matt Elkins <notreal fake.com> writes:
Hi all, I'm a C++ programmer trying to decide whether to switch 
my main focus to D, and so I'm working on a pet project using it. 
So far I really like some of the practical aspects of the 
language (built-in contracts are great, the metaprogramming is 
very accessible, and I can't enough of these compile speeds!), 
but I keep finding myself frustrated by what seem like 
expressiveness limitations (unless, as I hope, they are just 
examples of my newbieness shining through). Case in point:

In an attempt to work around one apparent limitation (previously 
asked about here 
http://forum.dlang.org/thread/eizmagtimvetoganawrr forum.dlang.org) I came up
with an idea which would require storing internal points in a struct. A very
stripped-down but illustrative example would be something like this:

[code]
struct Foo
{
     invariant
     {
         assert(m_this == &this);
     }

      disable(this);

     this(/* arguments to populate stuff */)
     {
         m_this = &this;
         /* ... populate stuff ... */
     }

     this(this)
     {
         m_this = &this;
         /* ... do more stuff ... */
     }

     private:
         Foo* m_this;
         /* ... stuff ... */
}
[/code]

This is just a piece of what I am doing, if you are wondering why 
I am bothering to save a pointer to this. However, I was doing 
some reading on D and came across a section in TDPL which said 
internal pointers are verboten because objects must be 
relocatable. Does this mean my example is invalid (e.g., the 
invariant will not hold in all circumstances)? If it is invalid, 
does that mean there are circumstances under which the post-blit 
constructor can be elided when performing a copy or copy-like 
operation (such as a move)? I've been treating it like a sort of 
copy-constructor that lacks visibility on the copied-from object, 
but maybe that's a mistake...
Jan 29 2016
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 01/29/2016 05:07 PM, Matt Elkins wrote:


      this(/* arguments to populate stuff */)
      {
          m_this = &this;
          /* ... populate stuff ... */
      }



 a section in TDPL which said internal pointers are
 verboten because objects must be relocatable. Does this mean my example
 is invalid


Yes, D explicitly bans internal pointers.


 does that mean there are circumstances under which the
 post-blit constructor can be elided when performing a copy or copy-like
 operation (such as a move)?


Definitely so. Rvalues are moved around all the time. The following 
program has two rvalue moves without calling post-blits or destructors.

struct Foo {
     this(this) {
         assert(false);    // not expected to be called in this program
     }
}

Foo makeFoo() {
     return Foo();
}

void takesFoo(Foo foo) {
}

void main() {
     Foo foo;
     foo = makeFoo();    // post-blit not called
     takesFoo(Foo());    // post-blit not called
}

Ali
Jan 29 2016
parent reply Matt Elkins <notreal fake.com> writes:
On Saturday, 30 January 2016 at 01:18:33 UTC, Ali Çehreli wrote:

 Definitely so. Rvalues are moved around all the time. The 
 following program has two rvalue moves without calling 
 post-blits or destructors.


Oi, that makes life tough. Ok, I'll figure something else out, 
then...

Thanks for the response!
Jan 29 2016
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Sat, Jan 30, 2016 at 01:21:27AM +0000, Matt Elkins via Digitalmars-d-learn
wrote:

 On Saturday, 30 January 2016 at 01:18:33 UTC, Ali Çehreli wrote:

Definitely so. Rvalues are moved around all the time. The following
program has two rvalue moves without calling post-blits or
destructors.


 
 Oi, that makes life tough. Ok, I'll figure something else out, then...


[...]

Keep in mind that D structs are conceptually different from C++ structs
(even if they are similarly implemented). D structs are supposed to be
value types with POD-like semantics; so when passing structs around they
are bit-copied into the destination and then the postblit method
(this(this)) is called to "patch up" the copy. This is unlike in C++
where you have copy ctors and dtors and operator=() to manage copying.

Because there are no copy ctors, having internal pointers can be
dangerous, since structs can move around in memory without any warning
(e.g., returning a struct from a function generally involves copying it
from the callee's stack frame into a local variable in the caller's
stack frame).

If you need something with internal pointers, you might want to consider
classes instead. Either that, or be sure to allocate your structs on the
heap instead, and work with pointers instead of the struct values
directly. (Note that this is still risky, since somebody might
dereference the pointer and get a stack copy of the struct, which will
cause problems when it then gets passed around.)


T

-- 
I am a consultant. My job is to make your job redundant. -- Mr Tom
Jan 29 2016
parent Matt Elkins <notreal fake.com> writes:
On Saturday, 30 January 2016 at 01:28:54 UTC, H. S. Teoh wrote:

 On Sat, Jan 30, 2016 at 01:21:27AM +0000, Matt Elkins via 
 Digitalmars-d-learn wrote:

 On Saturday, 30 January 2016 at 01:18:33 UTC, Ali Çehreli 
 wrote:

Definitely so. Rvalues are moved around all the time. The 
following program has two rvalue moves without calling 
post-blits or destructors.


 
 Oi, that makes life tough. Ok, I'll figure something else out, 
 then...


 [...]

 Keep in mind that D structs are conceptually different from C++ 
 structs (even if they are similarly implemented). D structs are 
 supposed to be value types with POD-like semantics; so when 
 passing structs around they are bit-copied into the destination 
 and then the postblit method (this(this)) is called to "patch 
 up" the copy. This is unlike in C++ where you have copy ctors 
 and dtors and operator=() to manage copying.

 Because there are no copy ctors, having internal pointers can 
 be dangerous, since structs can move around in memory without 
 any warning (e.g., returning a struct from a function generally 
 involves copying it from the callee's stack frame into a local 
 variable in the caller's stack frame).

 If you need something with internal pointers, you might want to 
 consider classes instead. Either that, or be sure to allocate 
 your structs on the heap instead, and work with pointers 
 instead of the struct values directly. (Note that this is still 
 risky, since somebody might dereference the pointer and get a 
 stack copy of the struct, which will cause problems when it 
 then gets passed around.)


 T


Yeah, but the whole point of what I am doing is to avoid using 
the heap; I can think of several ways to implement this if I 
relax that restriction :). I'm basically trying to make C++'s 
std::unique_ptr for resource handles, a thin wrapper which 
ensures resource cleanup and allows moving the handle. Since I'm 
putting it in my lowest-level/most-generic library with no 
visibility on how it gets used, I want it very lightweight 
(ideally zero-cost, like I can do in C++11, or at least low-cost 
[sans heap] like I could do in C++98) so that I can use it with 
the broadest range of higher-level applications.
Jan 29 2016
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/29/16 8:07 PM, Matt Elkins wrote:

 [snip]
 on D and came across a section in TDPL which said internal pointers are
 verboten because objects must be relocatable. Does this mean my example
 is invalid (e.g., the invariant will not hold in all circumstances)? If
 it is invalid, does that mean there are circumstances under which the
 post-blit constructor can be elided when performing a copy or copy-like
 operation (such as a move)? I've been treating it like a sort of
 copy-constructor that lacks visibility on the copied-from object, but
 maybe that's a mistake...


No, you cannot have internal pointers. But...

I figured out a way to have them. You just have to guarantee you don't 
copy the actual "pointer" out of the struct:

https://forum.dlang.org/post/mk5k4l$s5r$1 digitalmars.com

-Steve
Jan 29 2016
parent reply Matt Elkins <notreal fake.com> writes:
On Saturday, 30 January 2016 at 02:09:55 UTC, Steven 
Schveighoffer wrote:

 I figured out a way to have them. You just have to guarantee 
 you don't copy the actual "pointer" out of the struct:

 https://forum.dlang.org/post/mk5k4l$s5r$1 digitalmars.com


Unfortunately, that won't work for what I was trying to do. The 
stuff I elided in the comments were more pointers to other Foo 
instances, used to create a linked-list (of stack-allocated 
objects); these would still break under the conditions Ali 
described. I was only storing the this pointer so that blitted 
objects could deduce where they came from (trying to turn the 
post-blit constructor into a copy-constructor).

Thanks, though. I'm thinking that maybe D just can't express 
these semantics without substantial overhead. While somewhat 
disappointing (I came into D with stars in my eyes :)), it's not 
enough by itself to make me go back to C++, at least not just 
yet. Not when I can just use a few static ifs to do what 
previously required careful template crafting that I wouldn't 
understand 3 months later. On the other hand, I'm falling behind 
on my library books since I no longer have any time for reading 
during compilations ;).
Jan 29 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/29/16 9:35 PM, Matt Elkins wrote:

 On Saturday, 30 January 2016 at 02:09:55 UTC, Steven Schveighoffer wrote:

 I figured out a way to have them. You just have to guarantee you don't
 copy the actual "pointer" out of the struct:

 https://forum.dlang.org/post/mk5k4l$s5r$1 digitalmars.com


 Unfortunately, that won't work for what I was trying to do. The stuff I
 elided in the comments were more pointers to other Foo instances, used
 to create a linked-list (of stack-allocated objects); these would still
 break under the conditions Ali described. I was only storing the this
 pointer so that blitted objects could deduce where they came from
 (trying to turn the post-blit constructor into a copy-constructor).


Ah, so you were actually counting on the postblit to have an *invalid* 
pointer to begin with :) Yeah, that isn't going to work. In D, it's 
legal to do something like memcpy struct data (with no postblit), and 
this is done quite often in many places because of that.


 Thanks, though. I'm thinking that maybe D just can't express these
 semantics without substantial overhead. While somewhat disappointing (I
 came into D with stars in my eyes :)), it's not enough by itself to make
 me go back to C++, at least not just yet. Not when I can just use a few
 static ifs to do what previously required careful template crafting that
 I wouldn't understand 3 months later. On the other hand, I'm falling
 behind on my library books since I no longer have any time for reading
 during compilations ;).


There are some really smart people who frequent these forums, if you 
post your actual use case, you may get an answer that you hadn't thought of.

I saw you were trying to implement something like std::unique_ptr? There 


Not sure if it helps.

-Steve
Jan 29 2016
parent reply Matt Elkins <notreal fake.com> writes:
On Saturday, 30 January 2016 at 03:00:11 UTC, Steven 
Schveighoffer wrote:

 There are some really smart people who frequent these forums, 
 if you post your actual use case, you may get an answer that 
 you hadn't thought of.


Yeah, I tried that first (on the general forum, since at the time 
I didn't know about this one).


 I saw you were trying to implement something like 
 std::unique_ptr? There is 



std.typecons.Unique seems to require heap allocation, which makes 
it a far cry from std::unique_ptr.
Jan 29 2016
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/29/16 10:13 PM, Matt Elkins wrote:

 On Saturday, 30 January 2016 at 03:00:11 UTC, Steven Schveighoffer wrote:

 There are some really smart people who frequent these forums, if you
 post your actual use case, you may get an answer that you hadn't
 thought of.


 Yeah, I tried that first (on the general forum, since at the time I
 didn't know about this one).


Ah, didn't notice that. You are likely to get more eyes there, so that 
is good.


 I saw you were trying to implement something like std::unique_ptr?



 std.typecons.Unique seems to require heap allocation, which makes it a
 far cry from std::unique_ptr.


I admit I'm not familiar with either's implementation. I just know that 
they have similar functions.

-Steve
Jan 29 2016
prev sibling parent timotheecour <timothee.cour2 gmail.com> writes:
On Saturday, 30 January 2016 at 03:13:59 UTC, Matt Elkins wrote:


 std.typecons.Unique seems to require heap allocation, which 
 makes it a far cry from std::unique_ptr.


isn't unique_ptr typically for heap allocation?
eg: 
https://stackoverflow.com/questions/42910711/unique-ptr-heap-and-stack-allocation

NOTE: calypso (ldc fork) should allow internal pointers now, see 
https://github.com/Syniurge/Calypso/issues/70 (cv::Mat.step.p is 
an internal pointer)
Jan 20 2018