www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Advice wanted on garbage collection of sockets for c++ programmer

reply John Burton <john.burton jbmail.com> writes:
I'm coming from a C++ background so I'm not too used to garbage 
collection and it's implications. I have a function that creates 
a std.socket.Socket using new and connects to a tcp server, and 
writes some stuff to it. I then explicitly close the socket, and 
the socket object goes out of scope.

I assume that I do need to explicitly call close on the socket as 
there is no deterministic destructor for class objects. I further 
assume that the runtime will garbage collect any memory allocated 
to the socket object at a later time.

Am I doing this right with GC? In C++ I'd ensure that the Socket 
class had a destructor that closed the socket and I'd also assume 
that once it went out of scope there was no memory left 
allocated. In D am I right to assume I need to manually close the 
socket but there is no need to worry about the memory?

Now the issue is that I now need to call this function more than 
once every second. I worry that it will create large amounts of 
uncollected "garbage" which will eventually lead to problems.

Am I doing this right? Or is there a better way to do this in D?

Thanks.
Jun 27 2017
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:

 I'm coming from a C++ background so I'm not too used to garbage 
 collection and it's implications. I have a function that 
 creates a std.socket.Socket using new and connects to a tcp 
 server, and writes some stuff to it. I then explicitly close 
 the socket, and the socket object goes out of scope.

 I assume that I do need to explicitly call close on the socket 
 as there is no deterministic destructor for class objects. I 
 further assume that the runtime will garbage collect any memory 
 allocated to the socket object at a later time.

 Am I doing this right with GC? In C++ I'd ensure that the 
 Socket class had a destructor that closed the socket and I'd 
 also assume that once it went out of scope there was no memory 
 left allocated. In D am I right to assume I need to manually 
 close the socket but there is no need to worry about the memory?


Yes. from the documentation


Immediately drop any connections and release socket resources. 
Calling shutdown before close is recommended for 
connection-oriented sockets. The Socket object is no longer 
usable after close. Calling shutdown() before this is 
recommended for connection-oriented sockets.


The GC (if you use it) will pick up the garbage for you.


 Now the issue is that I now need to call this function more 
 than once every second. I worry that it will create large 
 amounts of uncollected "garbage" which will eventually lead to 
 problems.


Have a look at automem (https://github.com/atilaneves/automem/)


 Am I doing this right? Or is there a better way to do this in D?

 Thanks.


There is also vibe.d if you are looking for more networking stuff.
https://github.com/rejectedsoftware/vibe.d

Best of luck
Nic
Jun 27 2017
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Tuesday, June 27, 2017 09:54:19 John Burton via Digitalmars-d-learn 
wrote:

 I'm coming from a C++ background so I'm not too used to garbage
 collection and it's implications. I have a function that creates
 a std.socket.Socket using new and connects to a tcp server, and
 writes some stuff to it. I then explicitly close the socket, and
 the socket object goes out of scope.

 I assume that I do need to explicitly call close on the socket as
 there is no deterministic destructor for class objects. I further
 assume that the runtime will garbage collect any memory allocated
 to the socket object at a later time.

 Am I doing this right with GC? In C++ I'd ensure that the Socket
 class had a destructor that closed the socket and I'd also assume
 that once it went out of scope there was no memory left
 allocated. In D am I right to assume I need to manually close the
 socket but there is no need to worry about the memory?

 Now the issue is that I now need to call this function more than
 once every second. I worry that it will create large amounts of
 uncollected "garbage" which will eventually lead to problems.

 Am I doing this right? Or is there a better way to do this in D?


Arguably, std.socket should have used structs instead of classes for sockets
for precisely this reason (though there are some advantages in using
inheritance with sockets). But yes, calling close manually is the correct
thing to do. Relying on the GC to call a destructor/finalizer is
error-prone. There is no guarantee that the memory will ever be freed (e.g.
the runtime could choose to not bother doing cleanup on shutdown), and even
if the GC does collect it, there are no guarantees about how soon it will do
so. However, if you keep allocating memory with the GC, then over time, the
GC will collect GC-allocated memory that isn't currently being used so that
it can reuse the memory. So, you really don't need to worry about the memory
unless it becomes a bottleneck. It will be collected and reused, not leaked.

And if in C++, you would be newing up a socket object each time and then
deleting it rather than having the socket on the stack, then that
performance could actually be worse than using the GC and having it collect
the memory when it determines that it needs more memory.

If you do find that it becomes a bottleneck to be allocating all of these
sockets, then you can look into using std.typecons.scoped, which would
allocate a class on the stack, but that really only works if you're just
dealing with a local variable, and in general, I wouldn't worry about
anything like that unless the heap allocations are proving to be a
performance problem - particularly because you need to be very careful when
using it to avoid screwing up and having memory corruption, because you used
an object after it was freed.

- Jonathan M Davis
Jun 27 2017
next sibling parent reply John Burton <john.burton jbmail.com> writes:
On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote:

 On Tuesday, June 27, 2017 09:54:19 John Burton via 
 Digitalmars-d-learn wrote:

 I'm coming from a C++ background so I'm not too used to 
 garbage collection and it's implications. I have a function 
 that creates a std.socket.Socket using new and connects to a 
 tcp server, and writes some stuff to it. I then explicitly 
 close the socket, and the socket object goes out of scope.




Thank you for the advice everyone.

The hardest part about learning D isn't the language, or how to 
program, it's unlearning what you know from C++ and learning the 
proper way to do things in D. I've tried D several times before 
and eventually stopped when I get to the stage of "how do I do 
this c++ thing in d" proves to be hard.

Instead this time, I've started writing D programs as "better C" 
and then slowly started adding in higher level d features. It's 
going much better as I'm no longer trying so hard to write bad 
C++ in D :)
Jun 27 2017
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 27 June 2017 at 11:43:27 UTC, John Burton wrote:

 On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis 
 wrote:

 On Tuesday, June 27, 2017 09:54:19 John Burton via 
 Digitalmars-d-learn wrote:

 I'm coming from a C++ background so I'm not too used to 
 garbage collection and it's implications. I have a function 
 that creates a std.socket.Socket using new and connects to a 
 tcp server, and writes some stuff to it. I then explicitly 
 close the socket, and the socket object goes out of scope.




 Thank you for the advice everyone.

 The hardest part about learning D isn't the language, or how to 
 program, it's unlearning what you know from C++ and learning 
 the proper way to do things in D. I've tried D several times 
 before and eventually stopped when I get to the stage of "how 
 do I do this c++ thing in d" proves to be hard.

 Instead this time, I've started writing D programs as "better 
 C" and then slowly started adding in higher level d features. 
 It's going much better as I'm no longer trying so hard to write 
 bad C++ in D :)


Heh, there's no reason you can't, you'll just cause a few head 
scratches when you post code while we figure out what it does. 
Idiomatic D is rather different to C++ and is centered around 
ranges + algorithms, but you can write code in pretty much any 
style. Some of the advanced metaprogramming takes a bit to get 
your head around.

If you do get stuck do post a question here, we're more than 
happy to help.
Jun 27 2017
prev sibling parent bauss <jj_1337 live.dk> writes:
On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote:

 On Tuesday, June 27, 2017 09:54:19 John Burton via 
 Digitalmars-d-learn wrote:

 [...]


 Arguably, std.socket should have used structs instead of 
 classes for sockets for precisely this reason (though there are 
 some advantages in using inheritance with sockets). But yes, 
 calling close manually is the correct thing to do. Relying on 
 the GC to call a destructor/finalizer is error-prone. There is 
 no guarantee that the memory will ever be freed (e.g. the 
 runtime could choose to not bother doing cleanup on shutdown), 
 and even if the GC does collect it, there are no guarantees 
 about how soon it will do so. However, if you keep allocating 
 memory with the GC, then over time, the GC will collect 
 GC-allocated memory that isn't currently being used so that it 
 can reuse the memory. So, you really don't need to worry about 
 the memory unless it becomes a bottleneck. It will be collected 
 and reused, not leaked.

 [...]


I agree with that it should have been structs. The inheritance 
issue could be fixed by having a private member of the struct in 
a class, that way there could still be a class wrapper around it.
Jun 27 2017
prev sibling next sibling parent Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:

 Now the issue is that I now need to call this function more 
 than once every second. I worry that it will create large 
 amounts of uncollected "garbage" which will eventually lead to 
 problems.


Since nobody has mentioned Allocator, yet:
As you seem to know the lifetime of the socket statically, you 
can just use std.experimental.allocator.{make,dispose} [1]. With 
regards to reusing the memory: Simply use a freelist allocator 
[2].

[1] https://dlang.org/phobos/std_experimental_allocator.html#.make
[2] 
https://dlang.org/phobos/std_experimental_allocator_building_blocks_free_list.html
Jun 27 2017
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:

 I assume that I do need to explicitly call close on the socket 
 as there is no deterministic destructor for class objects.


Yes.



 I further assume that the runtime will garbage collect any 
 memory allocated to the socket object at a later time.


Most probably.


 Am I doing this right with GC? In C++ I'd ensure that the 
 Socket class had a destructor that closed the socket and I'd 
 also assume that once it went out of scope there was no memory 
 left allocated. In D am I right to assume I need to manually 
 close the socket but there is no need to worry about the memory?


Yes.
You can also call a destructor manually with destroy(obj);
This avoids having a forest of 'close' methods, who are 
duplicates of the destructor in spirit, and let's you use 
destructors like you are accustomed in C++.

Generally, it is dangerous to let the GC handle resource release:
https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors


 Now the issue is that I now need to call this function more 
 than once every second. I worry that it will create large 
 amounts of uncollected "garbage" which will eventually lead to 
 problems.


If this is a problem you can create it on the stack with 
std.typecons.scoped



 Am I doing this right? Or is there a better way to do this in D?


What you are doing it OK.
Jun 27 2017
next sibling parent cym13 <cpicard openmailbox.org> writes:
On Tuesday, 27 June 2017 at 12:29:03 UTC, Guillaume Piolat wrote:

 [...]


Hmm... Isn't it possible to just allocate the object/a pool of 
objects outside the loop and reuse it? Everybody's proposing 
other means of allocations which is nice, but I wonder why there 
is such a need for reallocations in the first place.

Of course the specifics depend on the implementation, maybe it 
isn't as straightforward. It's just that no allocation is faster 
than reallocation no matter what reallocation method is used.
Jun 27 2017
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 8:29 AM, Guillaume Piolat wrote:

 On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:

 Am I doing this right with GC? In C++ I'd ensure that the Socket class 
 had a destructor that closed the socket and I'd also assume that once 
 it went out of scope there was no memory left allocated. In D am I 
 right to assume I need to manually close the socket but there is no 
 need to worry about the memory?


 
 Yes.
 You can also call a destructor manually with destroy(obj);
 This avoids having a forest of 'close' methods, who are duplicates of 
 the destructor in spirit, and let's you use destructors like you are 
 accustomed in C++.
 
 Generally, it is dangerous to let the GC handle resource release:
 https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors


This is the best to read, it's very tricky for people coming from other 
languages I think to understand the rules around the GC destructor.

I think you have gotten sound advice from many people, but I wanted to 
chime in as someone who used the GC to clean up sockets and file handles 
in a long-running program. Even with all these warnings, it does seem to 
work. My advice is to close the socket in the destructor if still valid, 
and then manually close it also. This way you don't leak resources if 
you miss a close call.

The fundamental problem with closing sockets and other resources with 
the GC is that the resource management issues and rules for memory are 
vastly different than those of other resources. The OS can give you vast 
sums of memory, but will likely limit your ability to keep sockets open. 
It also keeps the other end potentially locked up. If you try to 
allocate a new socket and run out of sockets, there is no "socket 
collection cycle". This is why closing manually is advised.

But I would use a close method, and not destroy(obj). The reason is 
because often times, you have wrapper types around your socket type, and 
just one extra level of indirection means the destructor cannot be used 
to clean up the socket (you are not allowed to access GC-allocated 
resources in a destructor).

Using std.typecons.RefCounted also can help to ensure destruction of a 
socket when it's no longer used. But like I said, it's not always 
possible to use a destructor of a higher level object to clean up a socket.

-Steve
Jun 27 2017
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer 
wrote:

 But I would use a close method, and not destroy(obj). The 
 reason is because often times, you have wrapper types around 
 your socket type, and just one extra level of indirection means 
 the destructor cannot be used to clean up the socket (you are 
 not allowed to access GC-allocated resources in a destructor).


All destructor restrictions do not apply when it's not called by 
the GC.

There really are two categories of destructors: called by the GC 
and called deterministically. Their usage should not overlap.




My reasoning went with the following:

1 - "I should have close() methods so that I don't rely on the 
GC, and the GC is calling ~this from the wrong thread etc, not 
everything can be released in this context (eg: OpenGL objects 
should be released from one thread only). Close methods will call 
close methods of "owned" objects."

2 - "Uh, oh. Refcounted and Unique and Scoped all use .destroy, 
so destructors should call close() methods. To avoid double 
release, close should handle being called several times. The GC 
will close what I forgot!"

3 - "Actually there is no order of destruction when called by the 
GC. So I can't release owned objects when called by the GC. 
Better do nothing when close() is called by the GC, let's detect 
this.

4 - close() is now identical with ~this (at least for classes). 
Let's remove the close() method. Crash when a resource is freed 
by the GC.

That's how the GC-proof resource class came to existence, after 
many destruction bugs, and it let's you use the GC as a detector 
for non-deterministic destruction. I miss it in  nogc :)

https://p0nce.github.io/d-idioms/#GC-proof-resource-class

Remember years ago when Alexandrescu suggested the GC shouldn't 
call heap destructors? That's what we get for having said no at 
the time.
Jun 27 2017
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 9:25 AM, Guillaume Piolat wrote:

 On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote:

 But I would use a close method, and not destroy(obj). The reason is 
 because often times, you have wrapper types around your socket type, 
 and just one extra level of indirection means the destructor cannot be 
 used to clean up the socket (you are not allowed to access 
 GC-allocated resources in a destructor).


 
 All destructor restrictions do not apply when it's not called by the GC.
 
 There really are two categories of destructors: called by the GC and 
 called deterministically. Their usage should not overlap.


Yes, Tango solved this by having a separate "finalize()" method. I wish 
we had something like this.


 
 My reasoning went with the following:
 
 1 - "I should have close() methods so that I don't rely on the GC, and 
 the GC is calling ~this from the wrong thread etc, not everything can be 
 released in this context (eg: OpenGL objects should be released from one 
 thread only). Close methods will call close methods of "owned" objects."


I hadn't realized this, that is a good point. However, not all resources 
are restricted this much.


 That's how the GC-proof resource class came to existence, after many 
 destruction bugs, and it let's you use the GC as a detector for 
 non-deterministic destruction. I miss it in  nogc :)
 
 https://p0nce.github.io/d-idioms/#GC-proof-resource-class


There are definitely better ways to do this than trying to allocate and 
catching an error. The GC knows the GC is running, use that mechanism 
instead :)


 Remember years ago when Alexandrescu suggested the GC shouldn't call 
 heap destructors? That's what we get for having said no at the time.


No, the issue is that there isn't a separate call for destruction and 
GC. I think we should add this.

The GC still needs to call something to clean up non-memory resources, 
but having a call to use when cleaning up deterministically can make 
resource management more efficient (even memory resource management).

-Steve

-Steve
Jun 27 2017
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 11:24 AM, Steven Schveighoffer wrote:

 On 6/27/17 9:25 AM, Guillaume Piolat wrote:



 That's how the GC-proof resource class came to existence, after many 
 destruction bugs, and it let's you use the GC as a detector for 
 non-deterministic destruction. I miss it in  nogc :)

 https://p0nce.github.io/d-idioms/#GC-proof-resource-class


 
 There are definitely better ways to do this than trying to allocate and 
 catching an error. The GC knows the GC is running, use that mechanism 
 instead :)


https://github.com/dlang/druntime/blob/master/src/gc/gcinterface.d#L189
https://github.com/dlang/druntime/blob/master/src/core/memory.d#L150

Apparently that info is limited to the core.memory package. But it's 
extern(C), so declaring the prototype should work. Should be much more 
efficient than allocating a byte (which I see you don't delete if it 
works), and catching an error.

-Steve
Jun 27 2017
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 12:16 PM, Steven Schveighoffer wrote:

 On 6/27/17 11:24 AM, Steven Schveighoffer wrote:

 On 6/27/17 9:25 AM, Guillaume Piolat wrote:


 

 That's how the GC-proof resource class came to existence, after many 
 destruction bugs, and it let's you use the GC as a detector for 
 non-deterministic destruction. I miss it in  nogc :)

 https://p0nce.github.io/d-idioms/#GC-proof-resource-class


 There are definitely better ways to do this than trying to allocate 
 and catching an error. The GC knows the GC is running, use that 
 mechanism instead :)


 
 https://github.com/dlang/druntime/blob/master/src/gc/gcinterface.d#L189
 https://github.com/dlang/druntime/blob/master/src/core/memory.d#L150
 
 Apparently that info is limited to the core.memory package. But it's 
 extern(C), so declaring the prototype should work. Should be much more 
 efficient than allocating a byte (which I see you don't delete if it 
 works), and catching an error.


Just added this enhancement to publicize this function: 
https://issues.dlang.org/show_bug.cgi?id=17563

-Steve
Jun 27 2017
prev sibling next sibling parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer 
wrote:

 On 6/27/17 9:25 AM, Guillaume Piolat wrote:

 On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer 
 wrote:

 But I would use a close method, and not destroy(obj). The 
 reason is because often times, you have wrapper types around 
 your socket type, and just one extra level of indirection 
 means the destructor cannot be used to clean up the socket 
 (you are not allowed to access GC-allocated resources in a 
 destructor).


 
 All destructor restrictions do not apply when it's not called 
 by the GC.
 
 There really are two categories of destructors: called by the 
 GC and called deterministically. Their usage should not 
 overlap.


 Yes, Tango solved this by having a separate "finalize()" 
 method. I wish we had something like this.


Well, technically speaking the `~this` for D classes *is* a 
finalizer that you may optionally manually call (e.g. via 
destroy).
It would be nice, though, to change class `~this` into a 
destructor and move the finalization into an extra method like 
`finalize`. Write a DIP?
Jun 27 2017
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 2:04 PM, Moritz Maxeiner wrote:

 On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer wrote:

 On 6/27/17 9:25 AM, Guillaume Piolat wrote:

 On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote:

 But I would use a close method, and not destroy(obj). The reason is 
 because often times, you have wrapper types around your socket type, 
 and just one extra level of indirection means the destructor cannot 
 be used to clean up the socket (you are not allowed to access 
 GC-allocated resources in a destructor).


 All destructor restrictions do not apply when it's not called by the GC.

 There really are two categories of destructors: called by the GC and 
 called deterministically. Their usage should not overlap.


 Yes, Tango solved this by having a separate "finalize()" method. I 
 wish we had something like this.


 
 Well, technically speaking the `~this` for D classes *is* a finalizer 
 that you may optionally manually call (e.g. via destroy).
 It would be nice, though, to change class `~this` into a destructor and 
 move the finalization into an extra method like `finalize`. Write a DIP?


I don't have enough motivation to do this. But if we do anything, we 
should look at what Tango has done. It may not work exactly for druntime 
because of existing code, but it obviously didn't need compiler changes 
to work.

-Steve
Jun 27 2017
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 18:04:36 UTC, Moritz Maxeiner wrote:

 Well, technically speaking the `~this` for D classes *is* a 
 finalizer that you may optionally manually call (e.g. via 
 destroy).
 It would be nice, though, to change class `~this` into a 
 destructor and move the finalization into an extra method like 
 `finalize`. Write a DIP?


The only sane way out is to prevent the GC from calling 
constructors.
Jun 27 2017
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 27 June 2017 at 23:42:38 UTC, Guillaume Piolat wrote:

 On Tuesday, 27 June 2017 at 18:04:36 UTC, Moritz Maxeiner wrote:

 Well, technically speaking the `~this` for D classes *is* a 
 finalizer that you may optionally manually call (e.g. via 
 destroy).
 It would be nice, though, to change class `~this` into a 
 destructor and move the finalization into an extra method like 
 `finalize`. Write a DIP?


 The only sane way out is to prevent the GC from calling 
 constructors.


Do you mean destructors?
If so, that's what I meant with the "technically". If the may GC 
call it (automatically), it's a finalizer, not a destructor (in 
the usual sense, anyway).
- Replace calls by the GC to `~this` with calls to `finalize` (or 
invent some cool other shortened name for the latter)
- Reserve `~this` for being called by deterministic lifetime 
management (std.experimental.allocator.dispose, object.destroy, 
RefCounted, Unique, etc.)
Jun 27 2017
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:

 Do you mean destructors?


Yes.



 - Replace calls by the GC to `~this` with calls to `finalize` 
 (or invent some cool other shortened name for the latter)


My point is that in such a "finalize()" function the only sane 
things to do is to crash if the resource wasn't freed already. 
Why so?

See my texture example in: 
https://forum.dlang.org/post/kliasvljzwjhzuzibuae forum.dlang.org

More simply:
If A needs an alive B to be destroyed, then neither A-owns-B or 
B-owns-A can guarantee the ordering under GC destruction.

Furthermore, this is viral. Whoever owns A needs an alive B to be 
destroyed.




 - Reserve `~this` for being called by deterministic lifetime 
 management (std.experimental.allocator.dispose, object.destroy, 
 RefCounted, Unique, etc.)


That's precisely what the GC-proof-resource-class allows, by 
preventing defective, non-composable usages of ~this.
Jun 27 2017
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 00:05:20 UTC, Guillaume Piolat 
wrote:

 On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:

 - Replace calls by the GC to `~this` with calls to `finalize` 
 (or invent some cool other shortened name for the latter)


 My point is that in such a "finalize()" function the only sane 
 things to do is to crash if the resource wasn't freed already. 
 Why so?

 [...]


Not every class can't be finalized, so it might make sense for 
finalization to remain an available option.
What I think reasonable is treating destruction and finalization 
as two distinct things. This could look like the following:

---
// Does not refer to other GC memory, can be finalized
class File
{
private:
     int fd;
public:
     this() { fd = open(...); }
     ~this() { close(fd); }

     // If the GC collects a File object that hasn't been 
destroyed yet, it will call this finalizer,
     // which *manually* calls the destructor (because it's safe 
to do so in this case)
     // A File object can still be manually `destroy`ed 
beforehand, in which case this particular finalizer is still 
safe, since an object won't be destroyed twice
     finalize() { destroy(this); }
     // Or maybe prettier: alias finalize = ~this;
}

class Foo
{
private:
     File f;
public:
     this() { f = new File(); }

     // Make the process crash on finalization of an undestroyed 
Foo object
      disable finalize();
}
---



 - Reserve `~this` for being called by deterministic lifetime 
 management (std.experimental.allocator.dispose, 
 object.destroy, RefCounted, Unique, etc.)


 That's precisely what the GC-proof-resource-class allows, by 
 preventing defective, non-composable usages of ~this.


The GC proof resource class idiom is a necessary hack, but it 
*is* a hack with its own limitations (e.g. depending on Error 
being catchable).
Jun 27 2017
parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Wednesday, June 28, 2017 01:11:35 Moritz Maxeiner via Digitalmars-d-learn 
wrote:

 On Wednesday, 28 June 2017 at 00:05:20 UTC, Guillaume Piolat

 wrote:

 On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:

 - Replace calls by the GC to `~this` with calls to `finalize`
 (or invent some cool other shortened name for the latter)


 My point is that in such a "finalize()" function the only sane
 things to do is to crash if the resource wasn't freed already.
 Why so?

 [...]


 Not every class can't be finalized, so it might make sense for
 finalization to remain an available option.


There are definitely cases where finalizers make sense. Case in point: if
you have a socket class, it makes perfect sense for it to have a finalizer.
Yes, it's better to close it manually, but it will work just fine for the GC
to close it when finalizing the class object so long as you don't use so
many sockets that you run out before the GC collects them. So, having a
finalizer is a good backup to ensure that the socket resource doesn't leak.

- Jonathan M Davis
Jun 27 2017
next sibling parent Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
wrote:

 On Wednesday, June 28, 2017 01:11:35 Moritz Maxeiner via 
 Digitalmars-d-learn wrote:

 Not every class can't be finalized, so it might make sense for 
 finalization to remain an available option.


 There are definitely cases where finalizers make sense. Case in 
 point: if you have a socket class, it makes perfect sense for 
 it to have a finalizer. Yes, it's better to close it manually, 
 but it will work just fine for the GC to close it when 
 finalizing the class object so long as you don't use so many 
 sockets that you run out before the GC collects them. So, 
 having a finalizer is a good backup to ensure that the socket 
 resource doesn't leak.


Yes, I think that's like the File class I presented (at least on 
Posix).
But on the other hand, since there are also cases in which 
finalization of an alive (==undestroyed) object are unacceptable 
(e.g. when it requires other GC managed objects to be alive), 
splitting them off into two separate methods and allowing to 
forbid finalization seems sensible to me.
Jun 27 2017
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
wrote:

 There are definitely cases where finalizers make sense. Case in 
 point: if you have a socket class, it makes perfect sense for 
 it to have a finalizer. Yes, it's better to close it manually, 
 but it will work just fine for the GC to close it when 
 finalizing the class object so long as you don't use so many 
 sockets that you run out before the GC collects them. So, 
 having a finalizer is a good backup to ensure that the socket 
 resource doesn't leak.

 - Jonathan M Davis


So far everyone is ignoring my example when A needs B to be 
destroyed. This happens as soon as you use DerelictUtil for 
example.
Jun 28 2017
next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:

 On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
 wrote:

 There are definitely cases where finalizers make sense. Case 
 in point: if you have a socket class, it makes perfect sense 
 for it to have a finalizer. Yes, it's better to close it 
 manually, but it will work just fine for the GC to close it 
 when finalizing the class object so long as you don't use so 
 many sockets that you run out before the GC collects them. So, 
 having a finalizer is a good backup to ensure that the socket 
 resource doesn't leak.

 - Jonathan M Davis


 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.


And I'm gonna rant a little bit more.

Deterministic destruction is a _solved_ problem in C++, and a 
number of users to convert are now coming from C++.

We should:
   1. be honest and tell things as they are: it's more complicated 
than in C++, but also liberating when you know to juggle between 
GC and deterministic
   2. Avoid making bogus suggestions which don't always work, such 
as close() methods, releasing resource with GC, . They only work 
for _some_ resources.
   3. Suggest a systematic, always working, workaround (or 
language change such as "GC doesn't call ~this). C++ users have 
no difficulty having an object graph with detailed ownership 
schemes, that's what they do day in day out.

It's important to have a _simple_ story about resource release, 
else we will talk about "GC" every day, and hearing about "GC" is 
bad for marketing.
Jun 28 2017
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 09:22:07 UTC, Guillaume Piolat 
wrote:

 Deterministic destruction is a _solved_ problem in C++, and a 
 number of users to convert are now coming from C++.


It is also in D, as long as you don't use the GC (which is 
inherently non-deterministic).


   3. Suggest a systematic, always working, workaround (or 
 language change such as "GC doesn't call ~this). C++ users have 
 no difficulty having an object graph with detailed ownership 
 schemes, that's what they do day in day out.

 It's important to have a _simple_ story about resource release, 
 else we will talk about "GC" every day, and hearing about "GC" 
 is bad for marketing.


Requirement: Do not allocate using the GC
Option 1) Use structs with ` disable this`, ` disable 
this(this)`, and a destructor that checks whether the resource 
reference is != invalid resource reference before trying to 
release.
Option 2) Use classes with simple constructor/destructor layout.
If you want to integrate a check that the requirement holds (with 
either class or struct), put `if (gc_inFinalizer) throw 
SomeError` into the class/struct destructor
Jun 28 2017
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 11:34:17 UTC, Moritz Maxeiner wrote:

 Requirement: Do not allocate using the GC
 Option 1) Use structs with ` disable this`, ` disable 
 this(this)`, and a destructor that checks whether the resource 
 reference is != invalid resource reference before trying to 
 release.
 Option 2) Use classes with simple constructor/destructor layout.
 If you want to integrate a check that the requirement holds 
 (with either class or struct), put `if (gc_inFinalizer) throw 
 SomeError` into the class/struct destructor


I don't get it.
It's completely possible to use the full power of GC and be 
deterministic.
I'm out of this very _confused_ discussion.
Jun 28 2017
parent Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 12:33:24 UTC, Guillaume Piolat 
wrote:

 On Wednesday, 28 June 2017 at 11:34:17 UTC, Moritz Maxeiner 
 wrote:

 Requirement: Do not allocate using the GC
 Option 1) Use structs with ` disable this`, ` disable 
 this(this)`, and a destructor that checks whether the resource 
 reference is != invalid resource reference before trying to 
 release.
 Option 2) Use classes with simple constructor/destructor 
 layout.
 If you want to integrate a check that the requirement holds 
 (with either class or struct), put `if (gc_inFinalizer) throw 
 SomeError` into the class/struct destructor


 I don't get it.


You asked for a "_simple_ story about resource release" and the 
above two options provide that IMHO.


 It's completely possible to use the full power of GC and be 
 deterministic.


Sure, but it's not "simple". With D's GC as the memory allocator 
you can currently have
(1) non-deterministic memory management and non-deterministic 
object lifetimes:
     Use `new` or 
`std.experimental.allocator.make!(std.experimental.allocator.gc_all
cator.GCAllocator)` for allocation&construction.
     The GC will finalize objects with no pointers to them 
non-deterministically and deallocate the memory after their 
respective finalization.
(2) non-deterministic memory management and deterministic object 
lifetimes:
     Construct the objects as with (1), but destruct them by 
calling `destroy` on them outside of a collection cycle (i.e. 
when they, and all their members are still considered "live" by 
the GC).
     The GC will collect the memory non-deterministically after 
there are no more pointers to it.
     Warning: You are responsible for ensuring that the GC will 
never see an undestroyed object in the collection cycle, because 
it might try to finalize (call its `~this`) it.
              You can protect yourself against such finalization 
attempts by putting `if (gc_inFinalizer) throw Error("Bug! I 
forgot to destruct!")` in the objects' respective destructors.
(3) deterministic memory management and deterministic object 
lifetimes:
     Construct the objects as with (1), but destruct&deallocate 
them by calling 
`std.experimental.allocator.dispose!(std.experimental.allocator.gc_alloc
tor.GCAllocator)`on them outside of a collection cycle.
     The GC will not do anything for those objects.
     Same warning as (2) applies.


 I'm out of this very _confused_ discussion.


I'm honestly not sure what's confusing about it.
Jun 28 2017
prev sibling next sibling parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:

 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.


I thought I had (implicitly): B needs to be ` disable finalize`.
Jun 28 2017
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 11:21:07 UTC, Moritz Maxeiner wrote:

 On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
 wrote:

 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.


 I thought I had (implicitly): B needs to be ` disable finalize`.


So in the current language, doesn't exist?
Jun 28 2017
parent Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 12:28:28 UTC, Guillaume Piolat 
wrote:

 On Wednesday, 28 June 2017 at 11:21:07 UTC, Moritz Maxeiner 
 wrote:

 On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
 wrote:

 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.


 I thought I had (implicitly): B needs to be ` disable 
 finalize`.


 So in the current language, doesn't exist?


I thought that was the premise of the discussion?
That the GC *currently* invokes the "destructors" as finalizers 
and that *currently* the only way to avoid that is calling 
`destroy` on them manually beforehand (since destructors won't be 
called twice on a single object).
Jun 28 2017
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:


 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.


What's the issue with DerelictUtil?
Jun 28 2017
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 13:02:04 UTC, Mike Parker wrote:

 On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
 wrote:


 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.


 What's the issue with DerelictUtil?


https://forum.dlang.org/post/pmulowxpikjjffkrscct forum.dlang.org

Not an issue with DerelictUtil, an issue with the strategy of 
closing resources in GC with this case.

Derelict loaders work-around this by not unloading shared 
libraries so the GC won't unload shared libs before the resources 
related to the shared library are freed.

https://github.com/DerelictOrg/DerelictAL/blob/master/source/derelict/openal/dynload.d#L366
Jun 28 2017
parent Mike Parker <aldacron gmail.com> writes:
On Wednesday, 28 June 2017 at 13:19:37 UTC, Guillaume Piolat 
wrote:


 https://forum.dlang.org/post/pmulowxpikjjffkrscct forum.dlang.org

 Not an issue with DerelictUtil, an issue with the strategy of 
 closing resources in GC with this case.

 Derelict loaders work-around this by not unloading shared 
 libraries so the GC won't unload shared libs before the 
 resources related to the shared library are freed.

 https://github.com/DerelictOrg/DerelictAL/blob/master/source/derelict/openal/dynload.d#L366


Yeah, the loaders all used to needlessly unload the shared 
libraries in a static destructor. The fact that they don't 
anymore isn't to avoid any GC/destructor issues, but because 
there's no point in unloading the libraries when the system is 
going to do it anyway.
Jun 28 2017
prev sibling next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer 
wrote:

 The GC still needs to call something to clean up non-memory 
 resources,


Well, I'd much prefer if the GC would simply not call 
destructors. Yes, destructor can work for _some_ resources, but 
because of ordering it also create accidental correctness, or 
just doesn't work with other resources.


 but having a call to use when cleaning up deterministically can 
 make resource management more efficient (even memory resource 
 management).


Yes. My point is that the destructor is uniquely positionned to 
be that call.
(This would also enables back failing destructors through the use 
of exceptions.)

In current D the problem with allowing the GC to close resources 
is that there is no ordering of destructors. A sub-tree of the 
ownership graph is freed by the GC together, but sometime an 
order is required for releasing.

Typically: you want to release a SDL_Texture but the Derelict 
SharedLib object has been freed already, so you can't call 
sdl_releaseTexture. It doesn't matter that the texture object 
held a reference to the ShareLib object since the sub-tree is 
destroyed together. close() methods don't help with that, the 
problem is with the GC calling destructors.


And if _some_ resource needs an ordering, then this property leak 
on their owners, so little by little the whole ownership graph 
need determinism.
Jun 27 2017
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-06-27 17:24, Steven Schveighoffer wrote:


 Yes, Tango solved this by having a separate "finalize()" method. I wish
 we had something like this.


Not sure if this is the same, but I remember that Tango had a separate 
method called "dispose" that was called if a class was allocated on the 
stack, i.e. with the "scope" keyword.

-- 
/Jacob Carlborg
Jun 28 2017
prev sibling next sibling parent Dsby <dushibaiyu yahoo.com> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:

 I'm coming from a C++ background so I'm not too used to garbage 
 collection and it's implications. I have a function that 
 creates a std.socket.Socket using new and connects to a tcp 
 server, and writes some stuff to it. I then explicitly close 
 the socket, and the socket object goes out of scope.

 [...]


May be you can see yu: https://github.com/dushibaiyu/yu
Jun 28 2017
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-06-27 11:54, John Burton wrote:

 I'm coming from a C++ background so I'm not too used to garbage
 collection and it's implications. I have a function that creates a
 std.socket.Socket using new and connects to a tcp server, and writes
 some stuff to it. I then explicitly close the socket, and the socket
 object goes out of scope.


Yes. You can use "scope (exit)", unless you're already doing so.


 Now the issue is that I now need to call this function more than once
 every second. I worry that it will create large amounts of uncollected
 "garbage" which will eventually lead to problems.


Sounds like you need a connection pool. The vibe.d [1] framework 
contains connection pools.

It's also possible to manually disable the GC for a while and then 
enable it again.

[1] http://vibed.org

-- 
/Jacob Carlborg
Jun 28 2017
prev sibling parent reply John Burton <john.burton jbmail.com> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:

 I'm coming from a C++ background so I'm not too used to garbage 
 collection and it's implications. I have a function that 
 creates a std.socket.Socket using new and connects to a tcp 
 server, and writes some stuff to it. I then explicitly close 
 the socket, and the socket object goes out of scope.



 Am I doing this right? Or is there a better way to do this in D?

 Thanks.



For my use case here, I'm increasingly thinking that just calling 
the underlying 'C' socket and send calls is better. No need for 
anything complicated at all for my actual program :)
Jun 28 2017
parent Meta <jared771 gmail.com> writes:
On Wednesday, 28 June 2017 at 15:55:41 UTC, John Burton wrote:

 On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:

 I'm coming from a C++ background so I'm not too used to 
 garbage collection and it's implications. I have a function 
 that creates a std.socket.Socket using new and connects to a 
 tcp server, and writes some stuff to it. I then explicitly 
 close the socket, and the socket object goes out of scope.



 Am I doing this right? Or is there a better way to do this in 
 D?

 Thanks.



 For my use case here, I'm increasingly thinking that just 
 calling the underlying 'C' socket and send calls is better. No 
 need for anything complicated at all for my actual program :)


One final piece of advice as this thread seemed to have gone off 
the rails a bit. You can always put a `scope(exit) 
socket.close();` after you create the socket. This will ensure 
that the socket will be closed once the scope is exited no matter 
what... almost, anyway. If an Error is thrown no stack unwinding 
is done but at this point your program is in an unrecoverable 
state anyway and you have a lot more to worry about than a socket 
that hasn't been closed.
Jun 29 2017