www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D wrapper classes leak memory. Next steps?

reply Marco Leise <Marco.Leise gmx.de> writes:
GtkD wraps all the objects in Gtk. In callbacks like "onDraw",
when they are called often, this creates heaps of tiny wrapper
objects around huge data from C libraries. Eventually system
memory is exhausted and the computer slows down and eats into
swap. So should we add a "run garbage collector" button to
every application to clean those up? Seems rather silly.

The GC should stick to its own memory and not try to manage
resources it doesn't understand. Reference counting applies to
external and system resources much better. To date neither the
language nor Phobos can ref count classes. And even with
library support it is not likely that it will get used widely,
because plain classes are so much easier (also on the eyes).

The extent of this is pretty huge. There are many D wrapper
libraries that build on classes and mimic what is available
for C++ or Python. Issues like this make D appear like a toy
language. How would one write a big desktop application when
a simple animation can exhaust system memory in seconds ?

-- 
Marco
Oct 30 2014
next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 05:38:48 +0100
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 GtkD wraps all the objects in Gtk. In callbacks like "onDraw",
 when they are called often, this creates heaps of tiny wrapper
 objects around huge data from C libraries. Eventually system
 memory is exhausted and the computer slows down and eats into
 swap. So should we add a "run garbage collector" button to
 every application to clean those up? Seems rather silly.
=20
 The GC should stick to its own memory and not try to manage
 resources it doesn't understand. Reference counting applies to
 external and system resources much better. To date neither the
 language nor Phobos can ref count classes. And even with
 library support it is not likely that it will get used widely,
 because plain classes are so much easier (also on the eyes).
=20
 The extent of this is pretty huge. There are many D wrapper
 libraries that build on classes and mimic what is available
 for C++ or Python. Issues like this make D appear like a toy
 language. How would one write a big desktop application when
 a simple animation can exhaust system memory in seconds ?


why do you blaming GC for what seems to be a library author's fault?
that was the library author who chooses to wrap externally-managed
objects in GC objects, and creating alot of throwaway classes for that.

alas, we haven't Skynet at the hand to guess what author really wants.

p.s. i was never using GtkD, so i don't know if it's really doing the
things you wrote or not, i'm just assuming that you are right.
Oct 30 2014
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 31 Oct 2014 06:44:48 +0200
schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:


 On Fri, 31 Oct 2014 05:38:48 +0100
 Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 

 GtkD wraps all the objects in Gtk. In callbacks like "onDraw",
 when they are called often, this creates heaps of tiny wrapper
 objects around huge data from C libraries. Eventually system
 memory is exhausted and the computer slows down and eats into
 swap. So should we add a "run garbage collector" button to
 every application to clean those up? Seems rather silly.
 
 The GC should stick to its own memory and not try to manage
 resources it doesn't understand. Reference counting applies to
 external and system resources much better. To date neither the
 language nor Phobos can ref count classes. And even with
 library support it is not likely that it will get used widely,
 because plain classes are so much easier (also on the eyes).
 
 The extent of this is pretty huge. There are many D wrapper
 libraries that build on classes and mimic what is available
 for C++ or Python. Issues like this make D appear like a toy
 language. How would one write a big desktop application when
 a simple animation can exhaust system memory in seconds ?


 why do you blaming GC for what seems to be a library author's fault?
 that was the library author who chooses to wrap externally-managed
 objects in GC objects, and creating alot of throwaway classes for that.


Can't blame the author when there is no other choice in the
language than to use tracing GC with inheritance.


 alas, we haven't Skynet at the hand to guess what author really wants.
 
 p.s. i was never using GtkD, so i don't know if it's really doing the
 things you wrote or not, i'm just assuming that you are right.


To be fair there is an overload for addOnDraw that takes a
delegate that has a Scoped!Context as parameter, but I cannot
use that with delegates marked as nothrow since Scoped's dtor
isn't nothrow. Which prompts the question again how to deal
with exceptions in dtors. All D delegates called from the C
side have to be nothrow though, because neither can druntime
deal with GCC's amd64 stack (omit-frame-pointer) nor is the C
side necessarily prepared for exceptions.

-- 
Marco
Oct 31 2014
parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 09:46:54 +0100
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 Can't blame the author when there is no other choice in the
 language than to use tracing GC with inheritance.


there are alot of choices. i'm succesfully using wrapper classes with
reference counting in my i/o streams library, for example. they are GC
roots, and they are freed in the same moment when their reference count
becomes zero.


 Which prompts the question again how to deal
 with exceptions in dtors.


class dtors should not throw any exceptions. stack-allocated struct
dtors can do what they want, just catch that and you'll got "nothrow"
function. what's the problem here?
Oct 31 2014
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 31 Oct 2014 10:51:04 +0200
schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:


 On Fri, 31 Oct 2014 09:46:54 +0100
 Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 

 Can't blame the author when there is no other choice in the
 language than to use tracing GC with inheritance.


 there are alot of choices. i'm succesfully using wrapper classes with
 reference counting in my i/o streams library, for example. they are GC
 roots, and they are freed in the same moment when their reference count
 becomes zero.


I meant obvious, clean, idiomatic choices. It's clear that D's
expressiveness makes pretty much anything work if you put a
week's worth of time into it. What I'm saying is, the language
should offer something to use external resources in class
hierarchies, because it is a common need. Something that comes
with so little friction that you don't think twice.


 Which prompts the question again how to deal
 with exceptions in dtors.


 class dtors should not throw any exceptions. stack-allocated struct
 dtors can do what they want, just catch that and you'll got "nothrow"
 function. what's the problem here?


E-he, stack sounds nice, but those are in D's "parameter
scope". You cannot wrap their destruction in try-catch.

-- 
Marco
Oct 31 2014
parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 10:53:27 +0100
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 Am Fri, 31 Oct 2014 10:51:04 +0200
 schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:
=20

 On Fri, 31 Oct 2014 09:46:54 +0100
 Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:
=20

 Can't blame the author when there is no other choice in the
 language than to use tracing GC with inheritance.


 there are alot of choices. i'm succesfully using wrapper classes with
 reference counting in my i/o streams library, for example. they are GC
 roots, and they are freed in the same moment when their reference count
 becomes zero.


=20
 I meant obvious, clean, idiomatic choices. It's clear that D's
 expressiveness makes pretty much anything work if you put a
 week's worth of time into it. What I'm saying is, the language
 should offer something to use external resources in class
 hierarchies, because it is a common need. Something that comes
 with so little friction that you don't think twice.


if you have something concrete in mind, write ER or forum post, so we
can destroy it. ;-)


 Which prompts the question again how to deal
 with exceptions in dtors.


 class dtors should not throw any exceptions. stack-allocated struct
 dtors can do what they want, just catch that and you'll got "nothrow"
 function. what's the problem here?


=20
 E-he, stack sounds nice, but those are in D's "parameter
 scope". You cannot wrap their destruction in try-catch.


there is no problem that can't be solved by introducing one more
wrapper. ;-)
Oct 31 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 09:58:41 UTC, ketmar via 
Digitalmars-d wrote:

 if you have something concrete in mind, write ER or forum post, 
 so we can destroy it. ;-)


Sure!

- dump the gc

- make uniqe_ptr a language builtin

- ditch c malloc

- make compiler 100% memory management aware

- make thread aware high speed class allocators a builtin

- ditch emplace

- default pointers to non-nullable

- do a complete redesign of phobos

I'm sure there is more…
Oct 31 2014
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 10:06:50 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On Friday, 31 October 2014 at 09:58:41 UTC, ketmar via=20
 Digitalmars-d wrote:

 if you have something concrete in mind, write ER or forum post,=20
 so we can destroy it. ;-)


=20
 Sure!
=20
 - dump the gc
=20
 - make uniqe_ptr a language builtin
=20
 - ditch c malloc
=20
 - make compiler 100% memory management aware
=20
 - make thread aware high speed class allocators a builtin
=20
 - ditch emplace
=20
 - default pointers to non-nullable
=20
 - do a complete redesign of phobos
=20
 I'm sure there is more=E2=80=A6


send your patches! ;-)
Oct 31 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 10:06:50 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:

p.s. and title that "Z", as "the last language you'll need ever. in all
meanings." ;-)
Oct 31 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 10:21:47 UTC, ketmar via 
Digitalmars-d wrote:

 On Fri, 31 Oct 2014 10:06:50 +0000
 via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 p.s. and title that "Z", as "the last language you'll need 
 ever. in all meanings." ;-)


Indeed, for the english speaking world, but as a patriotic 
scandinavian I'm inclined to think there are three more letters. 
And I cannot have that, can I?

In Denmark and Norway the alphabet ends with : Z, Æ, Ø, Å…
And in Sweden: Z, Å, Ä, Ö…

So the name of the ultimate language has to be one of:

ZÅÖ
ZÖÅ
ÅZÖ
ÅÖZ
ÖZÅ
or
ÖÅZ

Hm…
Oct 31 2014
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 12:09:29 UTC, Ola Fosheim Grøstad 
wrote:

 ZÅÖ
 ZÖÅ
 ÅZÖ
 ÅÖZ
 ÖZÅ
 or
 ÖÅZ

 Hm…


On second thought I think I should stick to the Norwegian 
alphabet, but resort to old spelling rules. That will make the 
name of the ultimate language: "AA", pronounced "Awe":

http://www.youtube.com/watch?v=BX5W8gXcAW4

And, as a bonus it will be listed first in programming language 
listings, such as Rosetta Code.

OK. That's it. I am from now on working on a new system level 
programming language called "AA", pronounced "Awe" which should 
be sorted first in all english listings, and sorted last in all 
scandinavian listings.

No more patches. I'm speccing!
Oct 31 2014
prev sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 12:09:27 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On Friday, 31 October 2014 at 10:21:47 UTC, ketmar via=20
 Digitalmars-d wrote:

 On Fri, 31 Oct 2014 10:06:50 +0000
 via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 p.s. and title that "Z", as "the last language you'll need=20
 ever. in all meanings." ;-)


=20
 Indeed, for the english speaking world, but as a patriotic=20
 scandinavian I'm inclined to think there are three more letters.=20
 And I cannot have that, can I?
=20
 In Denmark and Norway the alphabet ends with : Z, =C3=86, =C3=98, =C3=85=


=E2=80=A6

 And in Sweden: Z, =C3=85, =C3=84, =C3=96=E2=80=A6
=20
 So the name of the ultimate language has to be one of:
=20
 Z=C3=85=C3=96
 Z=C3=96=C3=85
 =C3=85Z=C3=96
 =C3=85=C3=96Z
 =C3=96Z=C3=85
 or
 =C3=96=C3=85Z
=20
 Hm=E2=80=A6


i bet you can find a nicely looking hieroglyph too. that would be cool
for graffiti.
Oct 31 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 12:22:56 UTC, ketmar via 
Digitalmars-d wrote:

 i bet you can find a nicely looking hieroglyph too. that would 
 be cool for graffiti.


Good point. So the logo would be "Å", it is spelled "AA", 
pronounced "Awe" and sorted both first and last in listings.

It has to be a very minimalistic language:

http://en.wikipedia.org/wiki/Angstrom
Oct 31 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 12:31:25 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 31 October 2014 at 12:22:56 UTC, ketmar via 
 Digitalmars-d wrote:

 i bet you can find a nicely looking hieroglyph too. that would 
 be cool for graffiti.


 Good point. So the logo would be "Å", it is spelled "AA", 
 pronounced "Awe" and sorted both first and last in listings.

 It has to be a very minimalistic language:

 http://en.wikipedia.org/wiki/Angstrom


Oh drat, someone is squatting my would-be website…

http://www.aa.org/
Oct 31 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 12:56:58 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 Oh drat, someone is squatting my would-be website=E2=80=A6
 http://www.aa.org/


ah. just wait until they got drunk and buy it in exchange for one more
shot.
Oct 31 2014
prev sibling parent reply "eles" <eles eles.com> writes:
On Friday, 31 October 2014 at 10:06:51 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 31 October 2014 at 09:58:41 UTC, ketmar via 
 Digitalmars-d wrote:

 if you have something concrete in mind, write ER or forum 
 post, so we can destroy it. ;-)





 - make uniqe_ptr a language builtin


I would go with this by default for all pointers/references.

Something that I read here:

https://news.ycombinator.com/item?id=7650917

My feeling too.

If you need more owners for a resource, then explicitly ask for.
Oct 31 2014
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 13:16:17 UTC, eles wrote:

 Something that I read here:

 https://news.ycombinator.com/item?id=7650917

 My feeling too.

 If you need more owners for a resource, then explicitly ask for.


Thank you for sharing, a very interesting discussion topic.

I am increasingly thinking that it might be possible to have a 
system level language where you can do progressive refinement 
from a garbage collection supported sketch to a hardcore real 
time app.

Basically have different compilation modes where you start out in 
executable pseudo-code and gradually turn on more switches which 
give you performance/ownership etc warnings (integrated in the 
IDE).

Kinda a like starting out with Python and ending up with C.

D supports this a little bit, but has not been designed for it (I 
think).
Oct 31 2014
parent reply "eles" <eles215 gzk.dot> writes:
On Friday, 31 October 2014 at 13:42:53 UTC, Ola Fosheim Grøstad 
wrote:

 On Friday, 31 October 2014 at 13:16:17 UTC, eles wrote:

 Something that I read here:

 https://news.ycombinator.com/item?id=7650917

 My feeling too.

 If you need more owners for a resource, then explicitly ask 
 for.


 Thank you for sharing, a very interesting discussion topic.


Just skimming over this:

http://dlang.org/const-faq.html#const-parameters

it would seem to make sense to make them all const by default, 
and one would have to specifically mark as mutable those that 
would be changed. The problems with this are:

     It would be a huge break from past D practice, and practice 


-- Actually, this is the sole reason, but the breakage would be 
far less that the breakage introduced by Rust & co. The two 
reasons that follow are almost hypocrisy to justify the fear in 
point 1.


     It would require a new keyword, say mutable.

-- Big deal. C++ has mutable too.

     And worst, it would make declarations inconsistent:

     void foo(int* p)
     {
         int* q;
         ...
     }

     p points to const, and q points to mutable. This kind of 
inconsistency leads to all sorts of mistakes. It also makes it 
very hard to write generic code that deals with types.

-- Why not make all pointer declarations as constant by default? 
Then, you explicitely ask for a mutable reference (hell, you 
could even limit the number of mutable references that could be 
asked for, if you want). Just reformulate to say:

"p points to const, and q points to const. This kind of 
consistency leads to all kind of great code. It also makes it 
very easy to write generic code that deals with types."
Oct 31 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 21:43:26 UTC, eles wrote:

 it would seem to make sense to make them all const by default, 
 and one would have to specifically mark as mutable those that 
 would be changed.


Yes, I think that would be interesting. I also thought about the 
possibility of putting all mutable parameters to the left of a 
dot-operator, but it takes some time getting used to (not sure if 
I could get used to it):

(arr1,arr2).swap_range(10,20);

arr1.merge(arr2); //merge by copy
(arr1,arr2).merge; // merge by moving


 -- Why not make all pointer declarations as constant by 
 default? Then, you explicitely ask for a mutable reference 
 (hell, you could even limit the number of mutable references 
 that could be asked for, if you want). Just reformulate to say:


I think immutable/const by default is the best option. It is 
primarily in loops that mutable is needed and since mutables are 
more likely to cause bugs it would be nice to have some attention 
drawn to them.
Oct 31 2014
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 31 Oct 2014 13:16:15 +0000
schrieb "eles" <eles eles.com>:


 On Friday, 31 October 2014 at 10:06:51 UTC, Ola Fosheim Gr=C3=B8stad=20
 wrote:

 On Friday, 31 October 2014 at 09:58:41 UTC, ketmar via=20
 Digitalmars-d wrote:

 if you have something concrete in mind, write ER or forum=20
 post, so we can destroy it. ;-)




=20

 - make uniqe_ptr a language builtin


=20
 I would go with this by default for all pointers/references.
=20
 Something that I read here:
=20
 https://news.ycombinator.com/item?id=3D7650917
=20
 My feeling too.
=20
 If you need more owners for a resource, then explicitly ask for.


In case of a onDraw callback the object will be passed as a
parameter into the drawing function. unique_ptr doesn't solve
this. In the particular case of callbacks from C libraries
that give us opaque "context" pointers we want to wrap with D
classes we can use either of:

- reference counted object (doesn't exist in D)
- unique_ptr (requires "scope" to be implemented)
- Scoped ?

Maybe Scoped wasn't explored enough for callbacks in GtkD.
But yeah, you'd still need one of the others to deal with
external resources where the lifetime is not a specific stack
frame.

--=20
Marco
Nov 01 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, 1 Nov 2014 19:10:15 +0100
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 - reference counted object (doesn't exist in D)


but it exists! and with a size of machine pointer! i swear it exists,
i'm using it!
Nov 01 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
On Friday, 31 October 2014 at 04:28:47 UTC, Marco Leise wrote:

 How would one write a big desktop application when
 a simple animation can exhaust system memory in seconds ?


Yep, it can be a long way from PoC to a quality implementation. 
Write GtkD 2.0 with careful approach w.r.t. resource management. 
Though, reference counting may not be a silver bullet, it still 
requires you to not forget to clear unneeded resources. And it's 
of no use if the reference is in an GC-managed object. Also you 
can still create big resources too eagerly instead of on demand, 
and hold onto them for longer than needed.
Oct 31 2014