www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How to return GC-d memory back to OS

reply Serg Kovrov <kovrov no.spam> writes:
Is it possible to make GC to free memory back to OS?

I'm asking because I'm writing a news feed notifier/reader, and my goal 
for the application is to be as much memory efficient as possible, as I 
intent to run it on my desktop 24/7.

In other words, I want my application to free any resources back to OS 
as soon as they no longer used. eg., it fetch a big document, process it 
store for later usage on disk cache, set a flag that new document 
available - then free all resources used for that task back to OS. 
Whatever peak memory usage will be, I want my application in idle state 
to have as small memory footprint as possible.

The 'high watermark' (if thats the name) approach in GC is not way to go 
in my case =(

There is not much info on GC on digitalmars site, so maybe someone know 
a workaround for my situation?

Thanks
--
serg.
Aug 03 2006
parent reply Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Serg Kovrov wrote:

 Is it possible to make GC to free memory back to OS?
 
 I'm asking because I'm writing a news feed notifier/reader, and my goal 
 for the application is to be as much memory efficient as possible, as I 
 intent to run it on my desktop 24/7.
 
 In other words, I want my application to free any resources back to OS 
 as soon as they no longer used. eg., it fetch a big document, process it 
 store for later usage on disk cache, set a flag that new document 
 available - then free all resources used for that task back to OS. 
 Whatever peak memory usage will be, I want my application in idle state 
 to have as small memory footprint as possible.
 
 The 'high watermark' (if thats the name) approach in GC is not way to go 
 in my case =(
 
 There is not much info on GC on digitalmars site, so maybe someone know 
 a workaround for my situation?


For a long time, even linux malloc()/free() would not return memory to 
the OS. Usually, memory is allocated from the OS using brk() or similar 
system calls and an application can only return memory from the end of 
the virtual address range the application has been allocated. In 
general, this requires a relocating/compacting GC, as even a single 
allocated byte at the top end of your virtual address space will prevent 
the OS from shrinking it.

But apart from needing much swap space, there are no draw backs of using 
much virtual memory. The memory you don't use (but exists in your 
virtual address range) will not map to physical memory and will not 
affect your computers performance.

/Oskar
Aug 03 2006
next sibling parent Serg Kovrov <kovrov no.spam> writes:
* Oskar Linde:

 For a long time, even linux malloc()/free() would not return memory to 
 the OS. Usually, memory is allocated from the OS using brk() or similar 
 system calls and an application can only return memory from the end of 
 the virtual address range the application has been allocated. In 
 general, this requires a relocating/compacting GC, as even a single 
 allocated byte at the top end of your virtual address space will prevent 
 the OS from shrinking it.
 
 But apart from needing much swap space, there are no draw backs of using 
 much virtual memory. The memory you don't use (but exists in your 
 virtual address range) will not map to physical memory and will not 
 affect your computers performance.
 
 /Oskar


I'm not sure, Oskar, as I am not an expert in that matter...

But... I'm not tried on linux but on Windows (my target OS is 2000/XP 
family so far) 'delete' do exactly what I want.

For example, in C++ (GCC/mingw) I could allocate with 'new' 50Mb, and 
see in task manager or performance monitor that my "private bytes" are 
increased about that size. When I 'delete' it, I see that "private 
bytes" count is dropped to the same level it was before. Nice.

As I understand "private bytes" are amount of memory my process uses 
exclusively and can not be used by anyone else. And that I define for 
myself as memory footprint - correct me if I wrong.

Same example in D show that GC greedily hold everything it get from OS. 
  And that is definitely what I do not want.
Aug 03 2006
prev sibling next sibling parent reply Karen Lanrap <karen digitaldaemon.com> writes:
Oskar Linde wrote:

 But apart from needing much swap space, there are no draw backs
 of using much virtual memory. The memory you don't use (but
 exists in your virtual address range) will not map to physical
 memory and will not affect your computers performance.


That is not true in general:
| "For the attack to succeed, one needs to find a reliable way to
| force interesting kernel code to be paged out, then find that
| code inside a page file and modify it. And finally, the kernel
| needs to load that code (now modified) again into physical
| memory and execute it,"

http://www.darkreading.com/document.asp?doc_id=99780

---and for the current GC of D it is especially not true, because the 
sweeps of the GC do use the memory.
Aug 04 2006
next sibling parent reply Oskar Linde <oskar.lindeREM OVEgmail.com> writes:
Karen Lanrap wrote:

 Oskar Linde wrote:

 But apart from needing much swap space, there are no draw backs
 of using much virtual memory. The memory you don't use (but
 exists in your virtual address range) will not map to physical
 memory and will not affect your computers performance.


 
 That is not true in general:
 | "For the attack to succeed, one needs to find a reliable way to
 | force interesting kernel code to be paged out, then find that
 | code inside a page file and modify it. And finally, the kernel
 | needs to load that code (now modified) again into physical
 | memory and execute it,"
 
 http://www.darkreading.com/document.asp?doc_id=99780
 
 ---and for the current GC of D it is especially not true, because the 
 sweeps of the GC do use the memory.


My point was that if the memory was collected by the GC, but not 
returned to the OS, the GC would not sweep it, and it would remain 
swapped out.

/Oskar
Aug 04 2006
parent Karen Lanrap <karen digitaldaemon.com> writes:
Oskar Linde wrote:


 My point was that if the memory was collected by the GC, but not
 returned to the OS, the GC would not sweep it, and it would
 remain swapped out.


I agree.
Aug 04 2006
prev sibling parent Sean Kelly <sean f4.ca> writes:
Karen Lanrap wrote:

 Oskar Linde wrote:

 But apart from needing much swap space, there are no draw backs
 of using much virtual memory. The memory you don't use (but
 exists in your virtual address range) will not map to physical
 memory and will not affect your computers performance.


 
 That is not true in general:
 | "For the attack to succeed, one needs to find a reliable way to
 | force interesting kernel code to be paged out, then find that
 | code inside a page file and modify it. And finally, the kernel
 | needs to load that code (now modified) again into physical
 | memory and execute it,"


Wouldn't this also require the attacker to have root access to modify 
the page file?  Or are these always written with the same permissions as 
the program is executing in?


Sean
Aug 04 2006
prev sibling parent Sean Kelly <sean f4.ca> writes:
Oskar Linde wrote:

 Serg Kovrov wrote:

 Is it possible to make GC to free memory back to OS?

 I'm asking because I'm writing a news feed notifier/reader, and my 
 goal for the application is to be as much memory efficient as 
 possible, as I intent to run it on my desktop 24/7.

 In other words, I want my application to free any resources back to OS 
 as soon as they no longer used. eg., it fetch a big document, process 
 it store for later usage on disk cache, set a flag that new document 
 available - then free all resources used for that task back to OS. 
 Whatever peak memory usage will be, I want my application in idle 
 state to have as small memory footprint as possible.

 The 'high watermark' (if thats the name) approach in GC is not way to 
 go in my case =(

 There is not much info on GC on digitalmars site, so maybe someone 
 know a workaround for my situation?


 
 For a long time, even linux malloc()/free() would not return memory to 
 the OS. Usually, memory is allocated from the OS using brk() or similar 
 system calls and an application can only return memory from the end of 
 the virtual address range the application has been allocated. In 
 general, this requires a relocating/compacting GC, as even a single 
 allocated byte at the top end of your virtual address space will prevent 
 the OS from shrinking it.
 
 But apart from needing much swap space, there are no draw backs of using 
 much virtual memory. The memory you don't use (but exists in your 
 virtual address range) will not map to physical memory and will not 
 affect your computers performance.


Just to clarify, the pages that are not in use will not map to physical 
memory.  This is another advantage of compacting collectors--they can 
empty out near-empty pages to avoid unnecessary swapping.


Sean
Aug 04 2006