• Leon (9/9) Aug 27 2011 I'm writing a simple program which uses merge sort to sort very large te...
• =?UTF-8?B?Q2hyaXN0aWFuIEvDtnN0bGlu?= (10/19) Aug 29 2011 Thats very interesting!
• torhu (2/11) Aug 29 2011 You could try just using the functions in std.stdio instead.

Leon <leonp gmail.com> writes:

I'm writing a simple program which uses merge sort to sort very large text
files. I split the text file into several temporary files which are sorted and
merged together.

The trouble I'm having is that the BufferedFile class seems to have a memory
leak. The memory usage of the process slowly goes up until it throws an out of
memory exception. I can fix the memory leak by switching to the File class,
but that makes the program much slower.

Calling GC.collect() didn't help, and neither did explicitly deleting the
BufferedFile object.

=?UTF-8?B?Q2hyaXN0aWFuIEvDtnN0bGlu?= <christian.koestlin gmail.com> writes:

On 8/28/11 1:00 , Leon wrote:
 I'm writing a simple program which uses merge sort to sort very large text
 files. I split the text file into several temporary files which are sorted and
 merged together.

 The trouble I'm having is that the BufferedFile class seems to have a memory
 leak. The memory usage of the process slowly goes up until it throws an out of
 memory exception. I can fix the memory leak by switching to the File class,
 but that makes the program much slower.

 Calling GC.collect() didn't help, and neither did explicitly deleting the
 BufferedFile object.

Thats very interesting!
Perhaps you can have a look at: https://github.com/gizmomogwai/performance
There I ran into performanceproblems using BufferedFile.
One of the best solutions I found till now is using mmfile as in 
https://github.com/gizmomogwai/performance/blob/master/src/d/readbytes5.d. 
Perhaps this could be of help for you.

I dont know how you have to access the data in the files, but perhaps 
the interface of mmfile is good enough for you!


christian

torhu <no spam.invalid> writes:

On 28.08.2011 01:00, Leon wrote:
 I'm writing a simple program which uses merge sort to sort very large text
 files. I split the text file into several temporary files which are sorted and
 merged together.

 The trouble I'm having is that the BufferedFile class seems to have a memory
 leak. The memory usage of the process slowly goes up until it throws an out of
 memory exception. I can fix the memory leak by switching to the File class,
 but that makes the program much slower.

 Calling GC.collect() didn't help, and neither did explicitly deleting the
 BufferedFile object.

You could try just using the functions in std.stdio instead.