www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 1980] New: memory allocated via double[][] is not being freed by the GC

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1980

           Summary: memory allocated via double[][] is not being freed by
                    the GC
           Product: D
           Version: 2.012
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: DMD
        AssignedTo: bugzilla digitalmars.com
        ReportedBy: markusle gmail.com


Hi,

For a data processing application I need to read a large number
of data sets from disk. Due to their size, they have to be read and
processed sequentially, i.e. in pseudocode

int main()
{
    while (some condition)
    {
         double[][] myLargeDataset = read_data();
         process_data(myLargeDataset);
         // free all memory here otherwise next cycle will
         // run out of memory
     }

   return 0;
}

Now, the "problem" is the fact that each single data-set saturates
system memory and hence I need to make sure that all memory
is freed after each process_data step is complete. Unfortunately,
using dmd-2.012 I have not been able to achieve this. Whatever
I do (including nothing, i.e., letting the GC do its job), the resulting
binary keeps accumulating memory and crashing shortly after.
I've tried deleting the array, setting it to null, setting the array 
lengths to 0, and manually forcing the GC to collect, to no avail. 
Hence, is there something I am doing terribly wrong or is this a 
bug in dmd? 

Thanks much,
Markus


--
Apr 08 2008
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1980






Below is a complete piece of code that still exhibits this
issue. The file "data_random.dat" simply contains a single
row of random ints (about 30M filesize). Each iteration
allocated additional heap and I don't seem to be able to
free foo no matter what.

import std.stream;
import std.stdio;
import std.contracts;
import std.gc;


double[][] parse(BufferedFile inputFile)
{

  double[][] array;
  foreach(char[] line; inputFile)
  {
    double[] temp;

    foreach(string item; std.string.split(assumeUnique(line)))
    {
       temp ~= std.string.atof(item);
    }

    array ~= temp;
  }

  /* rewind for next round */
  inputFile.seekSet(0);

  return array;
}



int main()
{
  BufferedFile inputFile = new BufferedFile("data_random.dat");

  while(1)
  {
    double[][] foo = parse(inputFile);
  }

  return 1;
}


--
Apr 08 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1980






The test program works fine for me on Windows.
I tried it with
  DMD/Phobos 1.028,
  DMD/Tango/Tangobos 1.028, and
  DMD/Phobos 2.012.

Although the data file I tested it with was only 50K.  That could make a
difference if it's being caused by "false pointers".  Or it could be a
Linux-only bug, I suppose.

Maybe for the lazy you could also modify your program to generate a 30MB file
full of random numbers.  :-)


--
Apr 08 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1980






Thanks much for testing Bill!

I just ran both the 32bit DMD executable and the
64bit GDC (v. 0.24) version of the above code
on my opteron box. The 32bit DMD version leaks memory 
whereas the GDC binary works just fine and properly 
free its memory.

Thanks,
Markus



--
Apr 08 2008
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1980






It's possible that this might be related to issue 1804
(http://d.puremagic.com/issues/show_bug.cgi?id=1804)


--
Apr 09 2008