www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Using closure causes GC allocation

reply Vino.B <vino.bheeman hotmail.com> writes:
Hi All,

    Request your help on how to solve the issue in the below code 
as when i execute the program with -vgc it state as below:

NewTD.d(21): vgc: using closure causes GC allocation
NewTD.d(25): vgc: array literal may cause GC allocation

void logClean (string[] Lglst, int LogAge) {   //Line 21
	if (!Lglst[0].exists) { mkdir(Lglst[0]); }
	auto ct1 = Clock.currTime();
	auto st1 = ct1 + days(-LogAge);
	auto dFiles = dirEntries(Lglst[0], SpanMode.shallow).filter!(a 
=> a.exists && a.isFile && a.timeCreated < st1).map!(a 
=>[a.name]).array;  // Line 25
	  dFiles.each!(f => f[0].remove);
}

From,
Vino.B
Sep 02 2017
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Saturday, 2 September 2017 at 17:43:08 UTC, Vino.B wrote:

 Hi All,

    Request your help on how to solve the issue in the below 
 code as when i execute the program with -vgc it state as below:

 NewTD.d(21): vgc: using closure causes GC allocation
 NewTD.d(25): vgc: array literal may cause GC allocation

 void logClean (string[] Lglst, int LogAge) {   //Line 21
 	if (!Lglst[0].exists) { mkdir(Lglst[0]); }
 	auto ct1 = Clock.currTime();
 	auto st1 = ct1 + days(-LogAge);
 	auto dFiles = dirEntries(Lglst[0], SpanMode.shallow).filter!(a 
 => a.exists && a.isFile && a.timeCreated < st1).map!(a 
 =>[a.name]).array;  // Line 25
 	  dFiles.each!(f => f[0].remove);
 }


Line 25 happens because of `[a.name]`. You request a new array: 
the memory for this has to be allocated (the reason why the 
compiler says "may" is because sometimes, e.g. if the array 
literal itself contains only literals, the allocations needn't 
happen at runtime and no GC call is necessary). Since you don't 
actually use the array, get rid of it:

---
void logClean (string[] Lglst, int LogAge) {   //Line 21
	if (!Lglst[0].exists) { mkdir(Lglst[0]); }
	auto ct1 = Clock.currTime();
	auto st1 = ct1 + days(-LogAge);
	auto dFiles = dirEntries(Lglst[0], SpanMode.shallow).filter!(a 
=> a.exists && a.isFile && a.timeCreated < st1).array;  // Line 25
	  dFiles.each!(f => f.remove);
}
---

I cannot reproduce the line 21 report, though.
Since you use `timeCreated` I assume you're on Windows, but 
what's your D compiler, which D frontend version are you using, 
etc. (all the things needed to attempt to reproduce the error).
Sep 02 2017
parent reply vino.b <vino.bheeman hotmail.com> writes:
On Saturday, 2 September 2017 at 18:02:06 UTC, Moritz Maxeiner 
wrote:

 On Saturday, 2 September 2017 at 17:43:08 UTC, Vino.B wrote:

 [...]


 Line 25 happens because of `[a.name]`. You request a new array: 
 the memory for this has to be allocated (the reason why the 
 compiler says "may" is because sometimes, e.g. if the array 
 literal itself contains only literals, the allocations needn't 
 happen at runtime and no GC call is necessary). Since you don't 
 actually use the array, get rid of it:

 [...]


Hi,

   Thank you for your help and the DMD version that i am using is 
DMD 2.076.0 and yes I am on windows.

From,
Vino.B
Sep 02 2017
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Saturday, 2 September 2017 at 18:08:19 UTC, vino.b wrote:

 On Saturday, 2 September 2017 at 18:02:06 UTC, Moritz Maxeiner 
 wrote:

 On Saturday, 2 September 2017 at 17:43:08 UTC, Vino.B wrote:

 [...]


 Line 25 happens because of `[a.name]`. You request a new 
 array: the memory for this has to be allocated (the reason why 
 the compiler says "may" is because sometimes, e.g. if the 
 array literal itself contains only literals, the allocations 
 needn't happen at runtime and no GC call is necessary). Since 
 you don't actually use the array, get rid of it:

 [...]


 Hi,

   Thank you for your help and the DMD version that i am using 
 is DMD 2.076.0 and yes I am on windows.


Please post a compilable, minimal example including how that 
function gets called that yields you that compiler output.
Sep 02 2017
parent reply Vino.B <vino.bheeman hotmail.com> writes:
On Saturday, 2 September 2017 at 18:32:55 UTC, Moritz Maxeiner 
wrote:

 On Saturday, 2 September 2017 at 18:08:19 UTC, vino.b wrote:

 On Saturday, 2 September 2017 at 18:02:06 UTC, Moritz Maxeiner 
 wrote:

 On Saturday, 2 September 2017 at 17:43:08 UTC, Vino.B wrote:

 [...]


 Line 25 happens because of `[a.name]`. You request a new 
 array: the memory for this has to be allocated (the reason 
 why the compiler says "may" is because sometimes, e.g. if the 
 array literal itself contains only literals, the allocations 
 needn't happen at runtime and no GC call is necessary). Since 
 you don't actually use the array, get rid of it:

 [...]


 Hi,

   Thank you for your help and the DMD version that i am using 
 is DMD 2.076.0 and yes I am on windows.


 Please post a compilable, minimal example including how that 
 function gets called that yields you that compiler output.


Hi,

  Please find the example code below,

import std.stdio: File,writeln;
import std.datetime.systime: Clock, days, SysTime;
import std.file: SpanMode, dirEntries, exists, isFile, mkdir, 
remove;
import std.typecons: tuple;
import std.algorithm:  filter, map, each;
import std.array: array;

void logClean (string[] Lglst, int LogAge) {
	if (!Lglst[0].exists) { mkdir(Lglst[0]); }
	auto ct1 = Clock.currTime();
	auto st1 = ct1 + days(-LogAge);
	auto dFiles = dirEntries(Lglst[0], SpanMode.shallow).filter!(a 
=> a.exists && a.isFile && a.timeCreated < st1).map!(a => 
tuple(a.name)).array;
	  dFiles.each!(a => a[0].remove);
}

void main () {
string[] LogDir = 
["C:\\Users\\bheev1\\Desktop\\Current\\Script\\D\\Logs"];
int  LogAge = 1;
logClean(LogDir,LogAge);
}

Another similar issue :
  I removed the [a.name] and the issue in line 25 has resolved, 
but for another function i am getting the same error

string[][] cleanFiles(string FFs, string Step) {
	auto dFiles = dirEntries(FFs, SpanMode.shallow).filter!(a => 
a.isFile).map!(a =>[a.name , a.timeCreated.toSimpleString[0 .. 
20]]).array;  -> Issue in this line
     if (Step == "run")
         dFiles.each!(a => a[0].remove);
		return dFiles;
}

if the replace the line in error as below then i am getting the 
error "Error: cannot implicitly convert expression dFiles of type 
Tuple!(string, string)[] to string[][]"

auto dFiles = dirEntries(FFs, SpanMode.shallow).filter!(a => 
a.isFile).map!(a => tuple(a.name, a.timeCreated.toSimpleString[0 
.. 20])).array;
Sep 02 2017
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Saturday, 2 September 2017 at 18:59:30 UTC, Vino.B wrote:

 On Saturday, 2 September 2017 at 18:32:55 UTC, Moritz Maxeiner 
 wrote:

 On Saturday, 2 September 2017 at 18:08:19 UTC, vino.b wrote:

 On Saturday, 2 September 2017 at 18:02:06 UTC, Moritz 
 Maxeiner wrote:

 On Saturday, 2 September 2017 at 17:43:08 UTC, Vino.B wrote:

 [...]


 Line 25 happens because of `[a.name]`. You request a new 
 array: the memory for this has to be allocated (the reason 
 why the compiler says "may" is because sometimes, e.g. if 
 the array literal itself contains only literals, the 
 allocations needn't happen at runtime and no GC call is 
 necessary). Since you don't actually use the array, get rid 
 of it:

 [...]


 Hi,

   Thank you for your help and the DMD version that i am using 
 is DMD 2.076.0 and yes I am on windows.


 Please post a compilable, minimal example including how that 
 function gets called that yields you that compiler output.


 Hi,

  Please find the example code below,

 [...]


Cannot reproduce under Linux with dmd 2.076.0 (with commented out 
Windows-only check). I'll try to see what happens on Windows once 
I have a VM setup.


 Another similar issue :
  I removed the [a.name] and the issue in line 25 has resolved, 
 but for another function i am getting the same error

 string[][] cleanFiles(string FFs, string Step) {
 	auto dFiles = dirEntries(FFs, SpanMode.shallow).filter!(a => 
 a.isFile).map!(a =>[a.name , a.timeCreated.toSimpleString[0 .. 
 20]]).array;  -> Issue in this line
     if (Step == "run")
         dFiles.each!(a => a[0].remove);
 		return dFiles;
 }

 if the replace the line in error as below then i am getting the 
 error "Error: cannot implicitly convert expression dFiles of 
 type Tuple!(string, string)[] to string[][]"

 auto dFiles = dirEntries(FFs, SpanMode.shallow).filter!(a => 
 a.isFile).map!(a => tuple(a.name, 
 a.timeCreated.toSimpleString[0 .. 20])).array;


You changed the type of dFiles, which you return from cleanFiles, 
without changing the return type of cleanFiles. Change the return 
type of cleanFiles to the type the compiler error above tells you 
it should be (`Tuple!(string, string)[]` instead of 
`string[][]`), or let the compiler infer it via auto (`auto 
cleanFiles(...`).
Sep 02 2017
parent reply Vino.B <vino.bheeman hotmail.com> writes:
On Saturday, 2 September 2017 at 20:10:58 UTC, Moritz Maxeiner 
wrote:

 On Saturday, 2 September 2017 at 18:59:30 UTC, Vino.B wrote:

 [...]


 Cannot reproduce under Linux with dmd 2.076.0 (with commented 
 out Windows-only check). I'll try to see what happens on 
 Windows once I have a VM setup.


 [...]


 You changed the type of dFiles, which you return from 
 cleanFiles, without changing the return type of cleanFiles. 
 Change the return type of cleanFiles to the type the compiler 
 error above tells you it should be (`Tuple!(string, string)[]` 
 instead of `string[][]`), or let the compiler infer it via auto 
 (`auto cleanFiles(...`).


Hi,

  Thank you very much, was able to resolve the second code issue 
by changing the return type of the function.
Sep 02 2017
parent reply Vino.B <vino.bheeman hotmail.com> writes:
On Saturday, 2 September 2017 at 20:54:03 UTC, Vino.B wrote:

 On Saturday, 2 September 2017 at 20:10:58 UTC, Moritz Maxeiner 
 wrote:

 On Saturday, 2 September 2017 at 18:59:30 UTC, Vino.B wrote:

 [...]


 Cannot reproduce under Linux with dmd 2.076.0 (with commented 
 out Windows-only check). I'll try to see what happens on 
 Windows once I have a VM setup.


 [...]


 You changed the type of dFiles, which you return from 
 cleanFiles, without changing the return type of cleanFiles. 
 Change the return type of cleanFiles to the type the compiler 
 error above tells you it should be (`Tuple!(string, string)[]` 
 instead of `string[][]`), or let the compiler infer it via 
 auto (`auto cleanFiles(...`).


 Hi,

  Thank you very much, was able to resolve the second code issue 
 by changing the return type of the function.


Hi,

   In order to resolve the issue "Using closure causes GC 
allocation" it was stated that we need to use delegates, can you 
please help me on how to as i have not gone that much far in D 
programming.

import std.stdio: File,writeln;
import std.datetime.systime: Clock, days, SysTime;
import std.file: SpanMode, dirEntries, exists, isFile, mkdir, 
remove;
import std.typecons: tuple, Tuple;
import std.algorithm:  filter, map, each;
import std.array: array;

Tuple!(string)[] logClean (string[] Lglst, int LogAge) {
	if (!Lglst[0].exists) { mkdir(Lglst[0]); }
	auto ct1 = Clock.currTime();
	auto st1 = ct1 + days(-LogAge);
	auto dFiles = dirEntries(Lglst[0], SpanMode.shallow).filter!(a 
=> a.exists && a.isFile && a.timeCreated < st1).map!(a => 
tuple(a.name)).array;
	dFiles.each!(a => a[0].remove);
	return dFiles;
}

void main () {
string[] LogDir = 
["C:\\Users\\bheev1\\Desktop\\Current\\Script\\D\\Logs"];
int  LogAge = 1;
logClean(LogDir,LogAge);
}

From,
Vino.B
Sep 03 2017
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 4 September 2017 at 05:45:18 UTC, Vino.B wrote:

 On Saturday, 2 September 2017 at 20:54:03 UTC, Vino.B wrote:

 On Saturday, 2 September 2017 at 20:10:58 UTC, Moritz Maxeiner 
 wrote:

 On Saturday, 2 September 2017 at 18:59:30 UTC, Vino.B wrote:

 [...]


 Cannot reproduce under Linux with dmd 2.076.0 (with commented 
 out Windows-only check). I'll try to see what happens on 
 Windows once I have a VM setup.


 [...]


 You changed the type of dFiles, which you return from 
 cleanFiles, without changing the return type of cleanFiles. 
 Change the return type of cleanFiles to the type the compiler 
 error above tells you it should be (`Tuple!(string, 
 string)[]` instead of `string[][]`), or let the compiler 
 infer it via auto (`auto cleanFiles(...`).


 Hi,

  Thank you very much, was able to resolve the second code 
 issue by changing the return type of the function.


 Hi,

   In order to resolve the issue "Using closure causes GC 
 allocation" it was stated that we need to use delegates, can 
 you please help me on how to as i have not gone that much far 
 in D programming.

 import std.stdio: File,writeln;
 import std.datetime.systime: Clock, days, SysTime;
 import std.file: SpanMode, dirEntries, exists, isFile, mkdir, 
 remove;
 import std.typecons: tuple, Tuple;
 import std.algorithm:  filter, map, each;
 import std.array: array;

 Tuple!(string)[] logClean (string[] Lglst, int LogAge) {
 	if (!Lglst[0].exists) { mkdir(Lglst[0]); }
 	auto ct1 = Clock.currTime();
 	auto st1 = ct1 + days(-LogAge);
 	auto dFiles = dirEntries(Lglst[0], SpanMode.shallow).filter!(a 
 => a.exists && a.isFile && a.timeCreated < st1).map!(a => 
 tuple(a.name)).array;
 	dFiles.each!(a => a[0].remove);
 	return dFiles;
 }

 void main () {
 string[] LogDir = 
 ["C:\\Users\\bheev1\\Desktop\\Current\\Script\\D\\Logs"];
 int  LogAge = 1;
 logClean(LogDir,LogAge);
 }

 From,
 Vino.B


If you are getting Using closure causes GC allocation from the 
above, it will be because you reference a local in the template 
delegate argument to filter:

filter!(a => a.exists &&
         a.isFile &&
         a.timeCreated < st1)
                        ^^^
because of the local (st1) it needs a closure. I can't remember 
how to transform delegates to take parameters and then call with 
the parameter set to the local, someone else will have to fill 
you in on that one.
Sep 04 2017
prev sibling parent reply Azi Hassan <azi.hassan live.fr> writes:
On Monday, 4 September 2017 at 05:45:18 UTC, Vino.B wrote:

   In order to resolve the issue "Using closure causes GC 
 allocation" it was stated that we need to use delegates


Alternatively you can drop the functional style and use a foreach 
loop that doesn't require delegates, but you'd still need the GC 
to store the result in an array. And even then you could still 
use Array (from std.container).
Sep 04 2017
parent Vino.B <vino.bheeman hotmail.com> writes:
On Monday, 4 September 2017 at 14:42:45 UTC, Azi Hassan wrote:

 On Monday, 4 September 2017 at 05:45:18 UTC, Vino.B wrote:

   In order to resolve the issue "Using closure causes GC 
 allocation" it was stated that we need to use delegates


 Alternatively you can drop the functional style and use a 
 foreach loop that doesn't require delegates, but you'd still 
 need the GC to store the result in an array. And even then you 
 could still use Array (from std.container).


Hi All,

  Was able to resolve this issue, thank you for your help, below 
is the changes that i did to resolve the issue.


import std.stdio: File,writeln;
import std.datetime.systime: Clock, days, SysTime;
import std.file: SpanMode, dirEntries, exists, isFile, mkdir, 
remove;
import std.typecons: tuple, Tuple;
import std.algorithm:  filter, map, each;
import std.array: array;

Tuple!(string)[] logClean (string[] Lglst, int LogAge) {
	if (!Lglst[0].exists) { mkdir(Lglst[0]); }
		auto dFiles = dirEntries(Lglst[0], SpanMode.shallow).filter!(a 
=a.exists && a.isFile && a.timeCreated < dtLogAge).map!(a 
=tuple(a.name)).array;
	dFiles.each!(a =a[0].remove);
	return dFiles;
}

SysTime dtLogAge () {
int LogAge = mParams[1];
auto ct2 = Clock.currTime();
auto st2 = ct2 + days(-LogAge);
return st2;
}

void main () {
string[] LogDir = 
["C:\\Users\\admin\\Desktop\\Current\\Script\\D\\Logs"];
logClean(LogDir);
}

"mParams" is another function that reads the value from the 
configuration file and returns the value for the LogAge as 
defined in the configuration file.

From,
Vino.B
Sep 05 2017