www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Is there something special required to use Appender.clear

reply Ellie Harper <elliefops gmail.com> writes:
Sorry if this is a stupid question, but is there something 
special required to call Appender.clear?  When I attempt even 
just a simple use I am getting compile errors relating to 
`template object.clear`.

When I try:

     import std.array;

     void main(string[] args){
       auto foo = appender!string;
       foo.clear;
     }

I receive the following output:

     $ ldc2 source/tmp.d
     source/tmp.d(5): Error: template object.clear cannot deduce 
function from argument types !()(Appender!string), candidates are:
     /usr/local/include/d/ldc/object.d(2041):        
object.clear(T : Value[Key], Value, Key)(T aa)
     /usr/local/include/d/ldc/object.d(2046):        
object.clear(T : Value[Key], Value, Key)(T* aa)

     $ dmd source/tmp.d
     source/tmp.d(5): Error: template object.clear cannot deduce 
function from argument types !()(Appender!string), candidates are:
     /usr/include/dmd/druntime/import/object.d(1983):        
object.clear(T : Value[Key], Value, Key)(T aa)
     /usr/include/dmd/druntime/import/object.d(1988):        
object.clear(T : Value[Key], Value, Key)(T* aa)


versions:

     $ dmd --version
     DMD64 D Compiler v2.078.0
     Copyright (c) 1999-2017 by The D Language Foundation written 
by Walter Bright

     $ ldc2 --version
     LDC - the LLVM D compiler (1.8.0git-921bb7f):
       based on DMD v2.078.3 and LLVM 4.0.1
       built with DMD64 D Compiler v2.078.0


Is there something I am missing here?

Thanks
Mar 27 2018
next sibling parent bauss <jj_1337 live.dk> writes:
On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
 Sorry if this is a stupid question, but is there something 
 special required to call Appender.clear?  When I attempt even 
 just a simple use I am getting compile errors relating to 
 `template object.clear`.

 When I try:

     import std.array;

     void main(string[] args){
       auto foo = appender!string;
       foo.clear;
     }

 I receive the following output:

     $ ldc2 source/tmp.d
     source/tmp.d(5): Error: template object.clear cannot deduce 
 function from argument types !()(Appender!string), candidates 
 are:
     /usr/local/include/d/ldc/object.d(2041):        
 object.clear(T : Value[Key], Value, Key)(T aa)
     /usr/local/include/d/ldc/object.d(2046):        
 object.clear(T : Value[Key], Value, Key)(T* aa)

     $ dmd source/tmp.d
     source/tmp.d(5): Error: template object.clear cannot deduce 
 function from argument types !()(Appender!string), candidates 
 are:
     /usr/include/dmd/druntime/import/object.d(1983):        
 object.clear(T : Value[Key], Value, Key)(T aa)
     /usr/include/dmd/druntime/import/object.d(1988):        
 object.clear(T : Value[Key], Value, Key)(T* aa)


 versions:

     $ dmd --version
     DMD64 D Compiler v2.078.0
     Copyright (c) 1999-2017 by The D Language Foundation 
 written by Walter Bright

     $ ldc2 --version
     LDC - the LLVM D compiler (1.8.0git-921bb7f):
       based on DMD v2.078.3 and LLVM 4.0.1
       built with DMD64 D Compiler v2.078.0


 Is there something I am missing here?

 Thanks
I would say that's a bug, because it seems like it's trying to get the clear() function used for associative arrays.
Mar 27 2018
prev sibling next sibling parent Anonymouse <asdf asdf.net> writes:
On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
 Sorry if this is a stupid question, but is there something 
 special required to call Appender.clear?  When I attempt even 
 just a simple use I am getting compile errors relating to 
 `template object.clear`.

 When I try:

     import std.array;

     void main(string[] args){
       auto foo = appender!string;
       foo.clear;
     }
Appender only implements clear for mutable types, and string is an array of immutables. I'm not sure but it looks like you're hitting the associative array clear by UFCS instead? Make it Appender!(char[]) and it will work.
 // only allow overwriting data on non-immutable and non-const 
 data
 static if (isMutable!T)
 {
     /**
       * Clears the managed array.  This allows the elements of 
 the array to be reused
       * for appending.
       *
       * Note: clear is disabled for immutable or const element 
 types, due to the
       * possibility that $(D Appender) might overwrite 
 immutable data.
       */
      void clear()  trusted pure nothrow
      {
          if (_data)
          {
              _data.arr = _data.arr.ptr[0 .. 0];
          }
      }
 [...]
https://github.com/dlang/phobos/blob/master/std/array.d#L3140
Mar 27 2018
prev sibling parent reply Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
 Sorry if this is a stupid question, but is there something 
 special required to call Appender.clear?  When I attempt even 
 just a simple use I am getting compile errors relating to 
 `template object.clear`.
From the documentation for Appender:
 Note
 clear is disabled for immutable or const element types, due to 
 the possibility that Appender might overwrite immutable data.
Since string is immutable(char)[], clear() is simply not available for appender!string. -- Simen
Mar 27 2018
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
 On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
 Sorry if this is a stupid question, but is there something 
 special required to call Appender.clear?  When I attempt even 
 just a simple use I am getting compile errors relating to 
 `template object.clear`.
From the documentation for Appender:
 Note
 clear is disabled for immutable or const element types, due to 
 the possibility that Appender might overwrite immutable data.
Since string is immutable(char)[], clear() is simply not available for appender!string. -- Simen
Isn't this wrong, though? Appender controls the memory it references. It could just choose to allocate non-immutable memory internally. As long as any const data put into the appender is *returned* as const, there is no chance of immutable memory being overwritten.
Jan 24
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 01/24/2019 04:35 AM, FeepingCreature wrote:
 On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
 On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
 Sorry if this is a stupid question, but is there something special
 required to call Appender.clear?  When I attempt even just a simple
 use I am getting compile errors relating to `template object.clear`.
From the documentation for Appender:
 Note
 clear is disabled for immutable or const element types, due to the
 possibility that Appender might overwrite immutable data.
Since string is immutable(char)[], clear() is simply not available for appender!string. -- Simen
Isn't this wrong, though? Appender controls the memory it references. It could just choose to allocate non-immutable memory internally. As long as any const data put into the appender is *returned* as const, there is no chance of immutable memory being overwritten.
I think Appender is trying to protect previous data from Appender's later use. If it handed out immutable data, the user is expecting it to not change. So, Appender cannot clear it for later use. Ali
Jan 24
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Thursday, 24 January 2019 at 17:49:34 UTC, Ali Çehreli wrote:
 On 01/24/2019 04:35 AM, FeepingCreature wrote:
 On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
 On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper
wrote:
 Sorry if this is a stupid question, but is there something
special
 required to call Appender.clear?  When I attempt even just
a simple
 use I am getting compile errors relating to `template
object.clear`.
 From the documentation for Appender:

 Note
 clear is disabled for immutable or const element types, due
to the
 possibility that Appender might overwrite immutable data.
Since string is immutable(char)[], clear() is simply not
available for
 appender!string.

 --
   Simen
Isn't this wrong, though? Appender controls the memory it
references. It
 could just choose to allocate non-immutable memory
internally. As long
 as any const data put into the appender is *returned* as
const, there is
 no chance of immutable memory being overwritten.
I think Appender is trying to protect previous data from Appender's later use. If it handed out immutable data, the user is expecting it to not change. So, Appender cannot clear it for later use. Ali
Aren't the semantics of .clear that it's invalid to access references to .data after calling .clear, period? And if not, then shouldn't they be? Consider if Appender managed its own memory and held on to previously-allocated arrays while you were appending, only to free them on .clear. That seems like something Appender should be allowed to do. If you don't want it, just reinitialize Appender instead of calling .clear.
 Appender is sometimes given a starting array. clear isn't 
 callable in that case, and we don't distinguish the difference 
 in the type or at runtime.
Any reason that the semantics of .clear should be different for a starting array? Anyway if so, I'd prefer to just make that a runtime error. I'm mostly fishing around if anyone has an objection to a PR to change this.
Jan 25
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/25/19 3:20 AM, FeepingCreature wrote:
 On Thursday, 24 January 2019 at 17:49:34 UTC, Ali Çehreli wrote:
 On 01/24/2019 04:35 AM, FeepingCreature wrote:
 On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
 On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper
wrote:
 Sorry if this is a stupid question, but is there something
special
 required to call Appender.clear?  When I attempt even just
a simple
 use I am getting compile errors relating to `template
object.clear`.
 From the documentation for Appender:

 Note
 clear is disabled for immutable or const element types, due
to the
 possibility that Appender might overwrite immutable data.
Since string is immutable(char)[], clear() is simply not
available for
 appender!string.

 --
   Simen
Isn't this wrong, though? Appender controls the memory it
references. It
 could just choose to allocate non-immutable memory
internally. As long
 as any const data put into the appender is *returned* as
const, there is
 no chance of immutable memory being overwritten.
I think Appender is trying to protect previous data from Appender's later use. If it handed out immutable data, the user is expecting it to not change. So, Appender cannot clear it for later use. Ali
Aren't the semantics of .clear that it's invalid to access references to .data after calling .clear, period? And if not, then shouldn't they be? Consider if Appender managed its own memory and held on to previously-allocated arrays while you were appending, only to free them on .clear. That seems like something Appender should be allowed to do. If you don't want it, just reinitialize Appender instead of calling .clear.
You are advised not to. But it's not unsafe, as the memory is still there.
 
 Appender is sometimes given a starting array. clear isn't callable in 
 that case, and we don't distinguish the difference in the type or at 
 runtime.
Any reason that the semantics of .clear should be different for a starting array? Anyway if so, I'd prefer to just make that a runtime error.
Yes, because Appender doesn't know where that memory comes from. A string could be, for instance, passed to another thread and being used, you wouldn't want to overwrite it. Generally speaking, overwriting immutable data is UB in D anyway. Making it a runtime error would be possible, but there has to be a good reason to make it that way.
 
 I'm mostly fishing around if anyone has an objection to a PR to change 
 this.
Without good reasons to change, I don't see why it would be accepted. Maybe you can describe your use case? -Steve
Jan 25
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 25 January 2019 at 14:33:16 UTC, Steven Schveighoffer 
wrote:
 On 1/25/19 3:20 AM, FeepingCreature wrote:
 On Thursday, 24 January 2019 at 17:49:34 UTC, Ali Çehreli 
 wrote:
 Aren't the semantics of .clear that it's invalid to access 
 references to .data after calling .clear, period? And if not, 
 then shouldn't they be? Consider if Appender managed its own 
 memory and held on to previously-allocated arrays while you 
 were appending, only to free them on .clear. That seems like 
 something Appender should be allowed to do. If you don't want 
 it, just reinitialize Appender instead of calling .clear.
You are advised not to. But it's not unsafe, as the memory is still there.
That's stupid. Why would you advise me not to, if there's no problem with it? Either it should be accepted or it should be forbidden, just like warnings.
 Any reason that the semantics of .clear should be different 
 for a starting array? Anyway if so, I'd prefer to just make 
 that a runtime error.
Yes, because Appender doesn't know where that memory comes from. A string could be, for instance, passed to another thread and being used, you wouldn't want to overwrite it.
In which case, you can't reuse the Appender and should not call `clear`! It seems like we're gimping functionality of a very basic tool for memory efficiency for the sake of not confusing users. This rarely pays off.
 Generally speaking, overwriting immutable data is UB in D 
 anyway.

 Making it a runtime error would be possible, but there has to 
 be a good reason to make it that way.

 
 I'm mostly fishing around if anyone has an objection to a PR 
 to change this.
Without good reasons to change, I don't see why it would be accepted. Maybe you can describe your use case?
My use case is simply the standard usecase of Appender: I want to build up an array in a way that reduces GC churn. Maybe it's an array of structs that contain const members that I'll serialize to JSON and send on a socket. In that case, I know for a fact that no references will hang around past the serialization. That's what clear _is for._ I don't see why this would be different with const or immutable data; if you hold references past .clear being called you're in trouble *anyways.* I consider initializing Appender with an array referencing immutable data a borderline error anyways. The entire point of Appender is that it caches capacity data of GC managed memory, which is never immutable. On the first append to an immutable-memory array, it has to reallocate *anyways*. There is no benefit to initializing an Appender with immutable memory over just appending it first thing, unless you never plan to append to it ever.
Jan 28
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/28/19 7:39 AM, FeepingCreature wrote:
 On Friday, 25 January 2019 at 14:33:16 UTC, Steven Schveighoffer wrote:
 On 1/25/19 3:20 AM, FeepingCreature wrote:
 On Thursday, 24 January 2019 at 17:49:34 UTC, Ali Çehreli wrote:
 Aren't the semantics of .clear that it's invalid to access references 
 to .data after calling .clear, period? And if not, then shouldn't 
 they be? Consider if Appender managed its own memory and held on to 
 previously-allocated arrays while you were appending, only to free 
 them on .clear. That seems like something Appender should be allowed 
 to do. If you don't want it, just reinitialize Appender instead of 
 calling .clear.
You are advised not to. But it's not unsafe, as the memory is still there.
That's stupid. Why would you advise me not to, if there's no problem with it? Either it should be accepted or it should be forbidden, just like warnings.
Just like this: int[] arr = [1,2,3,4,5]; auto slice = arr[0 .. 2]; slice.assumeSafeAppend; Now, you can use arr still, it's got elements in it beyond what slice has. You might have things change underneath you without expecting it. That's why it's advised not to do that. But if arr is immutable(int)[], then you are running into undefined behavior, it's a completely different problem.
 Without good reasons to change, I don't see why it would be accepted.

 Maybe you can describe your use case?
My use case is simply the standard usecase of Appender: I want to build up an array in a way that reduces GC churn. Maybe it's an array of structs that contain const members that I'll serialize to JSON and send on a socket. In that case, I know for a fact that no references will hang around past the serialization. That's what clear _is for._ I don't see why this would be different with const or immutable data; if you hold references past .clear being called you're in trouble *anyways.*
Right, this does seem like a big limitation. Keeping with the spirit of how slices don't own the memory in question, Appender is being conservative with what it doesn't know. I wonder if it may be more appropriate to instead of preventing clear() on immutable/const arrays, to make it system. Or maybe call it something different "dangerousClear" or something ;) There definitely should be some way to fail if clear is called on an array that was passed into the constructor. But I'm still not sure we should allow overwriting immutable memory without a cast, even in system code.
 I consider initializing Appender with an array referencing immutable 
 data a borderline error anyways. The entire point of Appender is that it 
 caches capacity data of GC managed memory, which is never immutable. On 
 the first append to an immutable-memory array, it has to reallocate 
 *anyways*. There is no benefit to initializing an Appender with 
 immutable memory over just appending it first thing, unless you never 
 plan to append to it ever.
It will inspect the allocated length from the GC if the array is appendable from the beginning. So it's not always going to reallocate. e.g.: string x = "abc".idup; auto app = x.appender; app ~= "xyz"; // does not reallocate. -Steve
Jan 28
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 28 January 2019 at 15:16:54 UTC, Steven Schveighoffer 
wrote:
 It will inspect the allocated length from the GC if the array 
 is appendable from the beginning. So it's not always going to 
 reallocate.

 e.g.:

 string x = "abc".idup;

 auto app = x.appender;

 app ~= "xyz"; // does not reallocate.

 -Steve
Fair enough.
 My use case is simply the standard usecase of Appender: I want 
 to build up an array in a way that reduces GC churn. Maybe 
 it's an array of structs that contain const members that I'll 
 serialize to JSON and send on a socket. In that case, I know 
 for a fact that no references will hang around past the 
 serialization. That's what clear _is for._ I don't see why 
 this would be different with const or immutable data; if you 
 hold references past .clear being called you're in trouble 
 *anyways.*
Right, this does seem like a big limitation. Keeping with the spirit of how slices don't own the memory in question, Appender is being conservative with what it doesn't know. I wonder if it may be more appropriate to instead of preventing clear() on immutable/const arrays, to make it system. Or maybe call it something different "dangerousClear" or something ;) There definitely should be some way to fail if clear is called on an array that was passed into the constructor. But I'm still not sure we should allow overwriting immutable memory without a cast, even in system code.
My problem is this. const and immutable are *not* well supported in the standard library at the moment. What I want is that I can use the idioms of the stdlib, semantically, in a way that lets me *not care* about const or immutable, that lets me express the patterns I want without having to worry about whether some type deep inside my code, a dub library, or phobos itself decided to declare a field immutable. Appender covers two usecases: "capacity caching" and "memory reuse." Both of those usecases have two const variations: "possibly immutable" and "mutable". mutable capacity caching | mutable memory reuse ---------------------------+--------------------- immutable capacity caching | immutable memory reuse But instead of cutting between capacity caching and memory reuse, down the middle, Appender cuts between mutable and immutable, left to right. I think this is a symptom of a broad negligence of constness as a first-class property - constness works sometimes, and it's nice if it does, but it can't be *relied* on to be well supported. This makes using const in D an eternal uphill struggle. Why even go to all the trouble to make a new major version of the language just to introduce constness, if it's not going to be treated as a first-class concern? I don't want to have to sprinkle static if(isAssignable!T) everytime I want to use a library type. If Phobos offers me as generic a data structure as Appender, parameterized on T, it should work for T, regardless of what T is doing, and *certainly* regardless of what fields in T are marked const. Especially considering that if a field is not marked as const, that hardly means it's not going to lead to bugs if its memory is reused while you're referencing it!
Jan 30
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/30/19 7:32 AM, FeepingCreature wrote:
 On Monday, 28 January 2019 at 15:16:54 UTC, Steven Schveighoffer wrote:
 It will inspect the allocated length from the GC if the array is 
 appendable from the beginning. So it's not always going to reallocate.

 e.g.:

 string x = "abc".idup;

 auto app = x.appender;

 app ~= "xyz"; // does not reallocate.
Fair enough.
Wow, I take this back. I tried it, and it DOES reallocate. In fact, appender when given data that's not mutable avoids the check for appendability, and extension into the block. So indeed, using appender on a string that is appendable reallocates on first append. I agree with you now, Appender on an existing string is near useless. Looks like there were some changes that fix one problem, but may have inadvertently created this problem. And I'm rereading my comments in this PR, finding that I was not exactly correct on my analysis: https://github.com/dlang/phobos/pull/2046 In short, there's a question of purity, which was already on the Appender's functions, causing issue with immutable data. I think this won't be strong pure anyway, because appender's functions are not strong-pure (there's always mutable data inside it). I think we should avoid the mutability check in that call.
 My problem is this.
 
 const and immutable are *not* well supported in the standard library at 
 the moment. What I want is that I can use the idioms of the stdlib, 
 semantically, in a way that lets me *not care* about const or immutable, 
 that lets me express the patterns I want without having to worry about 
 whether some type deep inside my code, a dub library, or phobos itself 
 decided to declare a field immutable.
 
 Appender covers two usecases: "capacity caching" and "memory reuse." 
 Both of those usecases have two const variations: "possibly immutable" 
 and "mutable".
 
    mutable capacity caching | mutable memory reuse
 ---------------------------+---------------------
 immutable capacity caching | immutable memory reuse
 
 But instead of cutting between capacity caching and memory reuse, down 
 the middle, Appender cuts between mutable and immutable, left to right. 
 I think this is a symptom of a broad negligence of constness as a 
 first-class property - constness works sometimes, and it's nice if it 
 does, but it can't be *relied* on to be well supported. This makes using 
 const in D an eternal uphill struggle. Why even go to all the trouble to 
 make a new major version of the language just to introduce constness, if 
 it's not going to be treated as a first-class concern? I don't want to 
 have to sprinkle static if(isAssignable!T) everytime I want to use a 
 library type. If Phobos offers me as generic a data structure as 
 Appender, parameterized on T, it should work for T, regardless of what T 
 is doing, and *certainly* regardless of what fields in T are marked 
 const. Especially considering that if a field is not marked as const, 
 that hardly means it's not going to lead to bugs if its memory is reused 
 while you're referencing it!
Appender covers appending, which works for both mutable and immutable data. The clear function, if called on immutable data, would allow for immutable data to be overwritten, which is undefined behavior. This should be harder to do than just calling a member function. Note that if we fix the above problem, there is a workaround: app = app.data[0 .. 0].assumeSafeAppend.appender; Which includes the greppable assumeSafeAppend property. The biggest problem I see is not necessarily that immutable arrays should be clearable, but that arrays of types with const items should be clearable. I can't say I agree with immutable pieces, as again, that leads to potential UB. I know it's a PITA, but we have to consider as a library type all possible usages, not just focused ones. I can see a PR that allows clear for types that are not assignable, but only have const data, being accepted. Const doesn't guarantee no mutability, so it's not UB to modify data that in some other alias has a const reference. I don't know if we have a way to determine this separate from immutability. -Steve
Jan 30
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/24/19 7:35 AM, FeepingCreature wrote:
 On Tuesday, 27 March 2018 at 12:31:05 UTC, Simen Kjærås wrote:
 On Tuesday, 27 March 2018 at 12:17:58 UTC, Ellie Harper wrote:
 Sorry if this is a stupid question, but is there something special 
 required to call Appender.clear?  When I attempt even just a simple 
 use I am getting compile errors relating to `template object.clear`.
From the documentation for Appender:
 Note
 clear is disabled for immutable or const element types, due to the 
 possibility that Appender might overwrite immutable data.
Since string is immutable(char)[], clear() is simply not available for appender!string.
Isn't this wrong, though? Appender controls the memory it references. It could just choose to allocate non-immutable memory internally. As long as any const data put into the appender is *returned* as const, there is no chance of immutable memory being overwritten.
Appender is sometimes given a starting array. clear isn't callable in that case, and we don't distinguish the difference in the type or at runtime. I did a lot of work on std.array.Appender when I redid the array runtime to fix array appending back in 2010 to make it safe. e.g.: https://github.com/schveiguy/phobos/commit/7e89201cdae96e71e550afe72b59ef3db145916f Disabling clear from appender for non-mutable element types was one of the changes: https://github.com/schveiguy/phobos/commit/6636569318d26545bf7c36ddac029830a6d52531 This is before we were on github, so I don't know if there's any discussion of the issues, perhaps on the mailing list, but looking at my emails around that time, there are some emails that aren't in the mailing list. A funny thing I wrote to Walter/Sean around that time, when I was just getting druntime commit access: "I don't mind helping out in druntime, but I'm not really any kind of expert on the runtime, and I doubt I'll make many contributions besides this one without assignment." ;) FYI, this thread is almost a year old. -Steve
Jan 24