digitalmars.D.learn - What type does byGrapheme() return?
- =?iso-8859-1?Q?Robert_M._M=FCnch?= (13/13) Dec 27 2019 I love these documentation lines in the D docs:
- Steven Schveighoffer (19/34) Dec 27 2019 This is the rub with ranges. You need to use typeof. There's no other
- =?iso-8859-1?Q?Robert_M._M=FCnch?= (13/31) Dec 29 2019 Hi, ok, thanks a lot and IMO these are the fundamental important
- H. S. Teoh (16/30) Dec 27 2019 [...]
- =?iso-8859-1?Q?Robert_M._M=FCnch?= (26/41) Dec 29 2019 Hi, my idea was that when I use a grapheme range, it will abstract away
- H. S. Teoh (17/46) Dec 30 2019 [...]
- H. S. Teoh (14/16) Dec 30 2019 [...]
- Steven Schveighoffer (8/26) Dec 31 2019 Then the original example should be fixable by putting "ref" in for all
- H. S. Teoh (14/35) Dec 31 2019 [...]
- Steven Schveighoffer (4/12) Dec 31 2019 Huh, it seemed to work for me. Got the full "Robert" with an R. map does...
- H. S. Teoh (17/29) Dec 31 2019 Here's my full non-working code:
- Steven Schveighoffer (10/41) Dec 31 2019 auto r = x.byGrapheme.array;
- H. S. Teoh (22/37) Dec 31 2019 [...]
- =?iso-8859-1?Q?Robert_M._M=FCnch?= (13/18) Jan 04 2020 Thanks for all the answers, pretty enlighting even I'm not sure I get
- H. S. Teoh (13/30) Jan 04 2020 At a minimum, I think we should file a bug report to investigate whether
- =?iso-8859-1?Q?Robert_M._M=FCnch?= (7/12) Jan 06 2020 Done... my first bug report :-) I copied togehter all the findings from
- Alex (3/15) Jan 06 2020 For the sake of completeness
- H. S. Teoh (12/30) Dec 30 2019 [...]
- Alexandru Ermicioi (12/13) Dec 30 2019 There are set of range interfaces that can be used to mask range
I love these documentation lines in the D docs: auto byGrapheme(Range)(Range range) How should I know what auto is? Why not write the explicit type so that I know what to expect? When declaring a variable as class/struct member I can't use auto but need the explicit type... I used typeof() but that doesn't help a lot: gText = [Grapheme(53, 0, 0, 72057594037927936, [83, ...., 1)]Result!string I want to iterate a string byGrapheme so that I can add, delete, change graphemes. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Dec 27 2019
On 12/27/19 12:26 PM, Robert M. Münch wrote:I love these documentation lines in the D docs:     auto byGrapheme(Range)(Range range) How should I know what auto is? Why not write the explicit type so that I know what to expect? When declaring a variable as class/struct member I can't use auto but need the explicit type... I used typeof() but that doesn't help a lot: gText = [Grapheme(53, 0, 0, 72057594037927936, [83, ...., 1)]Result!string I want to iterate a string byGrapheme so that I can add, delete, change graphemes.This is the rub with ranges. You need to use typeof. There's no other way to do it, because the type returned by byGrapheme depends on the type of Range. If you know what type Range is, it would be: struct S { typeof(string.init.byGrapheme()) gText; // or alias GRange = typeof(string.init.byGrapheme()); GRange gText; } Subbing in whatever your real range for `string`. Or if it's the result of a bunch of adapters, use the whole call chain with typeof. Why not just declare the real range type? Because it's going to be ugly, especially if your underlying range is the result of other range algorithms. And really, typeof is going to be the better mechanism, even if it's not the best looking thing. -Steve
Dec 27 2019
On 2019-12-27 17:54:28 +0000, Steven Schveighoffer said:This is the rub with ranges. You need to use typeof. There's no other way to do it, because the type returned by byGrapheme depends on the type of Range.Hi, ok, thanks a lot and IMO these are the fundamental important information for people using D (beginners, causual programmers, etc.) to understand how things fit together.If you know what type Range is, it would be: struct S { typeof(string.init.byGrapheme()) gText; // or alias GRange = typeof(string.init.byGrapheme()); GRange gText; }Ah... I didn't know that I can use a basic type "string" combined with ".init" to manually build the type. Neat...Subbing in whatever your real range for `string`. Or if it's the result of a bunch of adapters, use the whole call chain with typeof.Ok, and these are good candidates for alias definitions to avoid re-typing it many times.Why not just declare the real range type? Because it's going to be ugly, especially if your underlying range is the result of other range algorithms. And really, typeof is going to be the better mechanism, even if it's not the best looking thing.I think I got it... thanks a lot. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Dec 29 2019
On Fri, Dec 27, 2019 at 06:26:58PM +0100, Robert M. Münch via Digitalmars-d-learn wrote:I love these documentation lines in the D docs: auto byGrapheme(Range)(Range range) How should I know what auto is? Why not write the explicit type so that I know what to expect? When declaring a variable as class/struct member I can't use auto but need the explicit type... I used typeof() but that doesn't help a lot: gText = [Grapheme(53, 0, 0, 72057594037927936, [83, ...., 1)]Result!string I want to iterate a string byGrapheme so that I can add, delete, change graphemes.[...] Since graphemes are variable-length in terms of code points, you can't exactly *edit* a range of graphemes -- you can't replace a 1-codepoint grapheme with a 6-codepoint grapheme, for example, since there's no space in the underlying string to store that. If you want to add/delete/change graphemes, what you *really* want is to use an array of Graphemes: Grapheme[] editableGraphs; You can then splice it, insert stuff, delete stuff, whatever. When you're done with it, convert it back to string with something like this: string result = editableGraphs.map!(g => g[]).joiner.to!string; T -- The irony is that Bill Gates claims to be making a stable operating system and Linus Torvalds claims to be trying to take over the world. -- Anonymous
Dec 27 2019
On 2019-12-27 19:44:59 +0000, H. S. Teoh said:Since graphemes are variable-length in terms of code points, you can't exactly *edit* a range of graphemes -- you can't replace a 1-codepoint grapheme with a 6-codepoint grapheme, for example, since there's no space in the underlying string to store that.Hi, my idea was that when I use a grapheme range, it will abstract away that graphemes consist of different sized code-points. And the docs at https://dlang.org/phobos/std_uni.html#byGrapheme show an example using this kind of range: auto gText = text.byGrapheme; gText.take(3); gText.drop(3); But maybe I need to get a better understanding of the ranges stuff too...If you want to add/delete/change graphemes, what you *really* want is to use an array of Graphemes: Grapheme[] editableGraphs; You can then splice it, insert stuff, delete stuff, whatever. When you're done with it, convert it back to string with something like this: string result = editableGraphs.map!(g => g[]).joiner.to!string;I played around with this approach... string r1 = "Robert M. Münch"; // Code-Units = 16 // Code-Points = 15 // Graphemes = 15 Grapheme[] gr1 = r1.byGrapheme.array; writeln(" Text = ", gr1.map!(g => g[]).joiner.to!string); // Text = obert M. Münch writeln("wText = ", gr1.map!(g => g[]).joiner.to!wstring); // wText = obert M. Münch writeln("dText = ", gr1.map!(g => g[]).joiner.to!dstring); // dText = obert M. Münch Why is the first letter missing? Is this a bug? -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Dec 29 2019
On Sun, Dec 29, 2019 at 01:19:09PM +0100, Robert M. Münch via Digitalmars-d-learn wrote:On 2019-12-27 19:44:59 +0000, H. S. Teoh said:[...][...] I suspect there's a scope-related bug/issue somewhere here. I did some experiments and discovered that using foreach to iterate over a Grapheme[] is OK, but somehow when using Grapheme[] with .map to slice over each one, I get random UTF-8 encoding errors and missing characters. I suspect the cause is that whatever Grapheme.opSlice returns is going out-of-scope when used with .map, that's why it's malfunctioning. The last time I looked at the Grapheme code, there's a bunch of memory-related stuff involving dtors that's *probably* the cause of this problem. Please file a bug for this. T -- Life is complex. It consists of real and imaginary parts. -- YHLIf you want to add/delete/change graphemes, what you *really* want is to use an array of Graphemes: Grapheme[] editableGraphs; You can then splice it, insert stuff, delete stuff, whatever. When you're done with it, convert it back to string with something like this: string result = editableGraphs.map!(g => g[]).joiner.to!string;I played around with this approach... string r1 = "Robert M. Münch"; // Code-Units = 16 // Code-Points = 15 // Graphemes = 15 Grapheme[] gr1 = r1.byGrapheme.array; writeln(" Text = ", gr1.map!(g => g[]).joiner.to!string); // Text = obert M. Münch writeln("wText = ", gr1.map!(g => g[]).joiner.to!wstring); // wText = obert M. Münch writeln("dText = ", gr1.map!(g => g[]).joiner.to!dstring); // dText = obert M. Münch Why is the first letter missing? Is this a bug?
Dec 30 2019
On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn wrote: [...]I suspect the cause is that whatever Grapheme.opSlice returns is going out-of-scope when used with .map, that's why it's malfunctioning.[...] Haha, it's actually right there in the Grapheme docs for the opSlice overloads: Random-access range over Grapheme's $(CHARACTERS). Warning: Invalidates when this Grapheme leaves the scope, attempts to use it then would lead to memory corruption. Looks like when you use .map over the Grapheme, it gets copied into a temporary, which gets invalidated when map.front returns. Somewhere we're missing a 'scope' qualifier... T -- War doesn't prove who's right, just who's left. -- BSD Games' Fortune
Dec 30 2019
On 12/30/19 6:31 PM, H. S. Teoh wrote:On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn wrote: [...]Then the original example should be fixable by putting "ref" in for all the lambdas. But this is kind of disturbing. Why does the grapheme do this? The original data is not scoped. e.g.: writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string); -SteveI suspect the cause is that whatever Grapheme.opSlice returns is going out-of-scope when used with .map, that's why it's malfunctioning.[...] Haha, it's actually right there in the Grapheme docs for the opSlice overloads: Random-access range over Grapheme's $(CHARACTERS). Warning: Invalidates when this Grapheme leaves the scope, attempts to use it then would lead to memory corruption. Looks like when you use .map over the Grapheme, it gets copied into a temporary, which gets invalidated when map.front returns. Somewhere we're missing a 'scope' qualifier...
Dec 31 2019
On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via Digitalmars-d-learn wrote:On 12/30/19 6:31 PM, H. S. Teoh wrote:[...]On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn wrote:[...]Haha, it's actually right there in the Grapheme docs for the opSlice overloads: Random-access range over Grapheme's $(CHARACTERS). Warning: Invalidates when this Grapheme leaves the scope, attempts to use it then would lead to memory corruption. Looks like when you use .map over the Grapheme, it gets copied into a temporary, which gets invalidated when map.front returns. Somewhere we're missing a 'scope' qualifier...Then the original example should be fixable by putting "ref" in for all the lambdas. But this is kind of disturbing. Why does the grapheme do this? The original data is not scoped.Honestly I have no idea, but glancing at the code in std.uni reveals that the returned slice is actually a wrapper object that contains a pointer to the parent Grapheme object. So if the parent was a temporary and goes out of scope before the wrapper does, we're left with a dangling pointer.e.g.: writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);[...] Unfortunately this doesn't work. Somehow the ref parameter doesn't match whatever it is std.algorithm.map is trying to pass to it. T -- They say that "guns don't kill people, people kill people." Well I think the gun helps. If you just stood there and yelled BANG, I don't think you'd kill too many people. -- Eddie Izzard, Dressed to Kill
Dec 31 2019
On 12/31/19 2:58 PM, H. S. Teoh wrote:On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via Digitalmars-d-learn wrote:Huh, it seemed to work for me. Got the full "Robert" with an R. map does support ref-ness. Maybe you didn't put ref in the right place? -Stevee.g.: writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);[...] Unfortunately this doesn't work. Somehow the ref parameter doesn't match whatever it is std.algorithm.map is trying to pass to it.
Dec 31 2019
On Tue, Dec 31, 2019 at 04:02:47PM -0500, Steven Schveighoffer via Digitalmars-d-learn wrote:On 12/31/19 2:58 PM, H. S. Teoh wrote:Here's my full non-working code: import std; void main() { auto x = "Bla\u0301hbla\u0310h\u0309!"; auto r = x.byGrapheme; writefln("%s", r.map!((ref g) => g[]).joiner.to!string); } The compiler says: /usr/src/d/phobos/std/algorithm/iteration.d(604): Error: template D main.__lambda1 cannot deduce function from argument types !()(Grapheme), candidates are: test.d(5): __lambda1 /usr/src/d/phobos/std/algorithm/iteration.d(499): Error: template instance test.main.MapResult!(__lambda1, Result!string) error instantiating test.d(5): instantiated from here: map!(Result!string) What did I do wrong? T -- What do you get if you drop a piano down a mineshaft? A flat minor.On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via Digitalmars-d-learn wrote:Huh, it seemed to work for me. Got the full "Robert" with an R. map does support ref-ness. Maybe you didn't put ref in the right place?e.g.: writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);[...] Unfortunately this doesn't work. Somehow the ref parameter doesn't match whatever it is std.algorithm.map is trying to pass to it.
Dec 31 2019
On 12/31/19 4:22 PM, H. S. Teoh wrote:On Tue, Dec 31, 2019 at 04:02:47PM -0500, Steven Schveighoffer via Digitalmars-d-learn wrote:auto r = x.byGrapheme.array; This is how Robert originally had it if you look a few messages up. Otherwise, it's not an lvalue. The fact that a Grapheme's return requires you keep the grapheme in scope for operations seems completely incorrect and dangerous IMO (note that operators are going to always have a ref this, even when called on an rvalue). So even though using ref works, I think the underlying issue here really is the lifetime problem. -SteveOn 12/31/19 2:58 PM, H. S. Teoh wrote:Here's my full non-working code: import std; void main() { auto x = "Bla\u0301hbla\u0310h\u0309!"; auto r = x.byGrapheme; writefln("%s", r.map!((ref g) => g[]).joiner.to!string); } The compiler says: /usr/src/d/phobos/std/algorithm/iteration.d(604): Error: template D main.__lambda1 cannot deduce function from argument types !()(Grapheme), candidates are: test.d(5): __lambda1 /usr/src/d/phobos/std/algorithm/iteration.d(499): Error: template instance test.main.MapResult!(__lambda1, Result!string) error instantiating test.d(5): instantiated from here: map!(Result!string) What did I do wrong?On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via Digitalmars-d-learn wrote:Huh, it seemed to work for me. Got the full "Robert" with an R. map does support ref-ness. Maybe you didn't put ref in the right place?e.g.: writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);[...] Unfortunately this doesn't work. Somehow the ref parameter doesn't match whatever it is std.algorithm.map is trying to pass to it.
Dec 31 2019
On Tue, Dec 31, 2019 at 04:36:56PM -0500, Steven Schveighoffer via Digitalmars-d-learn wrote:On 12/31/19 4:22 PM, H. S. Teoh wrote:[...][...]import std; void main() { auto x = "Bla\u0301hbla\u0310h\u0309!"; auto r = x.byGrapheme; writefln("%s", r.map!((ref g) => g[]).joiner.to!string); }Haha, in my hurry I totally forgot about the .array. Mea culpa. [...]What did I do wrong?auto r = x.byGrapheme.array;The fact that a Grapheme's return requires you keep the grapheme in scope for operations seems completely incorrect and dangerous IMO (note that operators are going to always have a ref this, even when called on an rvalue). So even though using ref works, I think the underlying issue here really is the lifetime problem.[...] After my wrong recollection of the history surrounding indexOf vs. countUntil, I'm not sure I can rely on my memory anymore, :-P but AIUI Dmitri implemented it this way because he wanted to avoid allocations (GC or otherwise) in the most common case of Grapheme containing just a small number of code points (usually 1 or 2). When the number of combining diacritics exceed the size of the Grapheme struct, then it would quietly switch to malloc or some such for holding the data. My guess is that this is the reason for passing &this to the wrapper range returned by opSlice(). And possibly it's also to allow mutation of the Grapheme via the returned slice? Perhaps this whole approach should be looked at again. Certainly, unless I'm missing something, it *ought* to be possible to implement Grapheme in a way that doesn't require this scoped reference business. T -- The diminished 7th chord is the most flexible and fear-instilling chord. Use it often, use it unsparingly, to subdue your listeners into submission!
Dec 31 2019
On 2019-12-31 21:36:56 +0000, Steven Schveighoffer said:The fact that a Grapheme's return requires you keep the grapheme in scope for operations seems completely incorrect and dangerous IMO (note that operators are going to always have a ref this, even when called on an rvalue). So even though using ref works, I think the underlying issue here really is the lifetime problem.Thanks for all the answers, pretty enlighting even I'm not sure I get everything 100%. So, what to do for now? File a bug-report? What needs to be fixed? I'm using the ref approach for now, in hope it will be OK for my use-case, which is converting a wstring to a grapheme[], alter the array, and map it back to a wstring. Sounds like a lot of porcessing for handling unicode text, but I don't think it gets a lot simpler than that. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Jan 04 2020
On Sat, Jan 04, 2020 at 08:19:14PM +0100, Robert M. Münch via Digitalmars-d-learn wrote:On 2019-12-31 21:36:56 +0000, Steven Schveighoffer said:At a minimum, I think we should file a bug report to investigate whether Grapheme.opSlice can be implemented differently, such that we avoid this obscure referential behaviour that makes it hard to work with in complex code. I'm not sure if this is possible, but IMO we should at least investigate the possibilities.The fact that a Grapheme's return requires you keep the grapheme in scope for operations seems completely incorrect and dangerous IMO (note that operators are going to always have a ref this, even when called on an rvalue). So even though using ref works, I think the underlying issue here really is the lifetime problem.Thanks for all the answers, pretty enlighting even I'm not sure I get everything 100%. So, what to do for now? File a bug-report? What needs to be fixed?I'm using the ref approach for now, in hope it will be OK for my use-case, which is converting a wstring to a grapheme[], alter the array, and map it back to a wstring. Sounds like a lot of porcessing for handling unicode text, but I don't think it gets a lot simpler than that.[...] Unicode is a beast. Be glad that you can even do this in the first place. If I were writing this in C, I wouldn't even know where to begin! T -- No! I'm not in denial!
Jan 04 2020
On 2020-01-05 04:18:34 +0000, H. S. Teoh said:At a minimum, I think we should file a bug report to investigate whether Grapheme.opSlice can be implemented differently, such that we avoid this obscure referential behaviour that makes it hard to work with in complex code. I'm not sure if this is possible, but IMO we should at least investigate the possibilities.Done... my first bug report :-) I copied togehter all the findings from this thread. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Jan 06 2020
On Monday, 6 January 2020 at 08:39:19 UTC, Robert M. Münch wrote:On 2020-01-05 04:18:34 +0000, H. S. Teoh said:For the sake of completeness https://issues.dlang.org/show_bug.cgi?id=20483At a minimum, I think we should file a bug report to investigate whether Grapheme.opSlice can be implemented differently, such that we avoid this obscure referential behaviour that makes it hard to work with in complex code. I'm not sure if this is possible, but IMO we should at least investigate the possibilities.Done... my first bug report :-) I copied togehter all the findings from this thread.
Jan 06 2020
On Mon, Dec 30, 2019 at 03:31:31PM -0800, H. S. Teoh via Digitalmars-d-learn wrote:On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn wrote: [...][...] Indeed, compiling with dmd -dip1000 produces this error message: test.d(15): Error: returning g.opSlice() escapes a reference to parameter g, perhaps annotate with return /usr/src/d/phobos/std/algorithm/iteration.d(499): instantiated from here: MapResult!(__lambda1, Grapheme[]) test.d(15): instantiated from here: map!(Grapheme[]) Not the most helpful message (the annotation has to go in Phobos code, not in user code), but it does at least point to the cause of the problem. T -- What doesn't kill me makes me stranger.I suspect the cause is that whatever Grapheme.opSlice returns is going out-of-scope when used with .map, that's why it's malfunctioning.[...] Haha, it's actually right there in the Grapheme docs for the opSlice overloads: Random-access range over Grapheme's $(CHARACTERS). Warning: Invalidates when this Grapheme leaves the scope, attempts to use it then would lead to memory corruption. Looks like when you use .map over the Grapheme, it gets copied into a temporary, which gets invalidated when map.front returns. Somewhere we're missing a 'scope' qualifier...
Dec 30 2019
On Friday, 27 December 2019 at 17:26:58 UTC, Robert M. Münch wrote:...There are set of range interfaces that can be used to mask range type. Check for https://dlang.org/library/std/range/interfaces/input_range.html for starting point, and for https://dlang.org/library/std/range/interfaces/input_range_object.html for wrapping any range to those interfaces. Note: resulting wrapped range is an object and has reference semantics, beware of using it directly with other range algorithms as they can consume your range. Best regards, Alexandru.
Dec 30 2019