digitalmars.D.learn - Autodecode in the wild and An Awful Hack to std.regex
- John Carter (40/40) Jul 27 2016 Don't you just hate it when you google a problem and find a post
- Kagamin (3/3) Jul 28 2016 Create an RFE? Given that regex returns results as slices of the
- Lodovico Giaretta (7/10) Jul 28 2016 (RFE = Request For Enhancement, right?)
- Kagamin (4/4) Jul 28 2016 A template parameter is usually needed when it affects the output
- Seb (4/7) Jul 28 2016 We call them DIP (D Improvement Proposals) and I think it's a lot
- John Carter (14/17) Jul 28 2016 Eh. I hoped that somewhere in that explosion of discussion on the
- Joakim (3/18) Aug 01 2016 I suggest you talk to Dmitry, who wrote std.regex, as he will be
Don't you just hate it when you google a problem and find a post from yourself asking the same question? In 2013 I ran into the UTF8 invalid char autodecode UTFException, and the answer then was "use std.encoding.sanitize" and my opinion looking at the implementation, was then, as is now... Eww! Since then, I'm glad to see Walter Bright agrees that autodecode is problematic. http://forum.dlang.org/thread/nh2o9i$hr0$1 digitalmars.com After wading through 46 pages of that, and Jack Stouffers handy blog entry and the longish discussion thread on it... https://forum.dlang.org/post/eozguhavggchzzruzkwk forum.dlang.org Maybe I missed something. What am I supposed to do here in 2016? I was happily churning through 25 gigabytes of data with foreach( line; File( file).byLine()) { auto c = line.matchFirst( myRegex); . . When I hit an invalid codepoint...Again. What is an efficient (elegant) solution? An inelegant solution was to hack into the point that throws the exception and "Do It Right" (for various values of Right) diff -u /usr/include/dmd/phobos/std/regex/internal/ir.d{~,} --- /usr/include/dmd/phobos/std/regex/internal/ir.d~ 2015-12-03 14:41:31.000000000 +1300 +++ /usr/include/dmd/phobos/std/regex/internal/ir.d 2016-07-28 11:04:55.525480585 +1200 -591,7 +591,7 pos = _index; if(_index == _origin.length) return false; - res = std.utf.decode(_origin, _index); + res = std.utf.decode!(UseReplacementDchar.yes)(_origin, _index); return true; } property bool atEnd(){ That "Works For Me". But it vaguely feels to me that that template parameter needs to be trickled all the way up the regex engine.
Jul 27 2016
Create an RFE? Given that regex returns results as slices of the input string, using the replacement character doesn't introduce data corruption.
Jul 28 2016
On Thursday, 28 July 2016 at 09:10:33 UTC, Kagamin wrote:Create an RFE? Given that regex returns results as slices of the input string, using the replacement character doesn't introduce data corruption.(RFE = Request For Enhancement, right?) Yes, all algorithms that use decode internally shall provide a template parameter to useReplacementDchar. There's no reason not to expose this option, given that it allows not to brutally abort any computation in such situations, and also has the bonus point of making decode nogc.
Jul 28 2016
A template parameter is usually needed when it affects the output data, but in case of regex it won't do much, because the output data are slices of the input string, so decoding doesn't affect them, only exceptions.
Jul 28 2016
On Thursday, 28 July 2016 at 09:10:33 UTC, Kagamin wrote:Create an RFE? Given that regex returns results as slices of the input string, using the replacement character doesn't introduce data corruption.We call them DIP (D Improvement Proposals) and I think it's a lot more productive way to discuss improvements than in the forum. For more info see: https://github.com/dlang/DIPs
Jul 28 2016
On Thursday, 28 July 2016 at 15:48:58 UTC, Seb wrote:We call them DIP (D Improvement Proposals) and I think it's a lot more productive way to discuss improvements than in the forum.Eh. I hoped that somewhere in that explosion of discussion on the topic the problem had been solved and I had just missed it and merely had to use that. Also this idea is a bit immature for a DIP... I haven't look at the regex code beyond the the stack trace it died on. ie. * Would this even be a Good Idea for a dip or is it better solve by another existing means? * I only inspected and changed one occurrence of decode (the one that broke) is there any other route in the regex engine that could throw a UTFException? * Would adding an additional template parameter with default break existing code? Or would I have to provide a shim?
Jul 28 2016
On Thursday, 28 July 2016 at 21:02:59 UTC, John Carter wrote:On Thursday, 28 July 2016 at 15:48:58 UTC, Seb wrote:I suggest you talk to Dmitry, who wrote std.regex, as he will be motivated to look into this.[...]Eh. I hoped that somewhere in that explosion of discussion on the topic the problem had been solved and I had just missed it and merely had to use that. Also this idea is a bit immature for a DIP... I haven't look at the regex code beyond the the stack trace it died on. ie. * Would this even be a Good Idea for a dip or is it better solve by another existing means? * I only inspected and changed one occurrence of decode (the one that broke) is there any other route in the regex engine that could throw a UTFException? * Would adding an additional template parameter with default break existing code? Or would I have to provide a shim?
Aug 01 2016