digitalmars.D.learn - Bug in csv or byLine ?
- Guillaume Chatelet (44/44) Jan 08 2016 $ cat debug.csv
- Tobi G. (8/9) Jan 08 2016 No, sorry. Under Windows DMD v2.069.2 it works perfectly in both
- Guillaume Chatelet (27/32) Jan 08 2016 - DMD64 D Compiler v2.069.2 on Linux.
- Tobi G. (8/15) Jan 08 2016 I ran it now under Linux/Ubuntu DMD64 D Compiler v2.069.2
- Guillaume Chatelet (6/22) Jan 08 2016 Indeed there's an empty line at the end of the csv.
- Keywan Ghadami (10/36) Jan 10 2016 I still do not understand half of the syntax(still learning) but
- Tobi G. (20/30) Jan 10 2016 Yes. It looks like there is no case to handle '\r' - line endings
- Tobi G. (1/1) Jan 10 2016 The bug has been fixed...
- Jesse Phillips (2/3) Jan 10 2016 Do you have a link for the fix? Is there a BugZilla entry?
- Tobi G. (6/9) Jan 10 2016 Yes sure..
- Guillaume Chatelet (2/13) Jan 11 2016 Thx for the fix !
$ cat debug.csv
timestamp,curr_property
2015-12-01 06:07:55,7035
----------------------------------------------------------------
$ cat process.d
import std.stdio;
import std.csv;
import std.algorithm;
import std.file;
void main(string[] args) {
version (Fail) {
File(args[1], "r").byLine.joiner("\n").csvReader.each!writeln;
} else {
readText(args[1]).csvReader.each!writeln;
}
}
----------------------------------------------------------------
$ dmd -run ./process.d debug.csv
["timestamp", "curr_property"]
["2015-12-01 06:07:55", "7035"]
----------------------------------------------------------------
$ dmd -version=Fail -run ./process.d debug.csv
["timestamp", "curr_property"]
["2015-12-01 06:07:55", "7035"]
core.exception.AssertError std/algorithm/iteration.d(2027):
Assertion failure
----------------
??:? _d_assert [0x4633d3]
??:? void std.algorithm.iteration.__assert(int) [0x46d770]
??:? pure property safe dchar
std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char,
char).ByLine,
immutable(char)[]).joiner(std.stdio.File.ByLine!(char,
char).ByLine, immutable(char)[]).Result.front() [0x44eaf0]
??:? void std.csv.CsvReader!(immutable(char)[], 1,
std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char,
char).ByLine,
immutable(char)[]).joiner(std.stdio.File.ByLine!(char,
char).ByLine, immutable(char)[]).Result, dchar,
immutable(char)[][]).CsvReader.popFront() [0x44f7fc]
??:? void
std.algorithm.iteration.__T4eachS183std5stdio7writelnZ.each!(std.csv.CsvReader
(immutable(char)[], 1,
std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine,
immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine,
immutable(char)[]).Result, dchar,
immutable(char)[][]).CsvReader).each(std.csv.CsvReader!(immutable(char)[], 1,
std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine,
immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine,
immutable(char)[]).Result, dchar, immutable(char)[][]).CsvReader) [0x4608f7]
??:? _Dmain [0x44bc93]
Any idea ?
Jan 08 2016
On Friday, 8 January 2016 at 09:59:26 UTC, Guillaume Chatelet wrote:Any idea ?No, sorry. Under Windows DMD v2.069.2 it works perfectly in both cases. Which compiler do you use? You could run DMD with the -g option. This will print often more useful output, if it fails. togrue
Jan 08 2016
On Friday, 8 January 2016 at 12:07:05 UTC, Tobi G. wrote:No, sorry. Under Windows DMD v2.069.2 it works perfectly in both cases. Which compiler do you use?- DMD64 D Compiler v2.069.2 on Linux. - LDC 0.16.1 (DMD v2.067.1, LLVM 3.7.0) So if it works on windows I guess it's a problem with the File implementation.You could run DMD with the -g option. This will print often more useful output, if it fails.-g didn't bring much. core.exception.AssertError std/algorithm/iteration.d(2027): Assertion failure ---------------- ??:? _d_assert [0x4a9c33] ??:? void std.algorithm.iteration.__assert(int) [0x4b8048] /usr/include/dmd/phobos/std/algorithm/iteration.d:2027 pure property safe dchar std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).Result.front() [0x495330] /usr/include/dmd/phobos/std/csv.d:1018 void std.csv.CsvReader!(immutable(char)[], 1, std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).Result, dchar, immutable(char)[][]).CsvReader.popFront() [0x49608c] /usr/include/dmd/phobos/std/algorithm/iteration.d:881 void std.algorithm.iteration.__T4eachS183std5stdio7writelnZ.each!(std.csv.CsvReader (immutable(char)[], 1, std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).Result, dchar, immutable(char)[][]).CsvReader).each(std.csv.CsvReader!(immutable(char)[], 1, std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).Result, dchar, immutable(char)[][]).CsvReader) [0x4a5063] ./process.d:8 _Dmain [0x49226c]
Jan 08 2016
On Friday, 8 January 2016 at 12:13:59 UTC, Guillaume Chatelet wrote:On Friday, 8 January 2016 at 12:07:05 UTC, Tobi G. wrote:I ran it now under Linux/Ubuntu DMD64 D Compiler v2.069.2 But both still worked.. Are there some characters in your input data which are invalid and not displayed in the forum? (multiple empty lines after the actual csv data for example) togrueNo, sorry. Under Windows DMD v2.069.2 it works perfectly in both cases. Which compiler do you use?- DMD64 D Compiler v2.069.2 on Linux. - LDC 0.16.1 (DMD v2.067.1, LLVM 3.7.0)
Jan 08 2016
On Friday, 8 January 2016 at 13:22:40 UTC, Tobi G. wrote:On Friday, 8 January 2016 at 12:13:59 UTC, Guillaume Chatelet wrote:Indeed there's an empty line at the end of the csv. Interestingly enough if I try with DMD64 D Compiler v2.069, the Fail version runs fine but the normal version returns: std.csv.CSVException /usr/include/dlang/dmd/std/csv.d(1246): Row 3's length 1 does not match previous length of 2.On Friday, 8 January 2016 at 12:07:05 UTC, Tobi G. wrote:I ran it now under Linux/Ubuntu DMD64 D Compiler v2.069.2 But both still worked.. Are there some characters in your input data which are invalid and not displayed in the forum? (multiple empty lines after the actual csv data for example) togrueNo, sorry. Under Windows DMD v2.069.2 it works perfectly in both cases. Which compiler do you use?- DMD64 D Compiler v2.069.2 on Linux. - LDC 0.16.1 (DMD v2.067.1, LLVM 3.7.0)
Jan 08 2016
On Friday, 8 January 2016 at 13:53:06 UTC, Guillaume Chatelet wrote:On Friday, 8 January 2016 at 13:22:40 UTC, Tobi G. wrote:I still do not understand half of the syntax(still learning) but my guess is that it is a bug in the csv reader: In https://github.com/D-Programming-Language/phobos/blob/67c95e6de21d5d627e3c57128b4d6e 32c82f785/std/csv.d line 1020 calls popfront on the input without checking that it input is empty. But this only happen in special case if the last line is \r\n if i understand it correctly. I might be totally wrong because it does not match your stackstrace and i did not yet understand everything about this.On Friday, 8 January 2016 at 12:13:59 UTC, Guillaume Chatelet wrote:Indeed there's an empty line at the end of the csv. Interestingly enough if I try with DMD64 D Compiler v2.069, the Fail version runs fine but the normal version returns: std.csv.CSVException /usr/include/dlang/dmd/std/csv.d(1246): Row 3's length 1 does not match previous length of 2.On Friday, 8 January 2016 at 12:07:05 UTC, Tobi G. wrote:I ran it now under Linux/Ubuntu DMD64 D Compiler v2.069.2 But both still worked.. Are there some characters in your input data which are invalid and not displayed in the forum? (multiple empty lines after the actual csv data for example) togrueNo, sorry. Under Windows DMD v2.069.2 it works perfectly in both cases. Which compiler do you use?- DMD64 D Compiler v2.069.2 on Linux. - LDC 0.16.1 (DMD v2.067.1, LLVM 3.7.0)
Jan 10 2016
On Sunday, 10 January 2016 at 09:41:16 UTC, Keywan Ghadami wrote:On Friday, 8 January 2016 at 13:53:06 UTC, Guillaume Chatelet wrote: I still do not understand half of the syntax(still learning) but my guess is that it is a bug in the csv reader: In https://github.com/D-Programming-Language/phobos/blob/67c95e6de21d5d627e3c57128b4d6e 32c82f785/std/csv.d line 1020 calls popfront on the input without checking that it input is empty. But this only happen in special case if the last line is \r\n if i understand it correctly. I might be totally wrong because it does not match your stackstrace and i did not yet understand everything about this.Yes. It looks like there is no case to handle '\r' - line endings correctly. https://github.com/D-Programming-Language/phobos/blob/67c95e6de21d5d627e3c57128b4d6e332c82f785/std/csv.d?L=1020#L1020 If '\r' is the last character it will fail, (because it tryes to read the next character). enum data = "name, surname, age Joe, Joker, 99\r"; // WILL FAIL import std.csv, std.stdio; void main(string[] args) { auto reader = csvReader(data); foreach(entry; reader) { writeln(entry); } } togrue
Jan 10 2016
On Sunday, 10 January 2016 at 18:09:23 UTC, Tobi G. wrote:The bug has been fixed...Do you have a link for the fix? Is there a BugZilla entry?
Jan 10 2016
On Sunday, 10 January 2016 at 19:07:52 UTC, Jesse Phillips wrote:On Sunday, 10 January 2016 at 18:09:23 UTC, Tobi G. wrote:Yes sure.. https://issues.dlang.org/show_bug.cgi?id=15545 and the fix at github https://github.com/D-Programming-Language/phobos/pull/3917 togrueThe bug has been fixed...Do you have a link for the fix? Is there a BugZilla entry?
Jan 10 2016
Guillaume Chatelet <chatelet.guillaume gmail.com> On Sunday, 10 January 2016 at 19:50:15 UTC, Tobi G. wrote:On Sunday, 10 January 2016 at 19:07:52 UTC, Jesse Phillips wrote:Thx for the fix !On Sunday, 10 January 2016 at 18:09:23 UTC, Tobi G. wrote:Yes sure.. https://issues.dlang.org/show_bug.cgi?id=15545 and the fix at github https://github.com/D-Programming-Language/phobos/pull/3917 togrueThe bug has been fixed...Do you have a link for the fix? Is there a BugZilla entry?
Jan 11 2016