www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - iopipe code to count lines in gzipped file works with v 0.1.7 but

reply Andrew <aabrown24 hotmail.com> writes:
Hi,

This code to count lines in a gzipped file exits with "Program 
exited with code -9" when run with the latest version of the 
library, I guess because I am doing unsafe things. Could someone 
tell me how to change it to make it work? The actual program I'm 
writing processes a file line by line, so ideally I'd like to 
keep the structure of open a file, then foreach over it.

Thanks very much

Andrew

/+ dub.sdl:
	name "hello"
	dependency "iopipe" version="~>0.2.0"
+/

import std.stdio;
import std.typecons;
import iopipe.textpipe;
import iopipe.zip;
import iopipe.bufpipe;
import std.io : File = File;

void main()
{
   auto counter = 0;
   auto fileToRead = 
File("file.gz").refCounted.bufd.unzip(CompressionFormat.gzip);

   foreach (line; fileToRead.assumeText.byLineRange!false)
   {
     counter++;
   }

   writeln(counter);
}
Aug 07 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/7/20 9:40 AM, Andrew wrote:

 Hi,
 
 This code to count lines in a gzipped file exits with "Program exited 
 with code -9" when run with the latest version of the library, I guess 
 because I am doing unsafe things. Could someone tell me how to change it 
 to make it work? The actual program I'm writing processes a file line by 
 line, so ideally I'd like to keep the structure of open a file, then 
 foreach over it.
 
 Thanks very much


As of iopipe v0.2.0, io is no longer a required dependency, it's 
optional. So you must also add a dependency for io.

I tried adding

dependency "io" version="~>0.3.0"

But it fails with:

Got no configuration for dependency io ~>0.3.1 of hello ~master!?

If I add

dependency "io" version="*"

it works.

I think this is an issue with dub when using an inline recipe file, but 
I don't know?

Note that in this simple example, the line count is stored in the line 
pipe, you can retreive the number of lines by accessing the `segments` 
member of the pipe (undocumented, I have to fix that). So my code looks 
like:

---
/+ dub.sdl:
     name "hello"
     dependency "iopipe" version="~>0.2.0"
     dependency "io" version="*"
+/

import std.stdio;
//import std.typecons; // refCounted not  safe
import iopipe.textpipe;
import iopipe.zip;
import iopipe.bufpipe;
import iopipe.refc; // refCounted that is  safe
import std.io : File = File; // just a note, I don't know why you are 
renaming here...

void main()  safe // yay  safe!
{
   auto counter = 0;
   auto fileToRead = File("file.gz").refCounted.bufd
       .unzip(CompressionFormat.gzip)
       .assumeText
       .byLine;

   fileToRead.process();
   writeln(fileToRead.segments);
}
---

FYI, I noticed that in my simple test, this outputs one less than the 
actual lines. I'll have to look into *that* too.

That dependency on writeln also irks me ;) I need to get working on that 
iopipe replacement for it...

-Steve
Aug 07 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/7/20 8:57 PM, Steven Schveighoffer wrote:

 I tried adding
 
 dependency "io" version="~>0.3.0"
 
 But it fails with:
 
 Got no configuration for dependency io ~>0.3.1 of hello ~master!?
 
 If I add
 
 dependency "io" version="*"
 
 it works.
 
 I think this is an issue with dub when using an inline recipe file, but 
 I don't know?


ugh. This is an issue with iopipe specifying io version 0.2.x. I will 
fix this.

-Steve
Aug 07 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/7/20 9:31 PM, Steven Schveighoffer wrote:

 On 8/7/20 8:57 PM, Steven Schveighoffer wrote:

 I think this is an issue with dub when using an inline recipe file, 
 but I don't know?


 
 ugh. This is an issue with iopipe specifying io version 0.2.x. I will 
 fix this.


OK, iopipe 0.2.2 is released, with no specific io dependency.

It should work with io 0.3.1 as a separate listed dependency.

-Steve
Aug 07 2020
parent Andrew <aabrown24 hotmail.com> writes:
On Saturday, 8 August 2020 at 02:06:36 UTC, Steven Schveighoffer 
wrote:

 On 8/7/20 9:31 PM, Steven Schveighoffer wrote:

 On 8/7/20 8:57 PM, Steven Schveighoffer wrote:

 I think this is an issue with dub when using an inline recipe 
 file, but I don't know?


 
 ugh. This is an issue with iopipe specifying io version 0.2.x. 
 I will fix this.


 OK, iopipe 0.2.2 is released, with no specific io dependency.

 It should work with io 0.3.1 as a separate listed dependency.

 -Steve


Thank you for looking into this. I assumed the problem was at my 
end, as you have actually saved my bacon before, processing 
exactly the same file, the problem being that it was a blocked 
gzip file 
(ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr22.phase3_shapeit2_mvncall_integrated_v5a.201305
2.genotypes.vcf.gz) I will raise an issue, as I think io is being imported and
linked ok.
Aug 08 2020
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/7/20 9:40 AM, Andrew wrote:

 Hi,
 
 This code to count lines in a gzipped file exits with "Program exited 
 with code -9" when run with the latest version of the library, I guess 
 because I am doing unsafe things.


BTW the safety improvements only change whether it compiles as  safe or not.

If it's building but running is exiting with a code then it's possible 
there's a bug somewhere.

I did have to change a lot of code to get it to build properly.

If you have it building, but it's exiting with an error, then please 
file an issue with an example file and sample source that causes the issue.

-Steve
Aug 07 2020