www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - File I/O performance pitfalls

reply 9898287 <relay.public.adnan outlook.com> writes:
Hi I want to find out what's causing my file writes to be so 
slow. I'm setting up buffer and locking the file and writing them 
to the file.

$ cat d.d && ldc2 -O5  d.d && time ./d >> /dev/null
void main() {
     import std.stdio;
     stdout.setvbuf(4096);
     stdout.lock();

     foreach(i; 0 .. 900_000_000)
         writef("Hello, {}\n", i);
}
real    0m58.790s
user    0m58.545s
sys     0m0.232s

For comparison, I come from the Rust land and here's somewhat 
equivalent code which is consistently faster:

$ cat rust.rs && rustc -C opt-level=3 rust.rs && time ./rust >> 
/dev/null
use std::io::Write;

fn main() {
     let stdout = std::io::stdout();
     let lock = stdout.lock();
     let mut buf = std::io::BufWriter::new(lock);
     for i in 0 .. 900_000_000 {
         buf.write_fmt(format_args!(
             "Hello, {}\n", i
         )).unwrap();
     }
}

real    0m46.502s
user    0m46.263s
sys     0m0.228s

What am I missing?
Oct 25 2019
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 26/10/2019 2:27 PM, 9898287 wrote:
 Hi I want to find out what's causing my file writes to be so slow. I'm 
 setting up buffer and locking the file and writing them to the file.
 
 $ cat d.d && ldc2 -O5  d.d && time ./d >> /dev/null
You probably want -O3 not -O5.
 void main() {
      import std.stdio;
      stdout.setvbuf(4096);
      stdout.lock();
 
      foreach(i; 0 .. 900_000_000)
          writef("Hello, {}\n", i);
I assume what you intended is: writeln("Hello, ", i); Also for format functions in D you should prefer the templated variation as it provides compile time verification of arguments e.g. writef!"Hello, {}\n%d"(i); Please note that {} is not a format specifier. printf style functions (which is what Phobos uses as the basis) for format specifiers begin with a percentage sign.
Oct 25 2019
parent 9898287 <relay.public.adnan outlook.com> writes:
On Saturday, 26 October 2019 at 02:42:04 UTC, rikki cattermole 
wrote:
 On 26/10/2019 2:27 PM, 9898287 wrote:
 [...]
You probably want -O3 not -O5.
  [...]
I assume what you intended is: writeln("Hello, ", i); Also for format functions in D you should prefer the templated variation as it provides compile time verification of arguments e.g. writef!"Hello, {}\n%d"(i); Please note that {} is not a format specifier. printf style functions (which is what Phobos uses as the basis) for format specifiers begin with a percentage sign.
Thank you for pointing that out.
Oct 25 2019