www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - About File.rawWrite

reply bearophile <bearophileHUGS lycos.com> writes:
This D2 program runs in about 5.13 seconds on my PC:


import std.stdio;
void main() {
    auto f = File("bytes_test.dat", "wb");
    ubyte[3] RGB;
    foreach (_; 0 .. 1_000_000)
        f.rawWrite(RGB);
}



While this C program runs in about 0.14 seconds:


#include <stdio.h>
int main() {
    FILE *f = fopen("bytes_test.dat", "wb");
    unsigned char RGB[3] = {0};
    int i;
    for (i = 0; i < 1000000; i++)
        fwrite(RGB, 1, 3, f);
    return 0;
}


Is my D2 program wrong, or is File.rawWrite in need of some improvements?

(Writing 3 bytes at a time is not efficient, but the C code shows that the
runtime is acceptable for me for small files).

Bye,
bearophile
Nov 29 2011
parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
29.11.2011 15:57, bearophile :

 This D2 program runs in about 5.13 seconds on my PC:


 import std.stdio;
 void main() {
      auto f = File("bytes_test.dat", "wb");
      ubyte[3] RGB;
      foreach (_; 0 .. 1_000_000)
          f.rawWrite(RGB);
 }



 While this C program runs in about 0.14 seconds:


 #include<stdio.h>
 int main() {
      FILE *f = fopen("bytes_test.dat", "wb");
      unsigned char RGB[3] = {0};
      int i;
      for (i = 0; i<  1000000; i++)
          fwrite(RGB, 1, 3, f);
      return 0;
 }


 Is my D2 program wrong, or is File.rawWrite in need of some improvements?

 (Writing 3 bytes at a time is not efficient, but the C code shows that the
runtime is acceptable for me for small files).

 Bye,
 bearophile


Your OS is Windows, right? On Windows, rawWrite and rawRead always 
flushes stream, sets binary mode, reads/writes, flushes stream again, 
sets previous mode. This is definitely unnecessary slow - at least it 
should change mode only if needed (the file is opened in a text mode). 
The better solution is to change all other functions so the file mode 
will be changed lazily (for a text file, raw* functions will change file 
mode and leave file in this mode until a call of a function, that really 
needs a file to be in a text mode).

By the way, why this changing mode behaviour is Windows only? Yes, it is 
clearly documented that this is the case, but it isn't documented _why_ 
this is the case.

Quick link to the current rawRead implementation (for happy owners of 
Google's Chrome, the fastest beautiful web browser that can BSOD your 
Windows even without administrator rights) (the link isn't for Firefox: 
it will slow down the browser and be displayed incorrectly):
https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L458
Nov 29 2011
next sibling parent reply torhu <no spam.invalid> writes:
On 29.11.2011 16:00, Denis Shelomovskij wrote:

 Your OS is Windows, right? On Windows, rawWrite and rawRead always
 flushes stream, sets binary mode, reads/writes, flushes stream again,
 sets previous mode. This is definitely unnecessary slow - at least it
 should change mode only if needed (the file is opened in a text mode).
 The better solution is to change all other functions so the file mode
 will be changed lazily (for a text file, raw* functions will change file
 mode and leave file in this mode until a call of a function, that really
 needs a file to be in a text mode).

 By the way, why this changing mode behaviour is Windows only? Yes, it is
 clearly documented that this is the case, but it isn't documented _why_
 this is the case.


Text mode means that \n in strings is translated to \r\n when writing, 
and the other way when reading. That way you can use only LF (\n) to get 
linefeeds on all platforms, even if Windows uses CRLF.
Nov 29 2011
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 11/29/2011 01:20 PM, torhu wrote:

 On 29.11.2011 16:00, Denis Shelomovskij wrote:

 Your OS is Windows, right? On Windows, rawWrite and rawRead always
 flushes stream, sets binary mode, reads/writes, flushes stream again,
 sets previous mode. This is definitely unnecessary slow - at least it
 should change mode only if needed (the file is opened in a text mode).
 The better solution is to change all other functions so the file mode
 will be changed lazily (for a text file, raw* functions will change file
 mode and leave file in this mode until a call of a function, that really
 needs a file to be in a text mode).

 By the way, why this changing mode behaviour is Windows only? Yes, it is
 clearly documented that this is the case, but it isn't documented _why_
 this is the case.


 Text mode means that \n in strings is translated to \r\n when writing,
 and the other way when reading. That way you can use only LF (\n) to get
 linefeeds on all platforms, even if Windows uses CRLF.


The "has no effect" part in the following paragraph has been news to me. 
 From 'man fopen' on my Lubuntu 11 based system:

<quote>
The  mode string can also include the letter 'b' either as a last char‐
acter or as a character between the characters in any of the  two-char‐
acter strings described above.  This is strictly for compatibility with
C89 and has no effect; the 'b' is ignored on all POSIX conforming  sys‐
tems,  including Linux.  (Other systems may treat text files and binary
files differently, and adding the 'b' may be a good idea if you do  I/O
to a binary file and expect that your program may be ported to non-Unix
environments.)
</quote>

Ali
Nov 29 2011
prev sibling parent Kai Meyer <kai unixlords.com> writes:
On 11/29/2011 08:00 AM, Denis Shelomovskij wrote:

 29.11.2011 15:57, bearophile :

 This D2 program runs in about 5.13 seconds on my PC:


 import std.stdio;
 void main() {
 auto f = File("bytes_test.dat", "wb");
 ubyte[3] RGB;
 foreach (_; 0 .. 1_000_000)
 f.rawWrite(RGB);
 }



 While this C program runs in about 0.14 seconds:


 #include<stdio.h>
 int main() {
 FILE *f = fopen("bytes_test.dat", "wb");
 unsigned char RGB[3] = {0};
 int i;
 for (i = 0; i< 1000000; i++)
 fwrite(RGB, 1, 3, f);
 return 0;
 }


 Is my D2 program wrong, or is File.rawWrite in need of some improvements?

 (Writing 3 bytes at a time is not efficient, but the C code shows that
 the runtime is acceptable for me for small files).

 Bye,
 bearophile


 Your OS is Windows, right? On Windows, rawWrite and rawRead always
 flushes stream, sets binary mode, reads/writes, flushes stream again,
 sets previous mode. This is definitely unnecessary slow - at least it
 should change mode only if needed (the file is opened in a text mode).
 The better solution is to change all other functions so the file mode
 will be changed lazily (for a text file, raw* functions will change file
 mode and leave file in this mode until a call of a function, that really
 needs a file to be in a text mode).

 By the way, why this changing mode behaviour is Windows only? Yes, it is
 clearly documented that this is the case, but it isn't documented _why_
 this is the case.

 Quick link to the current rawRead implementation (for happy owners of
 Google's Chrome, the fastest beautiful web browser that can BSOD your
 Windows even without administrator rights) (the link isn't for Firefox:
 it will slow down the browser and be displayed incorrectly):
 https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L458


My firefox handled the link just fine. Ofcourse my dev machine is an 8 
core i7 with 16 GB of ram.....

-Kai Meyer
Nov 29 2011