www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.stream very slow

reply "Bruno A. Costa" <bruno codata.com.br> writes:
Hi,

It seems to me that some parts of the std.stream module are working VERY
slow. I've done some tests with file manipulation and some comparissions
with C++ and the results are strange.

ccopy.cc: simple file copy application writen in C++
dcopy.d:  simple file copy application writen in D
text.txt: a text file with ~4MB

Here are the results:

$ time ./ccopy text.txt test1
real    0m0.985s
user    0m0.854s
sys     0m0.047s

Less than one second to copy 4MB.

$ time ./dcopy text.txt test2
real    1m22.886s
user    0m25.014s
sys     0m57.011s
$

More than one minute to copy 4MB.

Here are the sources (both do the same thing: read and write char by char)

/////////////////////////////////////////
// ccopy.cpp
/////////////////////////////////////////
#include <iostream>
#include <fstream>
                                                                               
                                             
void error (const char* p, const char* p2=" ")
{
    std::cerr << p << ' ' << p2 << '\n';
    std::exit(1);
}
                                                                               
                                             
int main ( int argc, char** argv )
{
    if (argc != 3)
        error ("usage: copy SOURCE DESTINATION");
                                                                               
                                             
    std::ifstream origem (argv[1]);
    if (!origem) error ("error openning file ", argv[1]);
                                                                               
                                             
    std::ofstream destino (argv[2]);
    if (!destino) error ("error copying file ", argv[2]);
                                                                               
                                             
    char ch;
    while ( origem.get(ch) ) destino.put(ch);
                                                                               
                                             
    origem.close();
    destino.close();
}

//////////////////////////////////////////
// dcopy.d
//////////////////////////////////////////
import std.stream;
                                                                               
                                             
int main (char[][] args)
{
    if (args.length <= 2) {
        printf ("Usage: dcopy <source> <destiny>\n");
        goto EXIT;
    }
                                                                               
                                             
    char c;
    File source = new File (args[1]);
    File dest = new File ();
    dest.create (args[2]);
                                                                               
                                             
    while (!source.eof()) {
        dest.write (source.getc());
        //printf ("%c", source.getc());
    }
                                                                               
                                             
EXIT:
    return 0;
}


Thanks,

Bruno.
Jun 22 2004
next sibling parent Martin <Martin_member pathlink.com> writes:
I think the problem is source.eof(). You can open the stream.d (in
source/phobos/std) and see for yourself. This eof() uses size() and size() seeks
to the end of stream to find out the length of it. It is quite slow. 
THERE IS A SIMPLE FIX: edit the stream.d a little and change the class Stream.
Make an extra variable called size_at_least. Streams can't shrink they only
grow. (Am I right? Well atleast it should be so with files.) So the function
eof() does not have to call size() every time. It can compare the position first
to the size_at_least and if position is bigger (or equal) then it updates the
size_at_least and compares again, and if position is still bigger (or equal)
then it decides that it really is the end of stream. So 99% of cases it only
uses the variable size_at_least not the function size()

There can be a better fix, but it is the simpliest, I think. 
So maybe WALTER BRIGHT can make this little change in Phobos?
But since phobos is open source, you can do it yourself too. (I mean, if you
need the Stream class to work faster.)

Thank you for pointing this problem out! This is very helpful for developers, I
think.

Greetengs, Martin.

In article <cb99vc$1uur$1 digitaldaemon.com>, Bruno A. Costa says...

Hi,

It seems to me that some parts of the std.stream module are working VERY
slow. I've done some tests with file manipulation and some comparissions
with C++ and the results are strange.

ccopy.cc: simple file copy application writen in C++
dcopy.d:  simple file copy application writen in D
text.txt: a text file with ~4MB

Here are the results:

$ time ./ccopy text.txt test1
real    0m0.985s
user    0m0.854s
sys     0m0.047s

Less than one second to copy 4MB.

$ time ./dcopy text.txt test2
real    1m22.886s
user    0m25.014s
sys     0m57.011s
$

More than one minute to copy 4MB.

Here are the sources (both do the same thing: read and write char by char)

/////////////////////////////////////////
// ccopy.cpp
/////////////////////////////////////////
#include <iostream>
#include <fstream>
                                                                               
                                             
void error (const char* p, const char* p2=" ")
{
    std::cerr << p << ' ' << p2 << '\n';
    std::exit(1);
}
                                                                               
                                             
int main ( int argc, char** argv )
{
    if (argc != 3)
        error ("usage: copy SOURCE DESTINATION");
                                                                               
                                             
    std::ifstream origem (argv[1]);
    if (!origem) error ("error openning file ", argv[1]);
                                                                               
                                             
    std::ofstream destino (argv[2]);
    if (!destino) error ("error copying file ", argv[2]);
                                                                               
                                             
    char ch;
    while ( origem.get(ch) ) destino.put(ch);
                                                                               
                                             
    origem.close();
    destino.close();
}

//////////////////////////////////////////
// dcopy.d
//////////////////////////////////////////
import std.stream;
                                                                               
                                             
int main (char[][] args)
{
    if (args.length <= 2) {
        printf ("Usage: dcopy <source> <destiny>\n");
        goto EXIT;
    }
                                                                               
                                             
    char c;
    File source = new File (args[1]);
    File dest = new File ();
    dest.create (args[2]);
                                                                               
                                             
    while (!source.eof()) {
        dest.write (source.getc());
        //printf ("%c", source.getc());
    }
                                                                               
                                             
EXIT:
    return 0;
}


Thanks,

Bruno.


Martin
Jun 22 2004
prev sibling next sibling parent Arcane Jill <Arcane_member pathlink.com> writes:
std.stream is slow because it's unbuffered. I only discovered this recently, but
you can speed it up DRAMATICALLY by constructing a BufferedStream using a File
as its construction parameter.

This is not documented. Bah.

Arcane Jill
Jun 22 2004
prev sibling next sibling parent Sean Kelly <sean f4.ca> writes:
The standard D streams are non-buffered.  You'll want to use BufferedFileStream
to get better performance.  Beyond that, I'm in the process of working on
stream.d to add some features to it, and I'll look at performance in the
process.  The way buffering is currently done is something I'd like to discuss
once I have a better idea of whether there are alternatives I think may work
better.  None of my changes are guranteed to make it into Phobos, but I'm hoping
that if the community is happy with them then perhaps they will.


Sean
Jun 22 2004
prev sibling next sibling parent Charlie <Charlie_member pathlink.com> writes:
All i can do is laugh at this really, how long has this been a problem ??

You should also checkout Mango's IO system ( http://dsource.org/projects/mango/
) , very clean , and I hear very speedy ( im checking this out myself now ).

Charlie

In article <cb99vc$1uur$1 digitaldaemon.com>, Bruno A. Costa says...

Hi,

It seems to me that some parts of the std.stream module are working VERY
slow. I've done some tests with file manipulation and some comparissions
with C++ and the results are strange.

ccopy.cc: simple file copy application writen in C++
dcopy.d:  simple file copy application writen in D
text.txt: a text file with ~4MB

Here are the results:

$ time ./ccopy text.txt test1
real    0m0.985s
user    0m0.854s
sys     0m0.047s

Less than one second to copy 4MB.

$ time ./dcopy text.txt test2
real    1m22.886s
user    0m25.014s
sys     0m57.011s
$

More than one minute to copy 4MB.

Here are the sources (both do the same thing: read and write char by char)

/////////////////////////////////////////
// ccopy.cpp
/////////////////////////////////////////
#include <iostream>
#include <fstream>
                                                                               
                                             
void error (const char* p, const char* p2=" ")
{
    std::cerr << p << ' ' << p2 << '\n';
    std::exit(1);
}
                                                                               
                                             
int main ( int argc, char** argv )
{
    if (argc != 3)
        error ("usage: copy SOURCE DESTINATION");
                                                                               
                                             
    std::ifstream origem (argv[1]);
    if (!origem) error ("error openning file ", argv[1]);
                                                                               
                                             
    std::ofstream destino (argv[2]);
    if (!destino) error ("error copying file ", argv[2]);
                                                                               
                                             
    char ch;
    while ( origem.get(ch) ) destino.put(ch);
                                                                               
                                             
    origem.close();
    destino.close();
}

//////////////////////////////////////////
// dcopy.d
//////////////////////////////////////////
import std.stream;
                                                                               
                                             
int main (char[][] args)
{
    if (args.length <= 2) {
        printf ("Usage: dcopy <source> <destiny>\n");
        goto EXIT;
    }
                                                                               
                                             
    char c;
    File source = new File (args[1]);
    File dest = new File ();
    dest.create (args[2]);
                                                                               
                                             
    while (!source.eof()) {
        dest.write (source.getc());
        //printf ("%c", source.getc());
    }
                                                                               
                                             
EXIT:
    return 0;
}


Thanks,

Bruno.
Jun 22 2004
prev sibling next sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
When I used BufferedFile the D code went twice as fast as the C++ code but
that probably is a function of the default buffer size. I don't know how big
the C++ buffer is, but in D it is 8K.
Also I noticed a bug (well, a bummer if not a bug) in the BufferedStream
class in stream.d. It needs a destructor that closes the stream:
 ~this(){ close(); }
Such a destructor is in File but not in Stream. I wonder if it should be
moved from File to Stream. Otherwise when I ran your code the final buffer
contents wouldn't be written to the output file.

-Ben

"Bruno A. Costa" <bruno codata.com.br> wrote in message
news:cb99vc$1uur$1 digitaldaemon.com...

 Hi,

 It seems to me that some parts of the std.stream module are working VERY
 slow. I've done some tests with file manipulation and some comparissions
 with C++ and the results are strange.

 ccopy.cc: simple file copy application writen in C++
 dcopy.d:  simple file copy application writen in D
 text.txt: a text file with ~4MB

 Here are the results:

 $ time ./ccopy text.txt test1
 real    0m0.985s
 user    0m0.854s
 sys     0m0.047s

 Less than one second to copy 4MB.

 $ time ./dcopy text.txt test2
 real    1m22.886s
 user    0m25.014s
 sys     0m57.011s
 $

 More than one minute to copy 4MB.

 Here are the sources (both do the same thing: read and write char by char)

 /////////////////////////////////////////
 // ccopy.cpp
 /////////////////////////////////////////
 #include <iostream>
 #include <fstream>

 void error (const char* p, const char* p2=" ")
 {
     std::cerr << p << ' ' << p2 << '\n';
     std::exit(1);
 }

 int main ( int argc, char** argv )
 {
     if (argc != 3)
         error ("usage: copy SOURCE DESTINATION");

     std::ifstream origem (argv[1]);
     if (!origem) error ("error openning file ", argv[1]);

     std::ofstream destino (argv[2]);
     if (!destino) error ("error copying file ", argv[2]);

     char ch;
     while ( origem.get(ch) ) destino.put(ch);

     origem.close();
     destino.close();
 }

 //////////////////////////////////////////
 // dcopy.d
 //////////////////////////////////////////
 import std.stream;

 int main (char[][] args)
 {
     if (args.length <= 2) {
         printf ("Usage: dcopy <source> <destiny>\n");
         goto EXIT;
     }

     char c;
     File source = new File (args[1]);
     File dest = new File ();
     dest.create (args[2]);

     while (!source.eof()) {
         dest.write (source.getc());
         //printf ("%c", source.getc());
     }

 EXIT:
     return 0;
 }


 Thanks,

 Bruno.
Jun 22 2004
parent "Bruno A. Costa" <bruno codata.com.br> writes:
Hi,

I switched from File to BufferedFile and it worked pretty well:
$ time ./dcopy text.txt teste9
real    0m0.848s
user    0m0.641s
sys     0m0.052s

Much Better. Thank you.

Bruno.

Ben Hinkle wrote:


 When I used BufferedFile the D code went twice as fast as the C++ code but
 that probably is a function of the default buffer size. I don't know how
 big the C++ buffer is, but in D it is 8K.
 Also I noticed a bug (well, a bummer if not a bug) in the BufferedStream
 class in stream.d. It needs a destructor that closes the stream:
  ~this(){ close(); }
 Such a destructor is in File but not in Stream. I wonder if it should be
 moved from File to Stream. Otherwise when I ran your code the final buffer
 contents wouldn't be written to the output file.
 
 -Ben
 
 "Bruno A. Costa" <bruno codata.com.br> wrote in message
 news:cb99vc$1uur$1 digitaldaemon.com...

 Hi,

 It seems to me that some parts of the std.stream module are working VERY
 slow. I've done some tests with file manipulation and some comparissions
 with C++ and the results are strange.
Jun 22 2004
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cb99vc$1uur$1 digitaldaemon.com>, Bruno A. Costa says...

                                                                               
                                        
    char ch;
    while ( origem.get(ch) ) destino.put(ch);


By the way, this may be a tad faster than the above:

destino << origem.rdbuf();

I'm planning on putting similar capability in stream.d, though it may not result
in much of a speedup.


Sean
Jun 22 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
Or use mango.io to do a

FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting);
FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate);

in.copyTo (out);

<g>




"Sean Kelly" <sean f4.ca> wrote in message
news:cb9vtv$2ma$1 digitaldaemon.com...

 In article <cb99vc$1uur$1 digitaldaemon.com>, Bruno A. Costa says...

    char ch;
    while ( origem.get(ch) ) destino.put(ch);


 By the way, this may be a tad faster than the above:

 destino << origem.rdbuf();

 I'm planning on putting similar capability in stream.d, though it may not


result

 in much of a speedup.


 Sean
Jun 22 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cba3cv$8l3$1 digitaldaemon.com>, Kris says...

Or use mango.io to do a

FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting);
FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate);

in.copyTo (out);

<g>


Yes, I'm convinced, mango is brilliant. But much of my work involves writing
libraries rather than applications, and there are arguments both for and against
any one library being dependent upon any other. If some small portion of Deimos
(for example) wanted to write to a file, would it be /fair/ to users of Deimos
to require them also to download and link against Mango? Should libraries be
independent of each other, or should at least the best of them become
intertwined.

Based on what I've seen so far, I reckon Mango ought to become /the/ streams
standard for D, in which case, no problem - but at least, I need to ask the
question.

Arcane Jill
Jun 22 2004
parent reply "teqDruid (formerly DemmeGod)" <me teqdruid.com> writes:
Perhaps, in this case, it would be a better idea for part of mango to move
to deimos, and for mango to be dependent on deimos.  As I understand it,
mango is supposed to be a server library, however something like streams
should be in a standard library, like deimos.

John

On Tue, 22 Jun 2004 21:42:04 +0000, Arcane Jill wrote:


 In article <cba3cv$8l3$1 digitaldaemon.com>, Kris says...

Or use mango.io to do a

FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting);
FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate);

in.copyTo (out);

<g>


 
 Yes, I'm convinced, mango is brilliant. But much of my work involves
 writing libraries rather than applications, and there are arguments both
 for and against any one library being dependent upon any other. If some
 small portion of Deimos (for example) wanted to write to a file, would it
 be /fair/ to users of Deimos to require them also to download and link
 against Mango? Should libraries be independent of each other, or should at
 least the best of them become intertwined.
 
 Based on what I've seen so far, I reckon Mango ought to become /the/
 streams standard for D, in which case, no problem - but at least, I need
 to ask the question.
 
 Arcane Jill
Jun 22 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"teqDruid (formerly DemmeGod)"  wrote

 Perhaps, in this case, it would be a better idea for part of mango to move
 to deimos, and for mango to be dependent on deimos.  As I understand it,
 mango is supposed to be a server library, however something like streams
 should be in a standard library, like deimos.


Overall, Mango has a bent towards server-side processing but can be just as
useful for client-side also. A case in point is mango.io, which is an
independent package from the rest of the Mango family (there's a simple
dependency list over here
http://svn.dsource.org/svn/projects/mango/trunk/doc/html/index.html).

Mango.io could indeed be placed into Demios without issue, if you folks feel
that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's lower
layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless of
whatever faults they may have. One way to fix such faults is to change the
standard, but doing so in a democratic manner is no easy task <g>

On the other hand, it's not uncommon to see a grass-roots shift in practice
once there's enough momentum behind a particular approach. Then it becomes
easy to make a choice as to which dependency you wish to live with. Getting
that initial momentum going is the hard part, regardless of any underlying
perceived value.

All that aside, I'm personally a big fan of plugable systems. Although the
extend of an IO package and the performance factors involved can sometimes
make that impractical. It can certainly be done effectively for isolated
scenarios.

- Kris
Jun 22 2004
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...

Mango.io could indeed be placed into Demios without issue, if you folks feel
that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's lower
layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless of
whatever faults they may have. One way to fix such faults is to change the
standard, but doing so in a democratic manner is no easy task <g>


Mango.io has some very nice design features and it may well be a candidate for
Demios.  I've been trying to get stream.d up to date mostly to have an alternate
technique available for comparison.  I'm interedted in seeing the pros and cons
of both methods :)

Sean
Jun 22 2004
next sibling parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Sean Kelly" wrote

 I've been trying to get stream.d up to date


Aren't you, Ben, and Walter far too busy doing other things ?
<G>
Jun 22 2004
parent Sean Kelly <sean f4.ca> writes:
In article <cbao56$17qa$1 digitaldaemon.com>, Kris says...

"Sean Kelly" wrote

 I've been trying to get stream.d up to date


Aren't you, Ben, and Walter far too busy doing other things ?
<G>


Free time is overrated ;)

Sean
Jun 22 2004
prev sibling parent John Reimer <brk_6502 NOSP_AM.yahoo.com> writes:
Sean Kelly wrote:


 Mango.io has some very nice design features and it may well be a candidate for
 Demios.  I've been trying to get stream.d up to date mostly to have an
alternate
 technique available for comparison.  I'm interedted in seeing the pros and cons
 of both methods :)
 
 Sean
 
 


My vote is to get mango.io into Demios.  If anything's lacking, 
contributors certainly can start brainstorming some improvements from 
within Demios.  It's an excellent base at the very least.

Come on people, let's decide on this one. :-)

Later,

John
Jun 23 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...

Mango.io could indeed be placed into Demios without issue, if you folks feel
that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's lower
layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless of
whatever faults they may have. One way to fix such faults is to change the
standard, but doing so in a democratic manner is no easy task <g>


While moving mango.io to Deimos is certainly possible, it wouldn't be my first
choice. My first choice would be that mango.io is moved to *PHOBOS*. D
absolutely requires a serious and well thought out streams package AS STANDARD.

Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't imagine
there's anything in Deimos/etc that wouldn't rather be in Phobos/std, but
obviously we accept that things have to be "out there in the field" for a while,
getting a good testing, getting all the bugs fixed, and so on. You wouldn't want
something in Phobos unless it had passed that test. But the thing about mango.io
is - it has ALREADY passed that test. It has been out there, in the field, for
some time now; all the major bugs have been found and fixed; no-one is planning
any reorganization of mango.io in the near future; the class and function names
are not going to change. In short, it is stable.

What can I say? It's a question of power. Only Walter (so far as I know) has the
power to move things to Phobos, but anyone can move things to Deimos.

But maybe it's also a question of organization. Maybe mango.io could divorce
itself from the rest of Mango and move to Deimos for now (in which case
everything else in Deimos will be able to call it without worrying about
requiring additional libraries, which would certainly be great). But Deimos
itself is nothing special - it's just another library, and there are plenty of
them around. True, it's a library full of things that would rather be in Phobos,
but Mango already has credibility in its own right. On the other hand, I'd be
reluntant to download/import/link against a whole server package, if all I
wanted to do was to open a file.

Let's be honest here - there are SERIOUS problems with std.stream. It is absurd
that calling new File(filename, FileMode.Out) will not delete (truncate) an old
file of the same name. It is absurd that, if you want to do this, you require
two statements: f = new File(); f.create(...);. It is absurd that if you want to
wrap this in a BufferedStream, you have to call create() /before/ you construct
the buffered stream (because BufferedStream has no create() function). And it is
pretty damn annoying that if you continually read from a BufferedStream which
buffers a File, using the readLine() function, then there appears to be no way
to distinguish between an empty line and end-of-file (they both return null, and
no exceptions are thrown). Why is there no Stream.eof() function? Please - let's
just replace it with something that JUST WORKS (and is actually documented).

So, personally, I think Walter should simply make us all happy by adding
mango.io to Phobos (possibly renaming it to std.io en route) and deprecating
std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be my
second choice.

Arcane Jill
Jun 23 2004
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cbc1t0$6h7$1 digitaldaemon.com>, Arcane Jill says...

Let's be honest here - there are SERIOUS problems with std.stream. It is absurd
that calling new File(filename, FileMode.Out) will not delete (truncate) an old
file of the same name. It is absurd that, if you want to do this, you require
two statements: f = new File(); f.create(...);. It is absurd that if you want to
wrap this in a BufferedStream, you have to call create() /before/ you construct
the buffered stream (because BufferedStream has no create() function). And it is
pretty damn annoying that if you continually read from a BufferedStream which
buffers a File, using the readLine() function, then there appears to be no way
to distinguish between an empty line and end-of-file (they both return null, and
no exceptions are thrown). Why is there no Stream.eof() function? Please - let's
just replace it with something that JUST WORKS (and is actually documented).


I'm in the process of fixing all of this... just got bogged down writing a real
ANSI compliant version of scanf (in my hour or two of free coding time per day).
I agree that Mango.io is quite nice, but I don't want it to be chosen simply
because there's nothing else comparable.

One thing I haven't addressed in stream.d and an area that Mango is also
deficient is localized formatted i/o.  This perhaps isn't a requirement for 1.0,
but it would be nice to see some discussion of what we may need in this area
among folks who know.  Numeric formatting is pretty easy, but that still leaves
character transcoding, doesn't it?

Sean
Jun 23 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Sean Kelly" wrote

 One thing I haven't addressed in stream.d and an area that Mango is also
 deficient is localized formatted i/o.  This perhaps isn't a requirement


for 1.0,

 but it would be nice to see some discussion of what we may need in this


area

 among folks who know.  Numeric formatting is pretty easy, but that still


leaves

 character transcoding, doesn't it?


I wholeheartedly agree. Didn't Ben post something about Perl formatting? I
think the combination of what Ben mentioned together with char transcoding
would be a great thing for any IO package.

Note that mango.io is also completely deficient regarding unicode. There
were too many unknowns at the time of writing, so it was left out until a
more formal approach could be introduced. Thanks to Hauke and AJ, the latter
appears to be happening (and mango.io will probably integrate their work as
an additional flavour of Reader & Writer).

In general, how would people like to see formatted IO operate? What about
Ben's idea? How should the upcoming unicode library operate within formatted
IO? Or should it?
Jun 23 2004
next sibling parent Gold Dragon <dragonwing dragonu.net> writes:
If you want my opinion (which you probably don't) then I pitch that the 
I/O for console and file operate like C++. With the exception of 
replacing the >> and << with ~ and damn the person who can't tell 
whether he is getting input or output. I believe that all the >> and << 
are doing is placing the stuff in a buffer anyway.

You could use modifiers like what C++ uses for formatting and it would 
be nice to have Perl Formatting. This perl formatting:
.	format $header =
.<<<<<<<<	<<<<<<<<	<<<<<<<
.$first, $second, $third
'.'

or something, it has been a while since I have done Perl but the general 
  idea is there. Since one of the problems with Perl's Formatting is 
that you have to init the variables before you use them. I pitch also 
that something like this is done:

.	classname ~ "<<<<<<<<" ~ "||||" ~ ">>>>>>>>>>>>>";

to set the up the formatting parameters and then for placing variables 
into the format:

.	classname ~ variable1 ~ variable2 ~ module::next ~ variable3 ~ variable4

If modifier 'next' is called then place the variables before it (if not 
before another 'next') into the same position else place each into a 
separate position. It would be better than having 'next' after all 
positions. The class should be smart enough to know what is happening. 
Or "Do What I Want" paradigm. I'm sure someone could think of a better 
way but here is my pitch: bow to or curse it.


 Kris wrote:
 In general, how would people like to see formatted IO operate? What about
 Ben's idea? How should the upcoming unicode library operate within formatted
 IO? Or should it?
Jun 23 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbcgcj$u8q$1 digitaldaemon.com>, Kris says...

"Sean Kelly" wrote
 Numeric formatting is pretty easy, but that still
leaves
 character transcoding, doesn't it?


I don't understand. What's character transcoding?



I wholeheartedly agree. Didn't Ben post something about Perl formatting? I
think the combination of what Ben mentioned together with char transcoding
would be a great thing for any IO package.


It's obviously important then. What is it?



Note that mango.io is also completely deficient regarding unicode. There
were too many unknowns at the time of writing, so it was left out until a
more formal approach could be introduced. Thanks to Hauke and AJ, the latter
appears to be happening (and mango.io will probably integrate their work as
an additional flavour of Reader & Writer).

In general, how would people like to see formatted IO operate? What about
Ben's idea? How should the upcoming unicode library operate within formatted
IO? Or should it?


Well, just to clarify what's "upcoming" on the Unicode front:

In the next few days, functions to access all Unicode character properties. This
is an important first step, because the Unicode algorithms are defined in terms
of those properties.

Sometime after that, normalization algorithms. This is slightly complicated in
that, for completeness, we'd want it continue to work in the event of PUA
(Private Use Area) customization.

I was planning on stopping after that for a short break and going back to crypto
(finish off the hash section). Presumably next on the list for Unicode should
come the full casing algorithms - these are actually locale-sensitive (though
only rarely, since all locales give identical results apart from "az", "lt" and
"tr").

Simply put, the improvement you will see will be gradual. We started with
functions like std.ctype.toupper(), defined only for ASCII characters; Hauke
gave us utype.toupper() and friends, which extended that to all characters.
Before long, we'll have a string toupper-ing function that does full casing.

I'm not completely sure I understand the question "how would people like to see
formatted IO operate?", or what it has to do with Unicode. Now, there are some
really big and powerful algorithms described out there on the Unicode web site,
and one of them concerns Unicode Regular Expressions - a staggeringly powerful
concept - but probably the most difficult of all Unicode algorithms to implement
in practice. You'll probably want that one, but alas I fear it will be one of
the last to arrive.

Anyway, if I'm way off track, forgive me.

Much of localization has to do with replacing text strings (normally English)
with a translation into some alternative language. This is usually done at the
pre-stream level (in my experience), so by the time it gets to the stream, it's
already in French/German/whatever.

But there's obviously something I'm not getting here. Ah well. Put it down to my
not being as bright as I'd like to be.

Jill
Jun 23 2004
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cbd3p3$1tef$1 digitaldaemon.com>, Arcane Jill says...

In article <cbcgcj$u8q$1 digitaldaemon.com>, Kris says...

"Sean Kelly" wrote
 Numeric formatting is pretty easy, but that still
leaves character transcoding, doesn't it?


I don't understand. What's character transcoding?


Say I has a string of dchars and I want to write it to a text file.  Is it
enough to just do an unformatted write of the string or does it have to be
converted to UTF-16 or some such?  ie. is there ever any difference between
internal character representation and external representation that an i/o lib
may care about?


I'm not completely sure I understand the question "how would people like to see
formatted IO operate?", or what it has to do with Unicode.


What I meant was any internationalization.  As I mentioned, the issues I know of
have to do with numeric formatting, but is there anything else?

Sean
Jun 23 2004
next sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbd6v9$21mu$1 digitaldaemon.com>, Sean Kelly says...

Say I has a string of dchars and I want to write it to a text file.  Is it
enough to just do an unformatted write of the string or does it have to be
converted to UTF-16 or some such?  ie. is there ever any difference between
internal character representation and external representation that an i/o lib
may care about?


Gotcha. Okay, I do know about that - it's just I've always called it ENcoding
(and DEcoding) rather than TRANScoding - but I guess "transcoding" is just as
correct, if not more correct.

Yeah, well that sounds like a simple enough thing to do. You know that the
encoding at one end (inside a D program) is going to be one of the Unicode
UTF-s. Presumably, you have some information about the encoding at the other end
(the console, a text window, a file, whatever), and is likely to be WINDOWS-1252
on Windows machines in western Europe, etc.. So all you have to do is convert
from one to the other.

My guess is, you'd want some sort of Transcoder class (see I've leant a new
word), so you could do something like:






and so on, and then use this either as a parameter to your stream constructor,
or a parameter to a formatted input/output command. A transcoder would simply
have to convert from a sequence of dchars to a sequence of ubytes (and vice
versa). (Note that the names of encodings are universally understood - we don't
have to make them up).

Currently, of course, D assumes UTF-8 on the console, which is, in fact, usually
wrong, which is why if you printf() a non-ASCII character to an MS-DOS command
window, they come out wrong. A transcoder scheme would fix this.










I'm not completely sure I understand the question "how would people like to see
formatted IO operate?", or what it has to do with Unicode.


What I meant was any internationalization.  As I mentioned, the issues I know of
have to do with numeric formatting, but is there anything else?

Sean
Jun 23 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
On Thu, 24 Jun 2004 06:57:13 +0000 (UTC), Arcane Jill 
<Arcane_member pathlink.com> wrote:

 In article <cbd6v9$21mu$1 digitaldaemon.com>, Sean Kelly says...

 Say I has a string of dchars and I want to write it to a text file.  Is 
 it
 enough to just do an unformatted write of the string or does it have to 
 be
 converted to UTF-16 or some such?  ie. is there ever any difference 
 between
 internal character representation and external representation that an 
 i/o lib
 may care about?


 Gotcha. Okay, I do know about that - it's just I've always called it 
 ENcoding
 (and DEcoding) rather than TRANScoding - but I guess "transcoding" is 
 just as
 correct, if not more correct.

 Yeah, well that sounds like a simple enough thing to do. You know that 
 the
 encoding at one end (inside a D program) is going to be one of the 
 Unicode
 UTF-s. Presumably, you have some information about the encoding at the 
 other end
 (the console, a text window, a file, whatever), and is likely to be 
 WINDOWS-1252
 on Windows machines in western Europe, etc.. So all you have to do is 
 convert
 from one to the other.

 My guess is, you'd want some sort of Transcoder class (see I've leant a 
 new
 word), so you could do something like:






 and so on, and then use this either as a parameter to your stream 
 constructor,
 or a parameter to a formatted input/output command. A transcoder would 
 simply
 have to convert from a sequence of dchars to a sequence of ubytes (and 
 vice
 versa). (Note that the names of encodings are universally understood - 
 we don't
 have to make them up).

 Currently, of course, D assumes UTF-8 on the console, which is, in fact, 
 usually
 wrong, which is why if you printf() a non-ASCII character to an MS-DOS 
 command
 window, they come out wrong. A transcoder scheme would fix this.


I always thought the best way to handle this sort of thing would be to 
have a pluggable modular system such that you'd have your input or 
'source', your output or 'sink', and you could plug any number of 
transcoders in between them.

So for example your source might be a file, and your sink a socket, you 
plug the transcoders for SSL, UTF-8 to Unicode, or anything else you want 
in, they all modify the data going thru them such that the data you get 
 from the source arrives at the sink transcoded in those ways.

I always wanted to try and write a stream library based on that idea...

Regan.

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Jun 24 2004
parent Sean Kelly <sean f4.ca> writes:
In article <opr93b3wzg5a2sq9 digitalmars.com>, Regan Heath says...

I always thought the best way to handle this sort of thing would be to 
have a pluggable modular system such that you'd have your input or 
'source', your output or 'sink', and you could plug any number of 
transcoders in between them.

So for example your source might be a file, and your sink a socket, you 
plug the transcoders for SSL, UTF-8 to Unicode, or anything else you want 
in, they all modify the data going thru them such that the data you get 
 from the source arrives at the sink transcoded in those ways.

I always wanted to try and write a stream library based on that idea...


I've written stream adaptors for this sort of thing.  So I might chain together
a GZip stream, a Base64 stream, and a socket stream and each would encode the
data and push it on to the next layer.  But for things like SSL this can get
pretty complicated, as you may have to deal with password prompting and other
configuration for that specific level.  This is one area where Aspect oriented
programming has a lot of promise.

Sean
Jun 24 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbd6v9$21mu$1 digitaldaemon.com>, Sean Kelly says...


As I mentioned, the issues I know of
have to do with numeric formatting,


Unfortunately, Unicode won't help you with numeric formatting. As you know, the
decimal point (for example) is encoded as U+002E if you're English, but as
U+002C if you're French. At one point, I did suggest to the Consortium folk that
they add an UNAMBIGUOUS DECIMAL POINT character, which would render as "." in an
English font or "," in a French font, but they didn't like that idea. As far as
the Unicode Consortium are concerned, Unicode is a repertoire of characters, not
a repertoire of semantic meanings. Therefore, when it comes to this sort of
localization issue, we're on our own.

In a way, Java does this quite well - at least in the sense that it is
sufficiently powerful. Where it falls down is in ease of use. It is not easy to
do even simple localization in Java. I'm wondering if that's something else
we're stuck with - it is even possible to make it BOTH easy AND powerful?

Unfortunately, I only have questions on this one, not answers.

Arcane Jill
Jun 24 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <cbe04d$6ri$1 digitaldaemon.com>, Arcane Jill says...

In a way, Java does this quite well - at least in the sense that it is
sufficiently powerful. Where it falls down is in ease of use. It is not easy to
do even simple localization in Java. I'm wondering if that's something else
we're stuck with - it is even possible to make it BOTH easy AND powerful?


Good question.  C++ has a global locale setting and then IIRC you can associate
a stream instance with a different locale.  Formatting information is stored in
a class or set of classes that defines the various separator characters and
such.  This is quite easy to use but fairly complicated to extend.  But the
basic idea does seem to work pretty well.  How does Java work?

Sean
Jun 24 2004
parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbesja$1for$1 digitaldaemon.com>, Sean Kelly says...

In article <cbe04d$6ri$1 digitaldaemon.com>, Arcane Jill says...

In a way, Java does this quite well - at least in the sense that it is
sufficiently powerful. Where it falls down is in ease of use. It is not easy to
do even simple localization in Java. I'm wondering if that's something else
we're stuck with - it is even possible to make it BOTH easy AND powerful?


Good question.  C++ has a global locale setting and then IIRC you can associate
a stream instance with a different locale.  Formatting information is stored in
a class or set of classes that defines the various separator characters and
such.  This is quite easy to use but fairly complicated to extend.  But the
basic idea does seem to work pretty well.  How does Java work?


There are basically three big problems to solve in localization. Java solves all
three. It doesn't necessarily have the BEST solutions, so we don't necessarily
have to rip off the way Java does it, but we do need to solve the same three
problems. These are:

PROBLEM 1 - HOW TO DEFINE A LOCALE
PROBLEM 2 - RESOURCE FILES
PROBLEM 3 - INDEXED VARIADIC FUNCTIONS


PROBLEM 1 - HOW TO DEFINE A LOCALE

The problem is how to define PRECISELY what you mean by "locale". All other
locale stuff will depend on this definition.

Conceptually, a locale is something like "en-us", a string - but it's convenient
not to have to deal with strings directly, because strings are slow, and you
have to deal with issues of case, punctuation (hyphen verses underscore), and so
on. Java solves this by having a class, Locale, which contains such strings, but
it converts them to some standard internal format so it only has to do all that
casing stuff ONCE.

My own view is that this is too complicated. Especially when you have to deal
with the three-letter ISO-639 / ISO-3166 codes (as opposed to the two-letter
codes). It strikes me that a simple enum would suffice. Then you could do:






Bingo. Problem solved. Now whenever you want to pass locale information you only
have to pass one or two enum values. In the case of ISO-3166 (countries) the
actual enum VALUES are even defined for us by the standard. ISO-639 (languages)
defines only the names, so we'd have to make up the values.

(And since emums will auto-cast into integers, you could templatize on them. I
haven't thought through the implications of this).

C++, of course, defines locales by its own internal means, without regard to any
ISO standard. I don't recommend we copy this. Way too messy.


PROBLEM 2 - RESOURCE FILES

This is a very simple problem to define. Given a locale, open a file containing
information relevant to that locale.

For instance, you have an application, and in the course of execution, it prints
stuff. You want it to print in English if the locale-language is Language.EN,
French if the locale-language is Language.FR, and so on. So you open the "right"
file (there will be one for each supported locale), read in all the text strings
into an array, and then spit out the appropriate one at the the appropriate
time.

The problem? Where do you find the file? 

See - your app could have been installed anywhere on the user's filesystem. Same
goes for your libraries. The current working directory could be anywhere. There
is no "obvious" place to look.

Traditionally, the Linux answer is to get the "right place to look" from an
environment variable. But then you have the problem of environment variable name
clashes.

Traditionally, the Windows answer is to get the "right place to look" from the
registry. No problem with name clashes here, but now you have a different
problem - it takes about fifteen thousand lines of code to do the equivalent of
getenv().

Enter Java's magnificent solution. Java has a class, ResourceBundle, which does
the job. There are two forms, if memory serves correct. In one version, it
parses a human-readable text file into a class. In the other version, it
deserializes an arbitrary serialized object into a class. But either way, it
knows where to look by SEARCHING THE JAVA CLASSPATH. What this means, in simple
terms, is that source code is able to locate the file RELATIVE TO THE SOURCE
FILE. And this alone is staggeringly powerful.

In D terms - you could put a file in the same directory as a source file, and
just OPEN IT. Or, if it were in a different directory (in an associated package,
say) which could be located RELATIVE TO THE SOURCE FILE then, again, that's
problem solved.

However, this is a difficult thing to do in D, because there is no classpath. We
may have to come up with other solutions to this one.



PROBLEM 3 - INDEXED VARIADIC FUNCTIONS

Once you've got your strings and your formatting classes, you're very nearly
sorted. Just one problem remains - word order. Consider - this will work in
standard English:






Prints: the server is running

Assume that the first three lines could be substituted with fetching the values
from a resource file or some other means, but the fourth line actually a part of
your program. Trouble is, it won't work with all languages, because of word
order. On planet Degobar (where Yoda comes from), you'd want to say "running,
the server is" (rather than "the server is running"), so that printf just won't
do. It can't change the word order.

Java has SOMETHING LIKE printf for this purpose. It takes a format string
containing escape sequences such as %1, %2, %3, and so on, and the numbers tell
you WHICH string you're referring to. (This is cumbersome in Java, because you
have to use an array). But, basically, it lets you do:




The printf-like function can now plug the RIGHT values into the right places.
Bingo - one string, correctly formatted for an arbitrary locale.

I understand that Walter is working on a D-printf() right now. If so, Walter, if
you could give us some means of accessing the /nth/ argument (instead of always
the /next/ argument), then that problem would be solved immediately.

Arcane Jill
Jun 25 2004
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
Just thought of another one.  What constitutes whitespace is locale dependent as
well.  So add that to the localization list.
Jun 23 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbd7ac$2284$1 digitaldaemon.com>, Sean Kelly says...

Just thought of another one.  What constitutes whitespace is locale dependent as
well.  So add that to the localization list.


Well, this is not true in Unicode, so - however you define that - you will have
to do it without the aid a Unicode Library. Whitespace according to Unicode is
defined by the property file http://www.unicode.org/Public/UNIDATA/PropList.txt.
The following characters - and ONLY the following characters, are deemed to be
whitespace according to Unicode:














My personal opinion (for what it's worth) is that this does not need to be
localized. Maybe if a particular application (such as a D compiler, for example)
required that (for example) U+00A0 be defined as non-whitespace, then the
application could simply work around that - I believe Java does something
similar, defining a "Java whitespace" with a slightly different list which
excludes U+00A0.

But as for localizing for a locale, my personal opinion is that Unicode's
definition is good enough. If you're going to use a different list, then you're
not using Unicode! Part of the whole point of Unicode is to ease localization
issues by simply eliminating them wherever possible. ONE standard. So far as I
can tell, the argument that "what is whitespace is locale-dependent" is
8-bit-standard-thinking, and we simply don't need to use that any more.

Arcane Jill
Jun 23 2004
parent Sean Kelly <sean f4.ca> writes:
Works for me.  I'll admit that most of what I know of localization has come from
C++, which hasn't quite embraced unicode yet :)

Sean
Jun 24 2004
prev sibling next sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cbc1t0$6h7$1 digitaldaemon.com...

 In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...

Mango.io could indeed be placed into Demios without issue, if you folks




feel

that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's




lower

layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless




of

whatever faults they may have. One way to fix such faults is to change




the

standard, but doing so in a democratic manner is no easy task <g>


 While moving mango.io to Deimos is certainly possible, it wouldn't be my


first

 choice. My first choice would be that mango.io is moved to *PHOBOS*. D
 absolutely requires a serious and well thought out streams package AS


STANDARD.

 Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't


imagine

 there's anything in Deimos/etc that wouldn't rather be in Phobos/std, but
 obviously we accept that things have to be "out there in the field" for a


while,

 getting a good testing, getting all the bugs fixed, and so on. You


wouldn't want

 something in Phobos unless it had passed that test. But the thing about


mango.io

 is - it has ALREADY passed that test. It has been out there, in the field,


for

 some time now; all the major bugs have been found and fixed; no-one is


planning

 any reorganization of mango.io in the near future; the class and function


names

 are not going to change. In short, it is stable.

 What can I say? It's a question of power. Only Walter (so far as I know)


has the

 power to move things to Phobos, but anyone can move things to Deimos.

 But maybe it's also a question of organization. Maybe mango.io could


divorce

 itself from the rest of Mango and move to Deimos for now (in which case
 everything else in Deimos will be able to call it without worrying about
 requiring additional libraries, which would certainly be great). But


Deimos

 itself is nothing special - it's just another library, and there are


plenty of

 them around. True, it's a library full of things that would rather be in


Phobos,

 but Mango already has credibility in its own right. On the other hand, I'd


be

 reluntant to download/import/link against a whole server package, if all I
 wanted to do was to open a file.




 Let's be honest here - there are SERIOUS problems with std.stream.


I don't know why std.stream is such a whipping boy, but I'll throw in my 2
cents:


 It is absurd
 that calling new File(filename, FileMode.Out) will not delete (truncate)


an old

 file of the same name. It is absurd that, if you want to do this, you


require

 two statements: f = new File(); f.create(...);.


seems easy enough to fix. I haven't played around with possible solutions
but it would require changing just a few lines of code - possibly adding an
optional input to the constructor to truncate or another FileMode or
something.


 It is absurd that if you want to
 wrap this in a BufferedStream, you have to call create() /before/ you


construct

 the buffered stream (because BufferedStream has no create() function).


I'm not sure what a BufferedStream create() function would do. Do you mean
BufferedFile?


 And it is
 pretty damn annoying that if you continually read from a BufferedStream


which

 buffers a File, using the readLine() function, then there appears to be no


way

 to distinguish between an empty line and end-of-file (they both return


null, and

 no exceptions are thrown). Why is there no Stream.eof() function?


Stream.eof() exists.


 Please - let's
 just replace it with something that JUST WORKS (and is actually


documented).

Improving the doc would be nice.


 So, personally, I think Walter should simply make us all happy by adding
 mango.io to Phobos (possibly renaming it to std.io en route) and


deprecating

 std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be my
 second choice.

 Arcane Jill
Jun 23 2004
parent "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Ben Hinkle" wrote

 I don't know why std.stream is such a whipping boy


I hope that's not the case, but the two certainly have alternate approaches.
I don't understand it, but that can sometimes cause heated debate on this NG
:-)

Personally, I feel std.stream and mango.io are very different animals ~ as
such it's hard to compare and contrast. Perhaps it might be useful to
clarify what some of the notions are/were behind mango.io, such that any
further discussion might be more finely tuned?
Jun 23 2004
prev sibling parent reply "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
My opinion is that neither should be in the 1.0 standard library, so we have
freedom to have a proper competitive, tested, comprehensive implementation for
2.0, which will be maximally efficient, usable, comprehensible and robust. The
C++ streams library only manages 1.5 / 4 there, and that took a long time to
develop.

The same goes for GUI.

It may be that Mango ends up being the standard streams for Phobos, or
std.stream, or a refactored of std.stream, or something entirely new.

Since std.stream is in there, I don't expect it to be taken out. But I do think
that the above open-mindedness should be manifest in the community, and clearly
expressed to all users, especially new ones.

D has the faculties to mean that we have the potential to have the best IO
libraries in the C-family world. I hope it will be realised.



"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cbc1t0$6h7$1 digitaldaemon.com...

 In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...

Mango.io could indeed be placed into Demios without issue, if you folks feel
that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's lower
layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless of
whatever faults they may have. One way to fix such faults is to change the
standard, but doing so in a democratic manner is no easy task <g>


 While moving mango.io to Deimos is certainly possible, it wouldn't be my first
 choice. My first choice would be that mango.io is moved to *PHOBOS*. D
 absolutely requires a serious and well thought out streams package AS STANDARD.

 Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't imagine
 there's anything in Deimos/etc that wouldn't rather be in Phobos/std, but
 obviously we accept that things have to be "out there in the field" for a


while,

 getting a good testing, getting all the bugs fixed, and so on. You wouldn't


want

 something in Phobos unless it had passed that test. But the thing about


mango.io

 is - it has ALREADY passed that test. It has been out there, in the field, for
 some time now; all the major bugs have been found and fixed; no-one is planning
 any reorganization of mango.io in the near future; the class and function names
 are not going to change. In short, it is stable.

 What can I say? It's a question of power. Only Walter (so far as I know) has


the

 power to move things to Phobos, but anyone can move things to Deimos.

 But maybe it's also a question of organization. Maybe mango.io could divorce
 itself from the rest of Mango and move to Deimos for now (in which case
 everything else in Deimos will be able to call it without worrying about
 requiring additional libraries, which would certainly be great). But Deimos
 itself is nothing special - it's just another library, and there are plenty of
 them around. True, it's a library full of things that would rather be in


Phobos,

 but Mango already has credibility in its own right. On the other hand, I'd be
 reluntant to download/import/link against a whole server package, if all I
 wanted to do was to open a file.

 Let's be honest here - there are SERIOUS problems with std.stream. It is absurd
 that calling new File(filename, FileMode.Out) will not delete (truncate) an old
 file of the same name. It is absurd that, if you want to do this, you require
 two statements: f = new File(); f.create(...);. It is absurd that if you want


to

 wrap this in a BufferedStream, you have to call create() /before/ you construct
 the buffered stream (because BufferedStream has no create() function). And it


is

 pretty damn annoying that if you continually read from a BufferedStream which
 buffers a File, using the readLine() function, then there appears to be no way
 to distinguish between an empty line and end-of-file (they both return null,


and

 no exceptions are thrown). Why is there no Stream.eof() function? Please -


let's

 just replace it with something that JUST WORKS (and is actually documented).

 So, personally, I think Walter should simply make us all happy by adding
 mango.io to Phobos (possibly renaming it to std.io en route) and deprecating
 std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be my
 second choice.

 Arcane Jill
Jun 23 2004
next sibling parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Matthew" wrote ...

 My opinion is that neither should be in the 1.0 standard library, so we


have

 freedom to have a proper competitive, tested, comprehensive implementation


for

 2.0, which will be maximally efficient, usable, comprehensible and robust.


I couldn't agree more Matthew. However, the fact that std.stream is already
in phobos gives it a decided advantage. That means the playing field is,
unfortunately, not as you idealize it above. Nor will it be regarding DTL,
since you already have a percieved advantage via your connection with Walter
(obviated by the digitalmars.D.dtl NG). Don't get me wrong ~ there's no sour
grapes here. Just making a point that reality is rarely unbiased. Let's not
give anyone the wrong impression.


 C++ streams library only manages 1.5 / 4 there, and that took a long time


to

 develop.


<black humour>
As you say; the C++ streams library did indeed take a long time to develop.
How long does it usually take to admit that a fundamental approach is
tragically flawed? (sorry; sorry ... just a bad joke)
</black humour>


 It may be that Mango ends up being the standard streams for Phobos, or
 std.stream, or a refactored of std.stream, or something entirely new.


*mango.io* is but one small independent branch of the Mango Tree, Matthew
<g> Perhaps we ought to rename it yet again. Also, as noted elsewhere,
mango.io and std.stream are comparable on a superficial basis only. Although
they both clearly perform IO of some kind, that's perhaps the extent of
their relationship. Where the Phobos library has many vaguely related, and
somewhat fractured modules related to IO, mango.io tries hard to be fairly
cohesive. Perhaps that is the primary reason why those who use it like it as
much as they do? I hope you'll like it too, when you start disseminating the
mango.servlet package :-)

For the record, let me state that I'm not personlly advocating mango.io
should be in Phobos. Someone please correct me if I'm wrong, but I've noted
only that I wish it to be reviewed (and perhaps ratified) by the
oft-discussed but non-existent "D standard library group committee", along
with other members of the Mango Tree family (see old NG for details on the
"committee"). The last thing I want (or need) is for mango.io and std.stream
to become political footballs; that would be entirely unhelpful to the whole
D community.

How about that darned DSLG committee? Lars? Walter?

- Kris



 Since std.stream is in there, I don't expect it to be taken out. But I do


think

 that the above open-mindedness should be manifest in the community, and


clearly

 expressed to all users, especially new ones.

 D has the faculties to mean that we have the potential to have the best IO
 libraries in the C-family world. I hope it will be realised.



 "Arcane Jill" <Arcane_member pathlink.com> wrote in message
 news:cbc1t0$6h7$1 digitaldaemon.com...

 In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...

Mango.io could indeed be placed into Demios without issue, if you folks






feel

that's the right home for it. However, the point about mixing portions






of

libraries is a good one: sometimes a library must expose some of it's






lower

layers (perhaps as method arguments, or delegate parameters), and if






said

lower layer uses mango.io rather than phobos.io then a user might






become

righteously indignant. Sometimes standards are a good thing, regardless






of

whatever faults they may have. One way to fix such faults is to change






the

standard, but doing so in a democratic manner is no easy task <g>


 While moving mango.io to Deimos is certainly possible, it wouldn't be my




first

 choice. My first choice would be that mango.io is moved to *PHOBOS*. D
 absolutely requires a serious and well thought out streams package AS




STANDARD.

 Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't




imagine

 there's anything in Deimos/etc that wouldn't rather be in Phobos/std,




but

 obviously we accept that things have to be "out there in the field" for




a

 while,

 getting a good testing, getting all the bugs fixed, and so on. You




wouldn't

 want

 something in Phobos unless it had passed that test. But the thing about


 mango.io

 is - it has ALREADY passed that test. It has been out there, in the




field, for

 some time now; all the major bugs have been found and fixed; no-one is




planning

 any reorganization of mango.io in the near future; the class and




function names

 are not going to change. In short, it is stable.

 What can I say? It's a question of power. Only Walter (so far as I know)




has

 the

 power to move things to Phobos, but anyone can move things to Deimos.

 But maybe it's also a question of organization. Maybe mango.io could




divorce

 itself from the rest of Mango and move to Deimos for now (in which case
 everything else in Deimos will be able to call it without worrying about
 requiring additional libraries, which would certainly be great). But




Deimos

 itself is nothing special - it's just another library, and there are




plenty of

 them around. True, it's a library full of things that would rather be in


 Phobos,

 but Mango already has credibility in its own right. On the other hand,




I'd be

 reluntant to download/import/link against a whole server package, if all




I

 wanted to do was to open a file.

 Let's be honest here - there are SERIOUS problems with std.stream. It is




absurd

 that calling new File(filename, FileMode.Out) will not delete (truncate)




an old

 file of the same name. It is absurd that, if you want to do this, you




require

 two statements: f = new File(); f.create(...);. It is absurd that if you




want

 to

 wrap this in a BufferedStream, you have to call create() /before/ you




construct

 the buffered stream (because BufferedStream has no create() function).




And it

 is

 pretty damn annoying that if you continually read from a BufferedStream




which

 buffers a File, using the readLine() function, then there appears to be




no way

 to distinguish between an empty line and end-of-file (they both return




null,

 and

 no exceptions are thrown). Why is there no Stream.eof() function?




Please -

 let's

 just replace it with something that JUST WORKS (and is actually




documented).

 So, personally, I think Walter should simply make us all happy by adding
 mango.io to Phobos (possibly renaming it to std.io en route) and




deprecating

 std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be




my

 second choice.

 Arcane Jill
Jun 23 2004
parent reply "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
 "Matthew" wrote ...

 My opinion is that neither should be in the 1.0 standard library, so we


 have

 freedom to have a proper competitive, tested, comprehensive implementation


 for

 2.0, which will be maximally efficient, usable, comprehensible and robust.


 I couldn't agree more Matthew. However, the fact that std.stream is already
 in phobos gives it a decided advantage. That means the playing field is,
 unfortunately, not as you idealize it above.


True. But if std.stream does not cut the mustard, I for one will be voicing for
change.

I think streaming, and especially object instance
serialisation/marshalling/streaming/pickling/whatever-you-want-to-call-it, will
be a feature that D *must* do right, in order to curtail comparisons with Java,
Python and other languages that do this well on at least one of the four
criteria
on which I've scored C++ 1.5.

I totally agree with Walter that this is something that 1.0 can't wait for. It's
also something that will require a very large amount of design, development and
refinement, including large amounts of Walter's time - as it'll probably have to
involve language as well as library - and lots of input from people with
different backgrounds, to get it 100% correct correct. (Well, 95%; there are no
100s.)


 Nor will it be regarding DTL,
 since you already have a percieved advantage via your connection with Walter


Maybe there's an advantage there. But any "connection" with Walter stems from
being enthusiastic, and putting as much effort into writing code as into writing
NG posts. It's not really anything more than that. (Of course, maybe when our
book's more corporeal than corporate schpiel we'll have an irrevocable
connection. Pity Walter from that point on. <g>)

Regarding DTL. I have had what I think is a good idea, and Walter has agreed
thus
far. But a combination of compiler bugs (which I've yet to distill) and an
all-consuming workload has meant that nothing's really happened from the initial
implementation in March. I intend and expect to rectify that by the end of July.
But there's a big gap in time. If someone else had a grand idea for DTL then
they've had plenty of time to develop/refine it. I've said all along that DTL-MW
is not a done deal, and that others should persue alternatives if they have
them,
and we can evaluate *all* options on their merits.


 (obviated by the digitalmars.D.dtl NG).


<pedantic>"obviated"? I think you mean "exemplified". Obviate means to render
unnecessary</pedantic>


 Don't get me wrong ~ there's no sour
 grapes here.


Kris. You don't need to say that. I know you know that you have my undying
respect ...


 Just making a point that reality is rarely unbiased. Let's not
 give anyone the wrong impression.


Again. Yes and no. Despite the occasional heat, I think just about everyone here
is more interested in D being really good, than in just getting their own
opinions swallowed. (At least I hope so!) So I think right will out in most
cases, especially the big ones.


 C++ streams library only manages 1.5 / 4 there, and that took a long time


 to

 develop.


 <black humour>
 As you say; the C++ streams library did indeed take a long time to develop.
 How long does it usually take to admit that a fundamental approach is
 tragically flawed? (sorry; sorry ... just a bad joke)
 </black humour>


Well, Java's been with us for nearly a decade ...

To be serious, for all that I really don't care for the IOStreams, they do have
the localisation issue relatively well handled. That's not something to sniff
at,
even though it does have high performance costs.


 It may be that Mango ends up being the standard streams for Phobos, or
 std.stream, or a refactored of std.stream, or something entirely new.


 *mango.io* is but one small independent branch of the Mango Tree, Matthew
 <g> Perhaps we ought to rename it yet again. Also, as noted elsewhere,
 mango.io and std.stream are comparable on a superficial basis only. Although
 they both clearly perform IO of some kind, that's perhaps the extent of
 their relationship. Where the Phobos library has many vaguely related, and
 somewhat fractured modules related to IO, mango.io tries hard to be fairly
 cohesive. Perhaps that is the primary reason why those who use it like it as
 much as they do?


I know, I know. Apologies for my abuse: loose use of language. I'm a goose. I'll
try and spruce it up, be less abstruse.


 I hope you'll like it too, when you start disseminating the
 mango.servlet package :-)


Yes. I've promised to do that July, and I'll keep it.


 For the record, let me state that I'm not personlly advocating mango.io
 should be in Phobos. Someone please correct me if I'm wrong, but I've noted
 only that I wish it to be reviewed (and perhaps ratified) by the
 oft-discussed but non-existent "D standard library group committee", along
 with other members of the Mango Tree family (see old NG for details on the
 "committee"). The last thing I want (or need) is for mango.io and std.stream
 to become political footballs; that would be entirely unhelpful to the whole
 D community.


Nope. AFAICR, you've not said that.

In any case, playing football with a mango would be a huge waste. Mangos must be
respectfully, even reverently, consumed.



 How about that darned DSLG committee? Lars? Walter?

 - Kris



 Since std.stream is in there, I don't expect it to be taken out. But I do


 think

 that the above open-mindedness should be manifest in the community, and


 clearly

 expressed to all users, especially new ones.

 D has the faculties to mean that we have the potential to have the best IO
 libraries in the C-family world. I hope it will be realised.



 "Arcane Jill" <Arcane_member pathlink.com> wrote in message
 news:cbc1t0$6h7$1 digitaldaemon.com...

 In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...

Mango.io could indeed be placed into Demios without issue, if you folks






 feel

that's the right home for it. However, the point about mixing portions






 of

libraries is a good one: sometimes a library must expose some of it's






 lower

layers (perhaps as method arguments, or delegate parameters), and if






 said

lower layer uses mango.io rather than phobos.io then a user might






 become

righteously indignant. Sometimes standards are a good thing, regardless






 of

whatever faults they may have. One way to fix such faults is to change






 the

standard, but doing so in a democratic manner is no easy task <g>


 While moving mango.io to Deimos is certainly possible, it wouldn't be my




 first

 choice. My first choice would be that mango.io is moved to *PHOBOS*. D
 absolutely requires a serious and well thought out streams package AS




 STANDARD.

 Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't




 imagine

 there's anything in Deimos/etc that wouldn't rather be in Phobos/std,




 but

 obviously we accept that things have to be "out there in the field" for




 a

 while,

 getting a good testing, getting all the bugs fixed, and so on. You




 wouldn't

 want

 something in Phobos unless it had passed that test. But the thing about


 mango.io

 is - it has ALREADY passed that test. It has been out there, in the




 field, for

 some time now; all the major bugs have been found and fixed; no-one is




 planning

 any reorganization of mango.io in the near future; the class and




 function names

 are not going to change. In short, it is stable.

 What can I say? It's a question of power. Only Walter (so far as I know)




 has

 the

 power to move things to Phobos, but anyone can move things to Deimos.

 But maybe it's also a question of organization. Maybe mango.io could




 divorce

 itself from the rest of Mango and move to Deimos for now (in which case
 everything else in Deimos will be able to call it without worrying about
 requiring additional libraries, which would certainly be great). But




 Deimos

 itself is nothing special - it's just another library, and there are




 plenty of

 them around. True, it's a library full of things that would rather be in


 Phobos,

 but Mango already has credibility in its own right. On the other hand,




 I'd be

 reluntant to download/import/link against a whole server package, if all




 I

 wanted to do was to open a file.

 Let's be honest here - there are SERIOUS problems with std.stream. It is




 absurd

 that calling new File(filename, FileMode.Out) will not delete (truncate)




 an old

 file of the same name. It is absurd that, if you want to do this, you




 require

 two statements: f = new File(); f.create(...);. It is absurd that if you




 want

 to

 wrap this in a BufferedStream, you have to call create() /before/ you




 construct

 the buffered stream (because BufferedStream has no create() function).




 And it

 is

 pretty damn annoying that if you continually read from a BufferedStream




 which

 buffers a File, using the readLine() function, then there appears to be




 no way

 to distinguish between an empty line and end-of-file (they both return




 null,

 and

 no exceptions are thrown). Why is there no Stream.eof() function?




 Please -

 let's

 just replace it with something that JUST WORKS (and is actually




 documented).

 So, personally, I think Walter should simply make us all happy by adding
 mango.io to Phobos (possibly renaming it to std.io en route) and




 deprecating

 std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be




 my

 second choice.

 Arcane Jill
Jun 23 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Matthew" wrote

 I think streaming, and especially object instance
 serialisation/marshalling/streaming/pickling/whatever-you-want-to-call-it,
 will be a feature that D *must* do right, in order to curtail comparisons
 with Java, Python and other languages that do this well on at least one
 of the four criteria on which I've scored C++ 1.5


Ah, pickling! There's been so many times we've worked around the predefined
Java serialization that I've had team members use an alternative. Especially
after tracing just how inefficient it is; not to mention arcane. The
Serializable interface is cool, but the hidden methods (private yet somehow
public) to intercept default processing are, uhhhh, funky. Then they had to
extend the language with keywords such as transient ... it's a solution that
looks immediately viable, yet soon runs out of steam. But it works, which is
the main thing.

It may be naive of me to think that explicitly coding which members of a
class are pickled is the better way to go, provided there's an easy,
flexible and consistent framework to operate within. But that's where my
thoughts lie ~ IMO, v1.0 should absolutely not wait for this to be ground
out.


 Maybe there's an advantage there. But any "connection" with Walter stems
 from being enthusiastic, and putting as much effort into writing code as


into

 writing NG posts. It's not really anything more than that.


If that's all it takes, the myriad projects at dsource.org should be given
an equal level of "connection" then! Can they all have sections on the NG
too please! <g>


 <pedantic>"obviated"? I think you mean "exemplified". Obviate means to
 render unnecessary</pedantic>


Yeah; it's a bit annoying when one changes the wording of something, and
then can't remedy errors after posting.


 <black humour>
 As you say; the C++ streams library did indeed take a long time to




develop.

 How long does it usually take to admit that a fundamental approach is
 tragically flawed? (sorry; sorry ... just a bad joke)
 </black humour>


 Well, Java's been with us for nearly a decade ...


Too true. Even Bruce Eckel could barely contain disparaging remarks
regarding Java IO in his truly excellent book (TIJ). It just goes to show
that such things shouldn't be hurried into a standard library for a v1.0
release (re std.streams); poor Java has paid for it in spades (and hacks)
ever since, due to backward compatibility. Even NIO is lumbered.


 To be serious, for all that I really don't care for the IOStreams, they do
 have the localisation issue relatively well handled. That's not something
 to sniff at, even though it does have high performance costs.


I think it would be really helpful to expound on this particular note over
in that "IO formatting" thread. How about it Matthew? I, for one, would
likely learn something anew.


 In any case, playing football with a mango would be a huge waste. Mangos
 must be respectfully, even reverently, consumed.


How about Australian Rugby then?
Jun 23 2004
parent Sean Kelly <sean f4.ca> writes:
In article <cbdfjg$2e4b$1 digitaldaemon.com>, Kris says...

"Matthew" wrote

 Well, Java's been with us for nearly a decade ...


Too true. Even Bruce Eckel could barely contain disparaging remarks
regarding Java IO in his truly excellent book (TIJ). It just goes to show
that such things shouldn't be hurried into a standard library for a v1.0
release (re std.streams); poor Java has paid for it in spades (and hacks)
ever since, due to backward compatibility.


And don't forget the Java GUI API, which was changed completely between 1.0 and
2.0.  I think basic i/o is important to have for 1.0, but I'd much rather have
the fancy stuff come later than be stuck supporting bad decisions.

Sean
Jun 23 2004
prev sibling parent reply John Reimer <brk_6502 NOSP_AM.yahoo.com> writes:
Mango.io may not be ready to go into the D std package as of yet; any 
discussion of the sort might be considered presumptuous.  But I still 
think it should go somewhere where people will give it serious thought. 
  Of late, my general feeling is that mango.io is not getting the chance 
in needs to make an impression on the D community. To improve this maybe 
these ideas should be considered:

1) A web page could be set up for listing several D stream packages for 
D community perusal

2) the page could have official endorsement from the www.digitalmars.com 
site to encourage people to sample, comment, and peruse the competing 
stream packages.

3) Perhaps that web page should indicate: "D Stream Packages to be 
considered for official D Phobos integration"

4) Each package should be well documented and easily extensible... 
(oops.. that just eliminated a few, didn't it?)  ;-)

5) Each package should get a review in some journal, perhaps even in the 
new journal that Walter mentioned in C++.announce.

Most of these ideas depend on how serious the D community is about 
seeing a successful stream library appear.  If we are all that serious, 
then we should be pushing for better publicity for the existing packages.

I really want to see Mango.io given a fair chance.  At the very least, 
mango.io should get consideration for inclusion in the more publically 
developed demios.

Later,

John
Jun 23 2004
parent reply =?ISO-8859-1?Q?Julio_C=E9sar_Carrascal_Urquijo?= writes:
John Reimer wrote:

 (...)
 5) Each package should get a review in some journal, perhaps even in the 
 new journal that Walter mentioned in C++.announce.
 (...)


6) Included in the compiler distribution.

-- 
Julio César Carrascal Urquijo
Jun 24 2004
parent John Reimer <jjreimer telus.net> writes:
Julio César Carrascal Urquijo wrote:


 John Reimer wrote:

 (...)
 5) Each package should get a review in some journal, perhaps even in the
 new journal that Walter mentioned in C++.announce.
 (...)


 
 6) Included in the compiler distribution.
 


Good point!
Jun 24 2004