www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Phobos Wish List/Next in Review Queue?

reply dsimcha <dsimcha yahoo.com> writes:
Now that we've got a lot of contributors to Phobos and many projects in 
the works, I decided to start a thread to help us make a rough plan for 
Phobos's short-to-medium term development.  There are three goals here:

1.  Determine what's next in the review queue after std.csv (voting on 
std.csv ends tonight, so **please vote**).

2.  Come up with a wish list of high-priority modules that Phobos is 
missing that would make D a substantially more attractive language than 
it is now.

3.  Figure out who's already working on what from the wish list and what 
bottlenecks, if any, are getting in the way and what can be done about them.

The following is the wish list as I see it.  Please suggest additions 
and correct any errors, as this is mostly off the top of my head.  Also, 
status updates if you're working on any of these and anything 
substantial has changed would be appreciated.

*  Some higher level networking support, such as HTTP, FTP, etc.  (Jonas 
Drewsen's CURL wrapper handles a lot of this and may be ready for a 
second round of review.)

*  Serialization.  (Jacob Carolberg's Orange library might be a good 
candidate.  IIRC he said it's close to ready for review.)

*  Encryption and hashing.  (This is more an implementation problem than 
a design problem and AFAIK noone is working on it.)

*  Containers.  (AFAIK noone is working on this.  It's tough to get 
started because, despite lots of discussion at various times on this 
forum, noone seems to really know what they want.  Since the containers 
in question are well-known, it's much more a design problem than an 
implementation problem.)

*  Allocators.  (I think Phobos desperately needs a segmented 
stack/region based allocator and I've written one.  I've also tried to 
define a generic allocator API, mostly following Andrei's suggestions, 
but I'll admit that I didn't really know what I was doing for the 
general API.  Andrei has suggested that allocators should have 
real-world testing on containers before being included in Phobos. 
Therefore, containers block allocators and if the same person doesn't 
write both, there will be a lot of communication overhead to make sure 
the designs are in sync.)

*  Streams.  (Another item where the bottleneck is mostly at the design 
level and people not really knowing what they want.)

*  Compression/archiving.  (Opening standard compressed/archived file 
formats needs to just work.  This includes at least zip, gzip, tar and 
bzip2.  Of course, zip already is available and gzip is supported by the 
zlib module but with a crufty C API.  At least gzip and bzip2, which are 
stream-based as opposed to file-based, should be handled via streams, 
which means that streams block compression/archiving.  Also, since tar 
and zip are both file based, they should probably be handled by the same 
API, which might mean deprecating std.zip and rewriting it.)

*  An improved std.xml.  (I think Thomas Sowinski is working on a 
replacement, but I haven't seen any updates in a long time.)

*  Matrices and linear algebra.  (Cristi Cobzarenco's GSoC project is a 
good starting point but it needs polish.  I've been in contact with him 
occasionally since GSoC ended and he indicated that he wants to get back 
to working on it but doesn't have time.  I've contributed to it 
sparingly, but find it difficult because I haven't gotten around to 
familiarizing myself with the implementation details yet, and it's hard 
to get into a project that complex with a few hours a week as opposed to 
focusing full time on it.)

*  std.database.  (Apparently Steve Teale is working on this.  This is a 
large, complicated project because we're trying to define a common API 
for a variety of RDBMSs.  Again, it's more a design problem than an 
implementation problem.)

*  Better support for creating processes/new std.process.  (Lars 
Kyllingstad wrote a replacement candidate for Posix and Steve 
Schveighoffer ported it to Windows, but issues with the DMC runtime 
prevent it from working on Windows.)

*  Parallel algorithms.  (I've implemented a decent amount of these in 
my std.parallel_algorithm Github project, but I've become somewhat 
frustrated and unmotivated to finish this project because so many of the 
relevant algorithms seem memory bandwidth bound and aren't substantially 
faster when parallelized than when run serially.)

After writing this, the general pattern I notice is that lots of stuff 
is blocked by design, not implementation.  In a lot of cases people 
don't really know what they want and analysis paralysis results.
Nov 19 2011
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
dsimcha:


 I decided to start a thread to help us make a rough plan for 
 Phobos's short-to-medium term development.


Not too much time ago there was a Phobos wish list thread. I suggest you to
take a look at it (it contains my answer too, so I think there is no need to
repeat it here).



 2.  Come up with a wish list of high-priority modules that Phobos is 
 missing that would make D a substantially more attractive language than 
 it is now.


It's not just a matter of modules, but also of few functions to add to existing
modules. I have some of those.
Another thing that will improve D attractiveness for people that try D for the
fist time is to find ways to make more Phobos functions work with
const/immutable input data (like string functions).

Bye,
bearophile
Nov 19 2011
prev sibling next sibling parent Gour <gour atmarama.net> writes:
On Sat, 19 Nov 2011 22:02:33 -0500
dsimcha <dsimcha yahoo.com> wrote:


 2.  Come up with a wish list of high-priority modules that Phobos is=20
 missing that would make D a substantially more attractive language
 than it is now.


That's  probably not for the Phobos, but having clear roadmap for the
GUI bindings projects (dwt2, gtkd, qtd, wxd,...) would definitely, imho,
make D more attractive language by allowing developers to jump into
writing GUI apps.


 *  Containers.  (AFAIK noone is working on this.  It's tough to get=20
 started because, despite lots of discussion at various times on this=20
 forum, noone seems to really know what they want.  Since the
 containers in question are well-known, it's much more a design
 problem than an implementation problem.)


+1


 *  Compression/archiving.  (Opening standard compressed/archived file=20
 formats needs to just work.  This includes at least zip, gzip, tar
 and bzip2.  Of course, zip already is available and gzip is supported
 by the zlib module but with a crufty C API.  At least gzip and bzip2,
 which are stream-based as opposed to file-based, should be handled
 via streams, which means that streams block compression/archiving.
 Also, since tar and zip are both file based, they should probably be
 handled by the same API, which might mean deprecating std.zip and
 rewriting it.)


+1


 *  std.database.  (Apparently Steve Teale is working on this.  This
 is a large, complicated project because we're trying to define a
 common API for a variety of RDBMSs. =20


+1



Sincerely,
Gour


--=20
A person who is not disturbed by the incessant flow of=20
desires =E2=80=94 that enter like rivers into the ocean, which is=20
ever being filled but is always still =E2=80=94 can alone achieve=20
peace, and not the man who strives to satisfy such desires.

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810
Nov 19 2011
prev sibling next sibling parent reply Johannes Pfau <spam example.com> writes:
dsimcha wrote:

Now that we've got a lot of contributors to Phobos and many projects
in the works, I decided to start a thread to help us make a rough plan
for Phobos's short-to-medium term development.  There are three goals
here:

1.  Determine what's next in the review queue after std.csv (voting on 
std.csv ends tonight, so **please vote**).

2.  Come up with a wish list of high-priority modules that Phobos is 
missing that would make D a substantially more attractive language
than it is now.

3.  Figure out who's already working on what from the wish list and
what bottlenecks, if any, are getting in the way and what can be done
about them.

The following is the wish list as I see it.  Please suggest additions 
and correct any errors, as this is mostly off the top of my head.
Also, status updates if you're working on any of these and anything 
substantial has changed would be appreciated.

*  Some higher level networking support, such as HTTP, FTP, etc.
(Jonas Drewsen's CURL wrapper handles a lot of this and may be ready
for a second round of review.)

*  Serialization.  (Jacob Carolberg's Orange library might be a good 
candidate.  IIRC he said it's close to ready for review.)

*  Encryption and hashing.  (This is more an implementation problem
than a design problem and AFAIK noone is working on it.)

*  Containers.  (AFAIK noone is working on this.  It's tough to get 
started because, despite lots of discussion at various times on this 
forum, noone seems to really know what they want.  Since the
containers in question are well-known, it's much more a design problem
than an implementation problem.)

*  Allocators.  (I think Phobos desperately needs a segmented 
stack/region based allocator and I've written one.  I've also tried to 
define a generic allocator API, mostly following Andrei's suggestions, 
but I'll admit that I didn't really know what I was doing for the 
general API.  Andrei has suggested that allocators should have 
real-world testing on containers before being included in Phobos. 
Therefore, containers block allocators and if the same person doesn't 
write both, there will be a lot of communication overhead to make sure 
the designs are in sync.)

*  Streams.  (Another item where the bottleneck is mostly at the
design level and people not really knowing what they want.)

*  Compression/archiving.  (Opening standard compressed/archived file 
formats needs to just work.  This includes at least zip, gzip, tar and 
bzip2.  Of course, zip already is available and gzip is supported by
the zlib module but with a crufty C API.  At least gzip and bzip2,
which are stream-based as opposed to file-based, should be handled via
streams, which means that streams block compression/archiving.  Also,
since tar and zip are both file based, they should probably be handled
by the same API, which might mean deprecating std.zip and rewriting
it.)


AFAIK almost all compression (and encryption) algorithms work like
streams. The directory structure is often simply added on top of the
streaming functionality (at least true for all .tar.* formats).

BTW: The unfinished stream API blocks lots of projects: Encryption,
Compression, the curl wrapper could be integrated with streams, http
servers and almost everything else related to I/O.


*  An improved std.xml.  (I think Thomas Sowinski is working on a 
replacement, but I haven't seen any updates in a long time.)

*  Matrices and linear algebra.  (Cristi Cobzarenco's GSoC project is
a good starting point but it needs polish.  I've been in contact with
him occasionally since GSoC ended and he indicated that he wants to
get back to working on it but doesn't have time.  I've contributed to
it sparingly, but find it difficult because I haven't gotten around to 
familiarizing myself with the implementation details yet, and it's
hard to get into a project that complex with a few hours a week as
opposed to focusing full time on it.)

*  std.database.  (Apparently Steve Teale is working on this.  This is
a large, complicated project because we're trying to define a common
API for a variety of RDBMSs.  Again, it's more a design problem than
an implementation problem.)

*  Better support for creating processes/new std.process.  (Lars 
Kyllingstad wrote a replacement candidate for Posix and Steve 
Schveighoffer ported it to Windows, but issues with the DMC runtime 
prevent it from working on Windows.)

*  Parallel algorithms.  (I've implemented a decent amount of these in 
my std.parallel_algorithm Github project, but I've become somewhat 
frustrated and unmotivated to finish this project because so many of
the relevant algorithms seem memory bandwidth bound and aren't
substantially faster when parallelized than when run serially.)


Inter-process communication, but as we have thrift now, that might be
ok. (Extending std.concurrency message passing to interprocess
communication with a low overhead serializer like messagepack would
still be great though)

Logging: AFAIK the new log module should be almost ready. I already
used it a few times, but you have to a patch druntime to use
it.

High performance IO: A reactor / proactor framework

A replacement for std.json


After writing this, the general pattern I notice is that lots of stuff 
is blocked by design, not implementation.  In a lot of cases people 
don't really know what they want and analysis paralysis results.



-- 
Johannes Pfau
Nov 20 2011
next sibling parent =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 20-11-2011 11:48, Johannes Pfau wrote:

 dsimcha wrote:

 Now that we've got a lot of contributors to Phobos and many projects
 in the works, I decided to start a thread to help us make a rough plan
 for Phobos's short-to-medium term development.  There are three goals
 here:

 1.  Determine what's next in the review queue after std.csv (voting on
 std.csv ends tonight, so **please vote**).

 2.  Come up with a wish list of high-priority modules that Phobos is
 missing that would make D a substantially more attractive language
 than it is now.

 3.  Figure out who's already working on what from the wish list and
 what bottlenecks, if any, are getting in the way and what can be done
 about them.

 The following is the wish list as I see it.  Please suggest additions
 and correct any errors, as this is mostly off the top of my head.
 Also, status updates if you're working on any of these and anything
 substantial has changed would be appreciated.

 *  Some higher level networking support, such as HTTP, FTP, etc.
 (Jonas Drewsen's CURL wrapper handles a lot of this and may be ready
 for a second round of review.)

 *  Serialization.  (Jacob Carolberg's Orange library might be a good
 candidate.  IIRC he said it's close to ready for review.)

 *  Encryption and hashing.  (This is more an implementation problem
 than a design problem and AFAIK noone is working on it.)

 *  Containers.  (AFAIK noone is working on this.  It's tough to get
 started because, despite lots of discussion at various times on this
 forum, noone seems to really know what they want.  Since the
 containers in question are well-known, it's much more a design problem
 than an implementation problem.)

 *  Allocators.  (I think Phobos desperately needs a segmented
 stack/region based allocator and I've written one.  I've also tried to
 define a generic allocator API, mostly following Andrei's suggestions,
 but I'll admit that I didn't really know what I was doing for the
 general API.  Andrei has suggested that allocators should have
 real-world testing on containers before being included in Phobos.
 Therefore, containers block allocators and if the same person doesn't
 write both, there will be a lot of communication overhead to make sure
 the designs are in sync.)

 *  Streams.  (Another item where the bottleneck is mostly at the
 design level and people not really knowing what they want.)

 *  Compression/archiving.  (Opening standard compressed/archived file
 formats needs to just work.  This includes at least zip, gzip, tar and
 bzip2.  Of course, zip already is available and gzip is supported by
 the zlib module but with a crufty C API.  At least gzip and bzip2,
 which are stream-based as opposed to file-based, should be handled via
 streams, which means that streams block compression/archiving.  Also,
 since tar and zip are both file based, they should probably be handled
 by the same API, which might mean deprecating std.zip and rewriting
 it.)


 AFAIK almost all compression (and encryption) algorithms work like
 streams. The directory structure is often simply added on top of the
 streaming functionality (at least true for all .tar.* formats).

 BTW: The unfinished stream API blocks lots of projects: Encryption,
 Compression, the curl wrapper could be integrated with streams, http
 servers and almost everything else related to I/O.


 *  An improved std.xml.  (I think Thomas Sowinski is working on a
 replacement, but I haven't seen any updates in a long time.)

 *  Matrices and linear algebra.  (Cristi Cobzarenco's GSoC project is
 a good starting point but it needs polish.  I've been in contact with
 him occasionally since GSoC ended and he indicated that he wants to
 get back to working on it but doesn't have time.  I've contributed to
 it sparingly, but find it difficult because I haven't gotten around to
 familiarizing myself with the implementation details yet, and it's
 hard to get into a project that complex with a few hours a week as
 opposed to focusing full time on it.)

 *  std.database.  (Apparently Steve Teale is working on this.  This is
 a large, complicated project because we're trying to define a common
 API for a variety of RDBMSs.  Again, it's more a design problem than
 an implementation problem.)

 *  Better support for creating processes/new std.process.  (Lars
 Kyllingstad wrote a replacement candidate for Posix and Steve
 Schveighoffer ported it to Windows, but issues with the DMC runtime
 prevent it from working on Windows.)

 *  Parallel algorithms.  (I've implemented a decent amount of these in
 my std.parallel_algorithm Github project, but I've become somewhat
 frustrated and unmotivated to finish this project because so many of
 the relevant algorithms seem memory bandwidth bound and aren't
 substantially faster when parallelized than when run serially.)


 Inter-process communication, but as we have thrift now, that might be
 ok. (Extending std.concurrency message passing to interprocess
 communication with a low overhead serializer like messagepack would
 still be great though)


It would be awesome if we could extend it enough to work over TCP or 
something similar.


 Logging: AFAIK the new log module should be almost ready. I already
 used it a few times, but you have to a patch druntime to use
 it.

 High performance IO: A reactor / proactor framework

 A replacement for std.json


 After writing this, the general pattern I notice is that lots of stuff
 is blocked by design, not implementation.  In a lot of cases people
 don't really know what they want and analysis paralysis results.




- Alex
Nov 20 2011
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Johannes Pfau Wrote:

 High performance IO: A reactor / proactor framework


Oh that reminds me, I'd like a nice thing for networking too, where you
can do something like register callback functions for events and it
handles the rest for you.


 A replacement for std.json


Blah, that will break a lot of stuff. There is one outstanding proposal in the
wild though, by Robert Jacques, and I think I'd like that.

But, at the same time, I actually use std.json and it's pretty frustrating to
have
the library developers break things so egregiously every other release.
Nov 20 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-11-20 15:28, Adam D. Ruppe wrote:

 Johannes Pfau Wrote:

 High performance IO: A reactor / proactor framework


 Oh that reminds me, I'd like a nice thing for networking too, where you
 can do something like register callback functions for events and it
 handles the rest for you.


 A replacement for std.json


 Blah, that will break a lot of stuff. There is one outstanding proposal in the
 wild though, by Robert Jacques, and I think I'd like that.

 But, at the same time, I actually use std.json and it's pretty frustrating to
have
 the library developers break things so egregiously every other release.


If you know a module will break you can keep your own copy and decide if 
you want to migrate to the new module or keep the old one.

-- 
/Jacob Carlborg
Nov 20 2011
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Jacob Carlborg Wrote:

 If you know a module will break you can keep your own copy and decide if 
 you want to migrate to the new module or keep the old one.


Encouraging countless forks probably isn't a good idea toward building
a coordinated community.
Nov 20 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-11-20 18:14, Adam D. Ruppe wrote:

 Jacob Carlborg Wrote:

 If you know a module will break you can keep your own copy and decide if
 you want to migrate to the new module or keep the old one.


 Encouraging countless forks probably isn't a good idea toward building
 a coordinated community.


No, but I don't mean you should for the whole Phobos, just keep your own 
copy of the module in the particular application that uses it.

-- 
/Jacob Carlborg
Nov 20 2011
prev sibling parent Tobias Pankrath <tobias pankrath.net> writes:
 
 If you know a module will break you can keep your own copy and decide if
 you want to migrate to the new module or keep the old one.
 


Instead of introducing big and breaking changes to a module, we cood keep 
std.json and introduce std.json2.
Nov 20 2011
prev sibling parent reply Somedude <lovelydear mailmetrash.com> writes:
Le 20/11/2011 15:28, Adam D. Ruppe a écrit :

 Johannes Pfau Wrote:

 High performance IO: A reactor / proactor framework


 
 Oh that reminds me, I'd like a nice thing for networking too, where you
 can do something like register callback functions for events and it
 handles the rest for you.


In Python, the main framework for that is called Twisted. I for one
would like a ZeroMQ binding.
Nov 23 2011
parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Wed, 23 Nov 2011 14:15:39 +0100, Somedude <lovelydear mailmetrash.com=

  =


wrote:


 Le 20/11/2011 15:28, Adam D. Ruppe a =C3=A9crit :

 Johannes Pfau Wrote:

 High performance IO: A reactor / proactor framework


 Oh that reminds me, I'd like a nice thing for networking too, where y=




ou

 can do something like register callback functions for events and it
 handles the rest for you.


 In Python, the main framework for that is called Twisted. I for one
 would like a ZeroMQ binding.


https://github.com/D-Programming-Deimos/ZeroMQ
Nov 23 2011
prev sibling parent reply Jimmy Cao <jcao219 gmail.com> writes:
2011/11/20 Johannes Pfau <spam example.com>

 A replacement for std.json


The biggest thing impairing std.json imo is this bug:
http://d.puremagic.com/issues/show_bug.cgi?id=2962
Nov 20 2011
parent Adam D. Ruppe <destructionator gmail.com> writes:
Jimmy Cao Wrote:

 The biggest thing impairing std.json imo is this bug:
 http://d.puremagic.com/issues/show_bug.cgi?id=2962


Note you can work around that with this.

Create a file called icehack.d:
module icehack;
import std.json;
static if(__traits(compiles, parseJSON("hello"))) {}


then on your compile command line, put this first:

dmd icehack.d [the rest of your arguments]


and it works. Really annoying but fairly simple workaround
anyway.
Nov 20 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-11-20 04:02, dsimcha wrote:

 Now that we've got a lot of contributors to Phobos and many projects in
 the works, I decided to start a thread to help us make a rough plan for
 Phobos's short-to-medium term development. There are three goals here:

 1. Determine what's next in the review queue after std.csv (voting on
 std.csv ends tonight, so **please vote**).


Isn't it better to add a list on the wiki so we don't have to do this 
all over yet again.


 2. Come up with a wish list of high-priority modules that Phobos is
 missing that would make D a substantially more attractive language than
 it is now.


I would really like to see high level networking support, something like 
the CURL wrapper.


 * Serialization. (Jacob Carolberg's Orange library might be a good
 candidate. IIRC he said it's close to ready for review.)


This is ready for (pre-)review and have been that for quite a while now.


 * Better support for creating processes/new std.process. (Lars
 Kyllingstad wrote a replacement candidate for Posix and Steve
 Schveighoffer ported it to Windows, but issues with the DMC runtime
 prevent it from working on Windows.)


Wasn't there a fix for the DMC runtime? In that case, why isn't it 
already applied?

-- 
/Jacob Carlborg
Nov 20 2011
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sun, 20 Nov 2011 08:53:27 -0500, Jacob Carlborg <doob me.com> wrote:


 On 2011-11-20 04:02, dsimcha wrote:

 * Better support for creating processes/new std.process. (Lars
 Kyllingstad wrote a replacement candidate for Posix and Steve
 Schveighoffer ported it to Windows, but issues with the DMC runtime
 prevent it from working on Windows.)


 Wasn't there a fix for the DMC runtime? In that case, why isn't it  
 already applied?


There is a fix.  I've sent it to Walter in the form of "replace this file  
with the attached version".  I tested the changes by replacing the object  
file in the snn.lib library.

Walter has since given me github access to the DMC runtime source, and I  
have not been able to successfully build the runtime due to dependence on  
obscure assembler tools that Walter has not offered, nor can I download  
them.  I'm hesitant to submit a pull request for something I can't  
successfully build or test from scratch, but I may just submit the  
original changes as a pull-request anyways to get the ball rolling.

-Steve
Nov 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-11-21 16:30, Steven Schveighoffer wrote:

 On Sun, 20 Nov 2011 08:53:27 -0500, Jacob Carlborg <doob me.com> wrote:


 On 2011-11-20 04:02, dsimcha wrote:

 * Better support for creating processes/new std.process. (Lars
 Kyllingstad wrote a replacement candidate for Posix and Steve
 Schveighoffer ported it to Windows, but issues with the DMC runtime
 prevent it from working on Windows.)


 Wasn't there a fix for the DMC runtime? In that case, why isn't it
 already applied?


 There is a fix. I've sent it to Walter in the form of "replace this file
 with the attached version". I tested the changes by replacing the object
 file in the snn.lib library.

 Walter has since given me github access to the DMC runtime source, and I
 have not been able to successfully build the runtime due to dependence
 on obscure assembler tools that Walter has not offered, nor can I
 download them. I'm hesitant to submit a pull request for something I
 can't successfully build or test from scratch, but I may just submit the
 original changes as a pull-request anyways to get the ball rolling.

 -Steve


Ok, I see.

-- 
/Jacob Carlborg
Nov 21 2011
prev sibling next sibling parent reply Mike Wey <mike-wey example.com> writes:
On 11/20/2011 04:02 AM, dsimcha wrote:

 * Streams. (Another item where the bottleneck is mostly at the design
 level and people not really knowing what they want.)


Steven Schveighoffer's new-stdio included a nice stream interface.
I don't know what the current state of the module is.

-- 
Mike Wey
Nov 20 2011
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sun, 20 Nov 2011 09:07:44 -0500, Mike Wey <mike-wey example.com> wrote:


 On 11/20/2011 04:02 AM, dsimcha wrote:

 * Streams. (Another item where the bottleneck is mostly at the design
 level and people not really knowing what they want.)


 Steven Schveighoffer's new-stdio included a nice stream interface.
 I don't know what the current state of the module is.


I'm slowly working on it.  I've had very little time these last few months  
to work on any extraneous projects.

However, I know the direction this needs to move, and the work is  
straightforward.  When I get something worth sharing, I'll share it.

Sorry I couldn't be more forthcoming.  If you follow my github account,  
you should see the new-io branch start to get new commits in the near  
future.  I really need to get working on this, before too much is built on  
top of the existing std.stdio.  The new stream interface should be a boon  
to most libraries (such as std.xml).

-Steve
Nov 21 2011
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
dsimcha Wrote:

 *  Compression/archiving.  (Opening standard compressed/archived file 
 formats needs to just work.  This includes at least zip, gzip, tar and 
 bzip2.  Of course, zip already is available and gzip is supported by the 
 zlib module but with a crufty C API.


I did a patch a couple months ago to let you get gzip through the D
wrapper too.

Nothing fancy (about a ten line patch), but it works and got pulled
in, so we have something for this.

http://www.d-programming-language.org/phobos/std_zlib.html#HeaderFormat
Nov 20 2011
prev sibling next sibling parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
dsimcha wrote:

 * Encryption and hashing. (This is more an implementation problem than a
 design problem and AFAIK noone is working on it.)


There were some attempts to do so. See "Early std.crypto" thread.
Nov 20 2011
prev sibling next sibling parent reply Jonas Drewsen <jdrewsen nospam.com> writes:
Den 20-11-2011 04:02, dsimcha skrev:

 Now that we've got a lot of contributors to Phobos and many projects in
 the works, I decided to start a thread to help us make a rough plan for
 Phobos's short-to-medium term development. There are three goals here:

 1. Determine what's next in the review queue after std.csv (voting on
 std.csv ends tonight, so **please vote**).

 2. Come up with a wish list of high-priority modules that Phobos is
 missing that would make D a substantially more attractive language than
 it is now.

 3. Figure out who's already working on what from the wish list and what
 bottlenecks, if any, are getting in the way and what can be done about
 them.

 The following is the wish list as I see it. Please suggest additions and
 correct any errors, as this is mostly off the top of my head. Also,
 status updates if you're working on any of these and anything
 substantial has changed would be appreciated.

 * Some higher level networking support, such as HTTP, FTP, etc. (Jonas
 Drewsen's CURL wrapper handles a lot of this and may be ready for a
 second round of review.)


As I've mentioned in another thread it is ready for a second round of 
review. We just need someone to step up and run the review since it 
wouldn't be apropriate for me to do it myself. Anyone?


 * Serialization. (Jacob Carolberg's Orange library might be a good
 candidate. IIRC he said it's close to ready for review.)

 * Encryption and hashing. (This is more an implementation problem than a
 design problem and AFAIK noone is working on it.)

 * Containers. (AFAIK noone is working on this. It's tough to get started
 because, despite lots of discussion at various times on this forum,
 noone seems to really know what they want. Since the containers in
 question are well-known, it's much more a design problem than an
 implementation problem.)

 * Allocators. (I think Phobos desperately needs a segmented stack/region
 based allocator and I've written one. I've also tried to define a
 generic allocator API, mostly following Andrei's suggestions, but I'll
 admit that I didn't really know what I was doing for the general API.
 Andrei has suggested that allocators should have real-world testing on
 containers before being included in Phobos. Therefore, containers block
 allocators and if the same person doesn't write both, there will be a
 lot of communication overhead to make sure the designs are in sync.)


I've though about doing some containers myself but have hesitated since 
the general opinion seem to be that allocators need to be in place first.


 * Streams. (Another item where the bottleneck is mostly at the design
 level and people not really knowing what they want.)


What does streams provide that could not be provided by ranges?

/Jonas
Nov 20 2011
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
On 11/20/2011 12:30 PM, Jonas Drewsen wrote:

 * Containers. (AFAIK noone is working on this. It's tough to get started
 because, despite lots of discussion at various times on this forum,
 noone seems to really know what they want. Since the containers in
 question are well-known, it's much more a design problem than an
 implementation problem.)


  >

 * Allocators. (I think Phobos desperately needs a segmented stack/region
 based allocator and I've written one. I've also tried to define a
 generic allocator API, mostly following Andrei's suggestions, but I'll
 admit that I didn't really know what I was doing for the general API.
 Andrei has suggested that allocators should have real-world testing on
 containers before being included in Phobos. Therefore, containers block
 allocators and if the same person doesn't write both, there will be a
 lot of communication overhead to make sure the designs are in sync.)


 I've though about doing some containers myself but have hesitated since
 the general opinion seem to be that allocators need to be in place first.


Yeah, this is problematic.  In voting against my allocator proposal, 
Andrei mentioned that he wanted the allocators to be well-tested in the 
container API first.  This means either we have a circular dependency or 
allocators and containers need to be co-developed.  Co-developing them 
is problematic.  If one person does containers and another allocators, 
the project might be overwhelmed by communication overhead.  If the same 
person does both, then this is asking a pretty lot for a hobby project.

Of course, I hope to graduate in <1 year and will be looking for a job 
when I do.  Any company out there have a strategic interest in D and 
want to hire me to work full-time on allocators and containers?


 * Streams. (Another item where the bottleneck is mostly at the design
 level and people not really knowing what they want.)


 What does streams provide that could not be provided by ranges?


If I understand correctly, streams _would_ be a flavor of ranges.  They 
would just be ranges that are geared towards being stacked on top of one 
another specifically for the purpose of I/O.  They would typically be 
design around the vanilla input range (not forward, random access, etc.) 
or output ranges.

Traditionally, streams would also be class based instead of template 
based.  However, IMHO a good case can be made for template based stream 
ranges in D because we have std.range.inputRangeObject and 
std.range.outputRangeObject.  This means that you can stack streams 
using templates, with no virtual function call overhead, and then if you 
need to respect some binary interface you could just stack an 
inputRangeObject() or outputRangeObject() on top of all your other crap 
and only have one virtual call.  Example:

auto lines = lineReader(
     gzipUncompresser(
         rawFile("foo.gz")
     )
);

// LineReader!(GzipUncompresser!(RawFile)))
pragma(msg, typeof(lines));

auto objectOriented = inputRangeObject(lines);

// InputRangeObject!(char[])
pragma(msg, typeof(objectOriented));
Nov 20 2011
parent "Martin Nowak" <dawg dawgfoto.de> writes:
 Yeah, this is problematic.  In voting against my allocator proposal,  
 Andrei mentioned that he wanted the allocators to be well-tested in the  
 container API first.  This means either we have a circular dependency or  
 allocators and containers need to be co-developed.  Co-developing them  
 is problematic.  If one person does containers and another allocators,  
 the project might be overwhelmed by communication overhead.  If the same  
 person does both, then this is asking a pretty lot for a hobby project.


There was the idea of a std.experimental (or stdexp) package floating  
around some time ago.
If I remember correctly it was found to be a useful addition.
Why not adding the current allocator interface with say a GC-only  
implementation there
so that container development can start off and provide the necessary  
feedback.

martin
Nov 23 2011
prev sibling parent reply dsimcha <dsimcha yahoo.com> writes:
On 11/20/2011 12:30 PM, Jonas Drewsen wrote:

 * Some higher level networking support, such as HTTP, FTP, etc. (Jonas
 Drewsen's CURL wrapper handles a lot of this and may be ready for a
 second round of review.)


 As I've mentioned in another thread it is ready for a second round of
 review. We just need someone to step up and run the review since it
 wouldn't be apropriate for me to do it myself. Anyone?


If noone else wants to volunteer, I will again.  Is there something I'm 
missing?  I find that being review manage takes very little effort: 
Post an initial review announcement, post a reminder, maybe post a 
summary/moderation message here and there, post a vote message, post a 
vote reminder, tally votes.  It really doesn't take much time.
Nov 20 2011
parent reply Jesse Phillips <jessekphillips+d gmail.com> writes:
On Sun, 20 Nov 2011 15:12:27 -0500, dsimcha wrote:


 On 11/20/2011 12:30 PM, Jonas Drewsen wrote:
 If noone else wants to volunteer, I will again.  Is there something I'm
 missing?  I find that being review manage takes very little effort: Post
 an initial review announcement, post a reminder, maybe post a
 summary/moderation message here and there, post a vote message, post a
 vote reminder, tally votes.  It really doesn't take much time.


I think there is at least one thing missing from that:

http://www.boost.org/community/reviews.html#Review_Manager

Making sure it is ready for review :D. There is also "If necessary, work 
with the submitter to verify the code compiles and runs correctly on 
several compilers and platforms." Not that you must do it yourself, or 
that we are collaborating with GDC and LDC yet.
Nov 20 2011
parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 20-11-2011 22:19, Jesse Phillips wrote:

 On Sun, 20 Nov 2011 15:12:27 -0500, dsimcha wrote:


 On 11/20/2011 12:30 PM, Jonas Drewsen wrote:
 If noone else wants to volunteer, I will again.  Is there something I'm
 missing?  I find that being review manage takes very little effort: Post
 an initial review announcement, post a reminder, maybe post a
 summary/moderation message here and there, post a vote message, post a
 vote reminder, tally votes.  It really doesn't take much time.


 I think there is at least one thing missing from that:

 http://www.boost.org/community/reviews.html#Review_Manager

 Making sure it is ready for review :D. There is also "If necessary, work
 with the submitter to verify the code compiles and runs correctly on
 several compilers and platforms." Not that you must do it yourself, or
 that we are collaborating with GDC and LDC yet.


Which, by the way, is a shame. druntime may be compiler-specific, but 
Phobos is the D standard library and, as such, should definitely compile 
on all mainstream compilers. This ought to be part of the review 
process. The problem with doing it, though, is that currently even 
Phobos is merged manually into GDC, and LDC maintains a fork...

- Alex
Nov 20 2011
prev sibling next sibling parent Steve Teale <steve.teale britseyeview.com> writes:
On Sat, 19 Nov 2011 22:02:33 -0500, dsimcha wrote:

 
 *  std.database.  (Apparently Steve Teale is working on this.  This is a
 large, complicated project because we're trying to define a common API
 for a variety of RDBMSs.  Again, it's more a design problem than an
 implementation problem.)
 


This is blocked in both directions. There's little in the way of 
agreement on design, and there are implementation difficulties caused by 
licensing issues and the fact that some potentially useful approaches 
don't appear to work - ODBC for example is a nightmare.

I am trying to sort out what is possible and what isn't as quickly as 
possible. Then when the current massive web site debate has wound down I 
might try to get people to return to the design issues.

Steve
Nov 21 2011
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On 20 November 2011 05:02, dsimcha <dsimcha yahoo.com> wrote:


 Now that we've got a lot of contributors to Phobos and many projects in
 the works, I decided to start a thread to help us make a rough plan for
 Phobos's short-to-medium term development.  There are three goals here:

 1.  Determine what's next in the review queue after std.csv (voting on
 std.csv ends tonight, so **please vote**).

 2.  Come up with a wish list of high-priority modules that Phobos is
 missing that would make D a substantially more attractive language than it
 is now.

 3.  Figure out who's already working on what from the wish list and what
 bottlenecks, if any, are getting in the way and what can be done about them.

 The following is the wish list as I see it.  Please suggest additions and
 correct any errors, as this is mostly off the top of my head.  Also, status
 updates if you're working on any of these and anything substantial has
 changed would be appreciated.

 *  Some higher level networking support, such as HTTP, FTP, etc.  (Jonas
 Drewsen's CURL wrapper handles a lot of this and may be ready for a second
 round of review.)

 *  Serialization.  (Jacob Carolberg's Orange library might be a good
 candidate.  IIRC he said it's close to ready for review.)

 *  Encryption and hashing.  (This is more an implementation problem than a
 design problem and AFAIK noone is working on it.)

 *  Containers.  (AFAIK noone is working on this.  It's tough to get
 started because, despite lots of discussion at various times on this forum,
 noone seems to really know what they want.  Since the containers in
 question are well-known, it's much more a design problem than an
 implementation problem.)

 *  Allocators.  (I think Phobos desperately needs a segmented stack/region
 based allocator and I've written one.  I've also tried to define a generic
 allocator API, mostly following Andrei's suggestions, but I'll admit that I
 didn't really know what I was doing for the general API.  Andrei has
 suggested that allocators should have real-world testing on containers
 before being included in Phobos. Therefore, containers block allocators and
 if the same person doesn't write both, there will be a lot of communication
 overhead to make sure the designs are in sync.)

 *  Streams.  (Another item where the bottleneck is mostly at the design
 level and people not really knowing what they want.)

 *  Compression/archiving.  (Opening standard compressed/archived file
 formats needs to just work.  This includes at least zip, gzip, tar and
 bzip2.  Of course, zip already is available and gzip is supported by the
 zlib module but with a crufty C API.  At least gzip and bzip2, which are
 stream-based as opposed to file-based, should be handled via streams, which
 means that streams block compression/archiving.  Also, since tar and zip
 are both file based, they should probably be handled by the same API, which
 might mean deprecating std.zip and rewriting it.)

 *  An improved std.xml.  (I think Thomas Sowinski is working on a
 replacement, but I haven't seen any updates in a long time.)

 *  Matrices and linear algebra.  (Cristi Cobzarenco's GSoC project is a
 good starting point but it needs polish.  I've been in contact with him
 occasionally since GSoC ended and he indicated that he wants to get back to
 working on it but doesn't have time.  I've contributed to it sparingly, but
 find it difficult because I haven't gotten around to familiarizing myself
 with the implementation details yet, and it's hard to get into a project
 that complex with a few hours a week as opposed to focusing full time on
 it.)

 *  std.database.  (Apparently Steve Teale is working on this.  This is a
 large, complicated project because we're trying to define a common API for
 a variety of RDBMSs.  Again, it's more a design problem than an
 implementation problem.)

 *  Better support for creating processes/new std.process.  (Lars
 Kyllingstad wrote a replacement candidate for Posix and Steve Schveighoffer
 ported it to Windows, but issues with the DMC runtime prevent it from
 working on Windows.)

 *  Parallel algorithms.  (I've implemented a decent amount of these in my
 std.parallel_algorithm Github project, but I've become somewhat frustrated
 and unmotivated to finish this project because so many of the relevant
 algorithms seem memory bandwidth bound and aren't substantially faster when
 parallelized than when run serially.)

 After writing this, the general pattern I notice is that lots of stuff is
 blocked by design, not implementation.  In a lot of cases people don't
 really know what they want and analysis paralysis results.


As a console/embedded games developer, my top 4:

*Allocators* - +1 .. GC is lovely to have, but in many cases, I need strict
control over my memory management. Any libraries that use the GC internally
are potentially unusable to me in some applications.

*Low level/hardware vector-maths library* - This is kinda blocked by the
lack of support for an intrinsic vector type in the compiler... the
language needs at very least a 128bit vector type to express the registers
and make it possible to allocate them in structures, and pass them between
functions. Architecture specific compilers/libraries can then build on top
of that, and then ideally, a low level maths library in Phobos might
collate various architectures vector hardware support into a common API.
I would happily volunteer to to handle this, but I need at least a 128bit
hardware vector type in the language first (GDC has this though proprietary
extension, rather, the language should define a standard). I guess I could
design the API without it, but the library would only really work
efficiently on flavours of GCC.

*Crypto *- +1 as mentioned above, blocked on streams...

*Higher level maths/signal processing *- The linear algebra library
mentioned in a previous post might be very useful, but also standard
algorithms like fourier transforms. Library implementations of these would
be amazing, potentially making use of hardware back ends where possible.
Nov 23 2011
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
 *Crypto *- +1 as mentioned above, blocked on streams...

 *Higher level maths/signal processing *- The linear algebra library
 mentioned in a previous post might be very useful, but also standard
 algorithms like fourier transforms. Library implementations of these  
 would


http://www.d-programming-language.org/phobos/std_numeric.html#Fft
The performance is actually quite good. I'd only wish for a real to
half-spectrum specialization (probably in-place) performance wise.


 be amazing, potentially making use of hardware back ends where possible.
Nov 23 2011
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
Oh wow, awesome! :)
I'm always surprised by the state of the D libraries.. There's already lots
of awesome obscure things are in there, but also completely obvious major
features blatantly missing :)

So how about a DCT? Image/video processing... Perhaps a simple adaptation
of FFT?

And it's not clear to me at all whether that class will work with fixed
point data... That's extremely important for a class like this (but
granted, not really in line with D's take on numeric precision).
Almost all audio and video data is in fixed point. Int->float conversion is
very expensive on many platforms.

On 23 November 2011 17:34, Martin Nowak <dawg dawgfoto.de> wrote:


 *Crypto *- +1 as mentioned above, blocked on streams...

 *Higher level maths/signal processing *- The linear algebra library

 mentioned in a previous post might be very useful, but also standard
 algorithms like fourier transforms. Library implementations of these would


 http://www.d-programming-**language.org/phobos/std_**numeric.html#Fft<http://www.d-programming-language.org/phobos/std_numeric.html#Fft>
 The performance is actually quite good. I'd only wish for a real to
 half-spectrum specialization (probably in-place) performance wise.


  be amazing, potentially making use of hardware back ends where possible.







 Nov 23 2011



prev sibling next sibling parent  "Martin Nowak" <dawg dawgfoto.de>  writes:


On Wed, 23 Nov 2011 18:20:35 +0100, Manu <turkeyman gmail.com> wrote:


 Oh wow, awesome! :)
 I'm always surprised by the state of the D libraries.. There's already  
 lots
 of awesome obscure things are in there, but also completely obvious major
 features blatantly missing :)


There are some real gems, e.g. findRoot.

 So how about a DCT? Image/video processing... Perhaps a simple adaptation
 of FFT?


Too many different DCTs, too many different needs I'd say.


 And it's not clear to me at all whether that class will work with fixed
 point data... That's extremely important for a class like this (but
 granted, not really in line with D's take on numeric precision).
 Almost all audio and video data is in fixed point. Int->float conversion  
 is
 very expensive on many platforms.


Too specialized. Given the obscurity of doing a fixed point FFT, the actual
implementation will be very tough, the log2N gain of calculating  
intermediate
results is directly opposed to precision.

It would be interesting to offer vectorized 'Integral <-> Float'  
conversions.


 On 23 November 2011 17:34, Martin Nowak <dawg dawgfoto.de> wrote:


 *Crypto *- +1 as mentioned above, blocked on streams...

 *Higher level maths/signal processing *- The linear algebra library

 mentioned in a previous post might be very useful, but also standard
 algorithms like fourier transforms. Library implementations of these  
 would


 http://www.d-programming-**language.org/phobos/std_**numeric.html#Fft<http://www.d-programming-language.org/phobos/std_numeric.html#Fft>
 The performance is actually quite good. I'd only wish for a real to
 half-spectrum specialization (probably in-place) performance wise.


  be amazing, potentially making use of hardware back ends where  
 possible.









 Nov 23 2011



prev sibling next sibling parent reply Jude Young <10equals2 gmail.com>  writes:


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/23/2011 11:20 AM, Manu wrote:

 Oh wow, awesome! :) I'm always surprised by the state of the D
 libraries.. There's already lots of awesome obscure things are in
 there, but also completely obvious major features blatantly missing
 :)
 
 So how about a DCT? Image/video processing... Perhaps a simple
 adaptation of FFT?
 


I'm currently working on getting as many bindings into Deimos that I can.
Mostly interesting things like ZeroMQ, I also have a good Binding for
CZMQ that needs to get into Deimos.  If you wanted something
higher-level than ZeroMQ, the project is here:
https://github.com/1100110/CZMQ


If you want me to consider working on a specific binding to a C
library, just drop me an email or something.
The small ones like ZeroMQ can be done in a few hours or so.  A little
longer to test properly.

I'd rather work on something that I know will be used, than my little
pet projects which will probably not be used for awhile.

 And it's not clear to me at all whether that class will work with
 fixed point data... That's extremely important for a class like
 this (but granted, not really in line with D's take on numeric
 precision). Almost all audio and video data is in fixed point.
 Int->float conversion is very expensive on many platforms.
 
 On 23 November 2011 17:34, Martin Nowak <dawg dawgfoto.de> wrote:
 

 *Crypto *- +1 as mentioned above, blocked on streams...

 
 *Higher level maths/signal processing *- The linear algebra
 library
 
 mentioned in a previous post might be very useful, but also
 standard algorithms like fourier transforms. Library
 implementations of these would
 


 http://www.d-programming-**language.org/phobos/std_**numeric.html#Fft<http://www.d-programming-language.org/phobos/std_numeric.html#Fft>

 




The performance is actually quite good. I'd only wish for a real to

 half-spectrum specialization (probably in-place) performance
 wise.
 
 
 be amazing, potentially making use of hardware back ends where
 possible.

 


 


 


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOzVkUAAoJENcHIWLyQiSlh00IAKLyKpYMN4Iy0JqsMfg8v647
7qntdKWerZ2myU3kgTioQeoybHvkTf+tzkEHEGaxcSA0AusinYK1sjnMrtC9o9eJ
A/2tCH4ronfmt/isBPnzoMInLVjKFaHTHGobUfgQsjXLRtqGFRu/X4Cis+TSEj0/
OIRg5HBYgPkUxLouAbKlTRS0bjqmBhWZfZpAn82Z2RdMZolzPajNNyoKHlxXyX2X
MHuGMgyvfzYIr8Y9s6eFavQnI7+DxNUvFdaLutjE5ua8Q5yH0NTRmaZh9y1IQfSm
o/dX+2/hu6FjPDJkS409MwzZH+XkAE1IHJnNKj9D49MWVbG549muID/PWeOM+6k=
=7R3i
-----END PGP SIGNATURE-----



 Nov 23 2011



  parent  Somedude <lovelydear mailmetrash.com>  writes:


Le 23/11/2011 21:35, Jude Young a écrit :

 er working on a specific binding to a C
 library, just drop me an email or something.
 The small ones like ZeroMQ can be done in a few hours or so.  A little
 longer to test properly.


Awesome. :)
I hope this one will get the publicity it deserves.



 Nov 24 2011



prev sibling next sibling parent  Manu <turkeyman gmail.com>  writes:


On 23 November 2011 21:35, Martin Nowak <dawg dawgfoto.de> wrote:


 On Wed, 23 Nov 2011 18:20:35 +0100, Manu <turkeyman gmail.com> wrote:

  Oh wow, awesome! :)

 I'm always surprised by the state of the D libraries.. There's already
 lots
 of awesome obscure things are in there, but also completely obvious major
 features blatantly missing :)


 There are some real gems, e.g. findRoot.



 So how about a DCT? Image/video processing... Perhaps a simple adaptation
 of FFT?

  Too many different DCTs, too many different needs I'd say.





Well... hardware accelerated video decoders have boiled it down to a
standard set of functions, so it's obviously not an endless amount, and I'd
suggest someone else (who wrote those apis) has already done the work of
identifying the ones that allow you to process most modern formats.



  And it's not clear to me at all whether that class will work with fixed

 point data... That's extremely important for a class like this (but
 granted, not really in line with D's take on numeric precision).
 Almost all audio and video data is in fixed point. Int->float conversion
 is
 very expensive on many platforms.

  Too specialized. Given the obscurity of doing a fixed point FFT, the


 actual
 implementation will be very tough, the log2N gain of calculating
 intermediate
 results is directly opposed to precision.


Obscurity? MP3 isn't obscure. I'm sure there are bunches of reference
implementations to draw from.
But again, I couldn't use the standard library on a PSP or iPhone for
instance if there is no fixed point version. And I'm left in a world where
I have to roll my own, just like I always do in C with respect to the the
CRT, because the standard library just isn't thorough >_<


It would be interesting to offer vectorized 'Integral <-> Float'

 conversions.


No reason not to. If D ever gets a hardware vector type, I'll start writing
that library the same day...
That doesn't address this issue though, I'm concerned with a lot of
architectures that either don't have float units, or extremely slow float
processing. Hand held gaming systems and phones.
I do think fixed point versions of these data shovelling DSP functions is
quite important to many people, but I'm sure 3rd party libs will come.



 Nov 23 2011



prev sibling next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/19/2011 7:02 PM, dsimcha wrote:

 * Compression/archiving. (Opening standard compressed/archived file formats
 needs to just work. This includes at least zip, gzip, tar and bzip2. Of course,
 zip already is available and gzip is supported by the zlib module but with a
 crufty C API. At least gzip and bzip2, which are stream-based as opposed to
 file-based, should be handled via streams, which means that streams block
 compression/archiving. Also, since tar and zip are both file based, they should
 probably be handled by the same API, which might mean deprecating std.zip and
 rewriting it.)


They should be range based, not stream based.



 Nov 23 2011



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/19/2011 7:02 PM, dsimcha wrote:

 * Streams. (Another item where the bottleneck is mostly at the design level and
 people not really knowing what they want.)


I'm not sure what the purpose of streams would be, now that we have ranges.



 Nov 23 2011



 next sibling parent reply dsimcha <dsimcha yahoo.com>  writes:


On 11/23/2011 9:26 PM, Walter Bright wrote:

 On 11/19/2011 7:02 PM, dsimcha wrote:

 * Streams. (Another item where the bottleneck is mostly at the design
 level and
 people not really knowing what they want.)


 I'm not sure what the purpose of streams would be, now that we have ranges.


Right.  As I mentioned in a previous post buried deep in this thread, I 
think streams should just be a flavor of ranges that have most or all of 
the following characteristics:

1.  Live in std.stream.

2.  Oriented toward I/O.

3.  Heavy use of higher order ranges/stacking for things like 
compression/decompression and encryption/decryption.

4.  Mostly focused on input ranges as opposed to random 
access/forward/bidirectional, since this is the best model for data from 
a network or stdin.



 Nov 23 2011



  parent  Jonathan M Davis <jmdavisProg gmx.com>  writes:


On Wednesday, November 23, 2011 22:21:47 dsimcha wrote:

 On 11/23/2011 9:26 PM, Walter Bright wrote:

 On 11/19/2011 7:02 PM, dsimcha wrote:

 * Streams. (Another item where the bottleneck is mostly at the design
 level and
 people not really knowing what they want.)


 
 I'm not sure what the purpose of streams would be, now that we have
 ranges.


 Right.  As I mentioned in a previous post buried deep in this thread, I
 think streams should just be a flavor of ranges that have most or all of
 the following characteristics:
 
 1.  Live in std.stream.
 
 2.  Oriented toward I/O.
 
 3.  Heavy use of higher order ranges/stacking for things like
 compression/decompression and encryption/decryption.
 
 4.  Mostly focused on input ranges as opposed to random
 access/forward/bidirectional, since this is the best model for data from
 a network or stdin.


Andrei's plan for streams was discussed a fair bit a while back. IIRC, it 
essentially involved a lower level range API which took chunks of elements 
rather than elements such that you were essentially dealing with a buffer that 
was reading in the data. A higher level range would then be put on top of that 
range which presented a more normal range API. So, a lower level stream of 
ubytes would be return ubyte[] for its front (the buffer which was ubyte[]), 
and then a higher level range which would return ubyte for its front would be 
on top of that. I'd have to go digging through the archives though to see 
exactly what Andrei was proposing, but that's more or less what I recall. 
AFAIK, he still intends for something along those lines to be implemented, but 
no one has done it yet.

- Jonathan M Davis



 Nov 23 2011



prev sibling  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Wed, 23 Nov 2011 21:26:45 -0500, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 11/19/2011 7:02 PM, dsimcha wrote:

 * Streams. (Another item where the bottleneck is mostly at the design  
 level and
 people not really knowing what they want.)


 I'm not sure what the purpose of streams would be, now that we have  
 ranges.


We still need streams for low-level i/o.  e.g. FILE * is a stream.

ranges are good for high-level concepts, like a range of lines from a  
stream.  Or a range of XML nodes.  Ranges are not good for reading N bytes  
 from a file descriptor.

-Steve



 Nov 26 2011



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/26/2011 5:46 AM, Steven Schveighoffer wrote:

 Ranges are not good for reading N bytes from a file
 descriptor.


Why not? Isn't that exactly what a range is supposed to be good for?



 Nov 26 2011



 next sibling parent reply Manu <turkeyman gmail.com>  writes:


On 26 November 2011 23:39, Walter Bright <newshound2 digitalmars.com> wrote:


 On 11/26/2011 5:46 AM, Steven Schveighoffer wrote:


 Ranges are not good for reading N bytes from a file
 descriptor.


 Why not? Isn't that exactly what a range is supposed to be good for?


It sounds like a bad idea to me... I can imagine ending up with a whole
pile of allocations of short ranges, and then people doing bunches of
range concatenations to re-assemble the buffer. That's a lot of pointless
allocation and memcopying. D's apis should discourage inefficient usage
like that.



 Nov 27 2011



  parent  Piotr Szturmaj <bncrbme jadamspam.pl>  writes:


Manu wrote:

 On 26 November 2011 23:39, Walter Bright <newshound2 digitalmars.com
 <mailto:newshound2 digitalmars.com>> wrote:

     On 11/26/2011 5:46 AM, Steven Schveighoffer wrote:

         Ranges are not good for reading N bytes from a file
         descriptor.


     Why not? Isn't that exactly what a range is supposed to be good for?


 It sounds like a bad idea to me... I can imagine ending up with a whole
 pile of allocations of short ranges, and then people doing bunches of
 range concatenations to re-assemble the buffer. That's a lot of
 pointless allocation and memcopying. D's apis should discourage
 inefficient usage like that.


I think ranges are here to avoid memcopying and pointless allocations.



 Nov 27 2011



prev sibling  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Sat, 26 Nov 2011 16:39:38 -0500, Walter Bright  
<newshound2 digitalmars.com> wrote:


 On 11/26/2011 5:46 AM, Steven Schveighoffer wrote:

 Ranges are not good for reading N bytes from a file
 descriptor.


 Why not? Isn't that exactly what a range is supposed to be good for?


A range has a specific interface:

T front()
void popFront()
bool empty()

An input stream has a specific interface:

size_t read(ubyte[] buffer)
or alternatively:
ubyte[] read(ubyte[] buffer)

How does one map that interface into an input range?  What is T?  I  
remember proposals where there was some extra function setFrontBytes, or  
an additional parameter to front, that allows you to set the number of  
bytes to read.  This seems like trying to fit a square peg in a round hole.

Now, a range fits very well on top of an input stream with a given  
context.  That context defines *what* front() is going to return.  For  
example, the next line of text.

But you still need a well-designed buffered stream interface that a range  
can use to do its work from.  Streams are not going to be the main  
interface for most users, but they will be the tools that range developers  
use to create easily iterated streams.  And when the user wants to read  
data from a file that has no well-defined T that you can define a range  
for, the stream should provide all the tools necessary to read that data.

Now, an output stream *does* fit quite well as a range, since an output  
range's single put method coincides with an ouptut stream's write method.

-Steve



 Nov 28 2011



  parent reply so <so so.so>  writes:


On Mon, 28 Nov 2011 14:54:07 +0200, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 On Sat, 26 Nov 2011 16:39:38 -0500, Walter Bright  
 <newshound2 digitalmars.com> wrote:


 On 11/26/2011 5:46 AM, Steven Schveighoffer wrote:

 Ranges are not good for reading N bytes from a file
 descriptor.


 Why not? Isn't that exactly what a range is supposed to be good for?


 A range has a specific interface:

 T front()
 void popFront()
 bool empty()

 An input stream has a specific interface:

 size_t read(ubyte[] buffer)
 or alternatively:
 ubyte[] read(ubyte[] buffer)

 How does one map that interface into an input range?  What is T?  I  
 remember proposals where there was some extra function setFrontBytes, or  
 an additional parameter to front, that allows you to set the number of  
 bytes to read.  This seems like trying to fit a square peg in a round  
 hole.

 Now, a range fits very well on top of an input stream with a given  
 context.  That context defines *what* front() is going to return.  For  
 example, the next line of text.

 But you still need a well-designed buffered stream interface that a  
 range can use to do its work from.  Streams are not going to be the main  
 interface for most users, but they will be the tools that range  
 developers use to create easily iterated streams.  And when the user  
 wants to read data from a file that has no well-defined T that you can  
 define a range for, the stream should provide all the tools necessary to  
 read that data.

 Now, an output stream *does* fit quite well as a range, since an output  
 range's single put method coincides with an ouptut stream's write method.

 -Steve


I was absent from range discussions, probably it has been said, sorry.



T front()
void popFront()
bool empty()

This matches with:

T peek()
void seek(1)
bool eof()

T is ubyte.

I believe we are asking the wrong questions, first question should be why  
someone would like this simple range interface (3 functions) for streams?
Any use? Yes with these 3 we can do quite many things, not optimal but we  
can. For the real use we need a new range.

T[] get(size_t)
bool put(size_t, T[])
bool seek(size_t)
...

Don't we need a range like this in any case? I doubt we can do much  
without it.



 Nov 28 2011



  parent reply "Steven Schveighoffer" <schveiguy yahoo.com>  writes:


On Mon, 28 Nov 2011 08:18:38 -0500, so <so so.so> wrote:


 On Mon, 28 Nov 2011 14:54:07 +0200, Steven Schveighoffer  
 <schveiguy yahoo.com> wrote:


 On Sat, 26 Nov 2011 16:39:38 -0500, Walter Bright  
 <newshound2 digitalmars.com> wrote:


 On 11/26/2011 5:46 AM, Steven Schveighoffer wrote:

 Ranges are not good for reading N bytes from a file
 descriptor.


 Why not? Isn't that exactly what a range is supposed to be good for?


 A range has a specific interface:

 T front()
 void popFront()
 bool empty()

 An input stream has a specific interface:

 size_t read(ubyte[] buffer)
 or alternatively:
 ubyte[] read(ubyte[] buffer)

 How does one map that interface into an input range?  What is T?  I  
 remember proposals where there was some extra function setFrontBytes,  
 or an additional parameter to front, that allows you to set the number  
 of bytes to read.  This seems like trying to fit a square peg in a  
 round hole.

 Now, a range fits very well on top of an input stream with a given  
 context.  That context defines *what* front() is going to return.  For  
 example, the next line of text.

 But you still need a well-designed buffered stream interface that a  
 range can use to do its work from.  Streams are not going to be the  
 main interface for most users, but they will be the tools that range  
 developers use to create easily iterated streams.  And when the user  
 wants to read data from a file that has no well-defined T that you can  
 define a range for, the stream should provide all the tools necessary  
 to read that data.

 Now, an output stream *does* fit quite well as a range, since an output  
 range's single put method coincides with an ouptut stream's write  
 method.

 -Steve


 I was absent from range discussions, probably it has been said, sorry.



 T front()
 void popFront()
 bool empty()

 This matches with:

 T peek()
 void seek(1)
 bool eof()

 T is ubyte.


The task is not to see if a stream can implement a range, it's to see if a  
range can implement a stream.  The target interface is the read function I  
mentioned.

Nobody wants to use the "one byte at a time" input range to interface with  
streams.  This just furthers my point.


 I believe we are asking the wrong questions, first question should be  
 why someone would like this simple range interface (3 functions) for  
 streams?
 Any use? Yes with these 3 we can do quite many things, not optimal but  
 we can. For the real use we need a new range.

 T[] get(size_t)
 bool put(size_t, T[])
 bool seek(size_t)
 ...

 Don't we need a range like this in any case? I doubt we can do much  
 without it.


Yes, but let's not call this a "range", since it does not work in  
functions that use ranges or with foreach (this is Walter's goal).

Ranges on top of streams make sense.  The most classic is byLine.  But  
byLine is not going to use a range interface to do it's underlying work.

IMO the layers from lowest to highest are:

device stream -> buffered stream -> range

In fact, that's *exactly* what phobos does today.  buffered stream (and  
the underlying device stream) is FILE *.

I'm working on a replacement for FILE * which allows much more  
flexibility, performance, while std.stdio will maintain the exact same  
interface.

-Steve



 Nov 28 2011



  parent  so <so so.so>  writes:


On Mon, 28 Nov 2011 15:32:23 +0200, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 Yes, but let's not call this a "range", since it does not work in  
 functions that use ranges or with foreach (this is Walter's goal).


How bout strange? (Yes, i am bored)



 Nov 28 2011