www.digitalmars.com         C & C++   DMDScript  

D - List directory contents (request)

reply Ant <Ant_member pathlink.com> writes:
Request for List directory contents

This was discussed here.

Do you guys have a working version for both
target operating systems?
Can you post it here so anybody can use it
and maybe we can persuade Walter to include them 
in the next phobos release?

Thanks
Ant
Oct 06 2003
parent reply "Matthew Wilson" <matthew stlsoft.org> writes:
Ant

I've got several STL-like directory system search classes in STLSoft. (Read
"http://www.windevnet.com/documents/win0303a/" for more info. Note that you
*must* put in the trailing slash in that URL, or the WinDevNet doc finder
will fail). There is the WinSTL basic_findfile_sequence<> template (uses
FindFirst/NextFile(Ex)(), ANSI or Unicode), and the UNIXSTL readdir_sequence
(uses opendir()/readdir()) and findfile_sequence (uses glob()) classes. If
you've already got STLSoft, check 'em out. (If you've not, why not? Maybe
you don't like C++. Try http://stlsoft.org/downloads.html or
http://www.digitalmars.com/download/freecompiler.html)

These three classes follow the STLSoft philosophy of adapting operating
system APIs to the STL way of doing things (e.g. for_each-able iterator
ranges), but all three do things slightly differently, reflecting the
considerable differences between the methods of searching on Win32 & UNIX.

If there was a petition got up, I've no doubt that I could do some porting,
putting a D (i.e. foreach) face onto them. If it was up to me, I'd probably
still go for separate types. Otherwise, we'd need to use only the common
functionality, which would be to return all entries, all file entries, all
directory entries (with or without dots). It may be pretty simple to add in
the regexp code, and then platform-independent pattern matching could be
done. (Hmmm, this is starting to sound tempting ...)

The other option, which I'd planned to bring up in a couple of months, is to
use a new library that I'm writing for my new C/C++ User's Journal. (It
features in the November issue, which should be available just about now.)
The library is called recls (for recursive ls), and it is a
platform-independent file-searching C library (implemented in C++). The
purpose of the column, called Positive Integration, is to show how to
integrate C and C++ with other languages/technologies. So the first few
instalments will feature recls, and provide mappings to a host of languages,

and Automation collections), D (foreach-able), Delphi, Java, Managed C++,
Perl, Python and Ruby. (I might do more, if the mood takes me ... :)

The first instalment contains the first version of the library, and the C++
wrapper classes. The second instalment (already written, coming out in

contain COM and D. Naturally the mappings can be made available when ready,
rather than waiting for the column instalments.

I'd planned to suggest recls be a D standard library outer (i.e. non core)
module, that would be treated in the same way other C components, such as
D.c.stdio, i.e.. D.c.recls. Or, it might just be an external library that
people would (hopefully) use, though this is less likely to happen than if
it's in the standard library.

At this point, I'm stupidly busy as usual, so would rather not be doing any
more D until the registry library is into Phobos and happy, but would be
interested in everyone's opinions on this issue. My ideal is to wait for the
D recls mapping (which I'll be doing in a couple of weeks), since once
that's done there's not really a need for any other search components: it
can do recursive or non-recursive searching, handles patterns, can search
for files and/or directories (and/or links), and will be
platform-independent, so no-one would have to care about platform-dependent
client code hassles.All you do is specify a root directory to search, a
search pattern (it will eventually support multiple search patterns, e.g.
"*.cpp;r*.d;*.xl?"), and search flags. I even have plans for letting it
handle searching FTP sites, if I can wrangle the URI syntax in a simple and
unambiguous fashion.

I'd be interested in everyone's thoughts on possible directions:

(1) Crib the WinSTL/UNIXSTL sequence types, and D-ise them into separate
types (e.g. D.win32.filesys.readdir, D.linux.filesys.readdir).
(2) Option (1), but abstract the common features, and D-ise into a single
type (e.g. D.filesys.readdir).
(3) Use the recls D mapping, as a part of the (outer) standard library (e.g.
D.c.(filesys.)recls)
(4) Use the recls D mapping, as a SynSoft offering (e.g. synsoft.recls).
(5) P**s off and leave it to the Java-ites. :)

Technically speaking, I'd prefer option (3). If people preferred recls (and,
importantly, Walter approved it!), I guess I could put some effort into it
soonish, and have a Win32 version working within the next couple of weeks.
Options (1) & (2) will take longer, since there'll be a lot more work
involved (and it's work I had not planned to do, unlike the recls D
mapping). Option (4) would be much less likely to get used in the main, and
will therefore still require a standard library system file-searcher, so one
wonders what the point would be. Option (5) is easiest (for me). :)

Cheers

-- 
Matthew Wilson

STLSoft moderator and C++ monomaniac       (http://www.stlsoft.org)
Contributing editor, C/C++ Users Journal
(www.synesis.com.au/articles.html#columns)

"But if less is more, think how much more more will be!" -- Dr Frazier Crane

----------------------------------------------------------------------------
---




"Ant" <Ant_member pathlink.com> wrote in message
news:bls8qm$2935$1 digitaldaemon.com...

 Request for List directory contents

 This was discussed here.

 Do you guys have a working version for both
 target operating systems?
 Can you post it here so anybody can use it
 and maybe we can persuade Walter to include them
 in the next phobos release?

 Thanks
 Ant
Oct 06 2003
next sibling parent reply Ant <Ant_member pathlink.com> writes:
In article <blsn7k$2v3p$1 digitaldaemon.com>, Matthew Wilson says...

Ant

I've got several STL-like directory system search classes in STLSoft.


You are very generous, however I wouldn't know where to start.

Let me propose another way of doing this:
(I'll repost this in a separate post)
Someone (Walter or any other smart guy) should make public the
desired interface for D and them some hands on guy (Ant or any other guy)
could make the implementation of that interface
(we will use the version() not the interface/implementation).
even a less then good (I don't mean buggy)
implementation is better then nothing (I guess).

bellow is what I mean by directory contents:
(this might be a 
"pointless wrappers around C runtime library functions or OS API functions",
let me know please)

First let me ask a couple of questions (see code bellow):

technical:
- why do I get segfault on free(void*) ??? I thought I need it...
- why is diret.d_name a D char[]? (I printf it with %.*s not %s)
- if I get a string from C "char *cString" do I need to free it after
   converting it to a D array with char[] s = string.toString(cString); ?

filosofical:
- do we need this?
- is this the way to go? I don't mean the specifics of my implementation
   but the general idea.
- should we try to extract a common interface with what Burton posted on
   http://www.digitalmars.com/drn-bin/wwwnews?D/4753
   so we can have a window and linux implementation?
- should I just drop it as it has no interest?

community:
- does anybody have something better that want's to share?

finally here it is what I waisted the entire night with:
(I believe is waisted time because some of you must
have already done it)



module utils.Directory;

private import string;

public:
enum FileTypes
{
DIRECTORY = 4,
FILE = 8,
S_LINK = 10
}

extern(C)
{

struct DIR;

// this structure will not work on all systems
// the the size of the integer will vary
struct dirent
{
uint d_ino;
uint d_off;
ushort d_reclen;
char d_type;
char[256] d_name;  /*this limitation is imposed by the system*/
}
;

DIR* opendir(char* name);
dirent* readdir(DIR* dir);
void rewinddir(DIR *dir);

void free(void*);
}

/**
* A directory error
* No error codes for now, just inadequate descriptive message
* \todo get the errno?
* \todo define some code errors as windows and un*x will be different?
*/
class DirectoryError : Error
{
this(char[] message)
{
super(message);
}
}

/**
* A directory content browser.
* \todo get meaninfull information on the directory entries
*/
public:
class Directory
{
int count = 0;

DIR* dir;
dirent* entry;

dirent*[] dirs;
dirent*[] files;


public:

/**
* Creates a Directory object for the directory
*  param directory must be a directory name
*/
this(char[] directory)
{
if ( directory === null )
{
throw new DirectoryError("Directory name not received");
}
dir = opendir((char*)directory);
if ( dir === null )
{
throw new DirectoryError("can't open directory ");
}
}

~this()
{
if ( dir !== null )
{
//free(dir);
}
if ( entry !== null )
{
//free(entry);
}
//freeAndClearDirs();
//freeAndClearFiles();
}

bit next()
{
if ( entry !== null )
{
free(entry);
entry = null;
}
//printf("\n");
dirent* entry = readdir(dir);
if ( entry !== null )
{
//printf("entry d_ino    %d\n",entry.d_ino);
//printf("entry d_off    %d\n",entry.d_off);
//printf("entry d_reclen %d\n",entry.d_reclen);
//printf("entry d_type   %c\n",entry.d_type);
printf("%d %d %.*s\n",count++,(int)entry.d_type,entry.d_name);
}
//return entry;
return entry !== null;
}

/**
* Resets the read pointer to the first entry
*/
void rewind()
{
rewinddir(dir);
}

private dirent*[] getEntries(FileTypes type)
{
dirent*[] entries;
dirent* entry;
do
{
entry = readdir(dir);
if ( entry !== null )
{
if ( entry.d_type == type )
{
entries ~= entry;
}
else
{
//free(entry);
}
}
} while (entry !== null);
return entries;
}

/**
* get and array with all the subdirectories of this directory
*  return and array with all the subdirectories
* \todo include the simbolic links
* \todo avoid scan the directory twice to get the subdirs and the files
*/
dirent*[] getDirs()
{
//freeAndClearDirs();
rewinddir(dir);
dirs = getEntries(FileTypes.DIRECTORY);
return dirs;
}

/**
* get and array with all the files of this directory
*  return and array with all the files
* \todo include the simbolic links
* \todo avoid scan the directory twice to get the subdirs and the files
*/
dirent*[] getFiles()
{
//freeAndClearDirs();
rewinddir(dir);
files = getEntries(FileTypes.FILE);
return dirs;
}

/**
* Gets a sorted array with all the subdirectory names
*  return a sorted array with all the subdirectory names
*/
char[][] getNames(FileTypes type)
{
char[][] names;
rewinddir(dir);
dirent* entry = readdir(dir);
while ( entry !== null )
{
if ( entry.d_type == type )
{
names[] ~= string.toString(entry.d_name);
}
//free(entry);
entry = readdir(dir);
}
return names.sort;
}

/**
* Gets a sorted array of the subdirectories
*  return a sorted array of the subdirectories
*/
char[][] getDirNames()
{
return getNames(FileTypes.DIRECTORY);
}

/**
* Gets a sorted array of the files on this directory
*  return a sorted array of the files on this directory
*/
char[][] getFileNames()
{
return getNames(FileTypes.FILE);
}

/**
* Frees the content of the dirs array and sets it's length to 0
*/
private void freeAndClearDirs()
{
foreach(dirent* d ; dirs)
{
free(d);
}
dirs.length = 0;
}

/**
* Frees the content of the files array and setst it;s length to 0
*/
private void freeAndClearFiles()
{
foreach(dirent* f ; files)
{
free(f);
}
files.length = 0;
}


}

/**
* just testing the thing
*/
void main(char[][] argv)
{

char [] dirName;

if ( argv.length < 2 )
{
dirName = ".";
}
else
{
dirName = argv[1];
}

Directory dir = new Directory(dirName);
bit d;





do
{
d = dir.next();
} while(d);





foreach(dirent* entry; dir.getDirs() )
{
printf("\t%.*s\n",entry.d_name);
}





foreach(dirent* entry; dir.getFiles() )
{
printf("\t%.*s\n",entry.d_name);
}





foreach(char[] name; dir.getDirNames() )
{
printf("\t%.*s\n",name);
}





foreach(char[] name; dir.getFileNames() )
{
printf("\t%.*s\n",name);
}

}



thanks

Ant
Oct 06 2003
next sibling parent "Matthew Wilson" <matthew stlsoft.org> writes:
 Let me propose another way of doing this:
 (I'll repost this in a separate post)
 Someone (Walter or any other smart guy) should make public the
 desired interface for D and them some hands on guy (Ant or any other guy)
 could make the implementation of that interface
 (we will use the version() not the interface/implementation).
 even a less then good (I don't mean buggy)
 implementation is better then nothing (I guess).


Sorry, I misunderstood. I thought you were asking for whether an
implementation was, or is soon likely to be, available; I didn't realise you
were wanting to implement it yourself. Go for it! :)
Oct 06 2003
prev sibling parent "Matthew Wilson" <matthew stlsoft.org> writes:
 Let me propose another way of doing this:
 (I'll repost this in a separate post)
 Someone (Walter or any other smart guy) should make public the
 desired interface for D and them some hands on guy (Ant or any other guy)
 could make the implementation of that interface
 (we will use the version() not the interface/implementation).
 even a less then good (I don't mean buggy)
 implementation is better then nothing (I guess).


Sorry, I misunderstood. I thought you were asking for whether an
implementation was, or is soon likely to be, available; I didn't realise you
were wanting to implement it yourself.

Go for it! :)
Oct 06 2003
prev sibling parent reply "Sean L. Palmer" <palmer.sean verizon.net> writes:
That's really cool Matthew.  I vote for option 2 (but 3 sounds nice too)

Sean

"Matthew Wilson" <matthew stlsoft.org> wrote in message
news:blsn7k$2v3p$1 digitaldaemon.com...

 At this point, I'm stupidly busy as usual, so would rather not be doing


any

 more D until the registry library is into Phobos and happy, but would be
 interested in everyone's opinions on this issue. My ideal is to wait for


the

 D recls mapping (which I'll be doing in a couple of weeks), since once
 that's done there's not really a need for any other search components: it
 can do recursive or non-recursive searching, handles patterns, can search
 for files and/or directories (and/or links), and will be
 platform-independent, so no-one would have to care about


platform-dependent

 client code hassles.All you do is specify a root directory to search, a
 search pattern (it will eventually support multiple search patterns, e.g.
 "*.cpp;r*.d;*.xl?"), and search flags. I even have plans for letting it
 handle searching FTP sites, if I can wrangle the URI syntax in a simple


and

 unambiguous fashion.

 I'd be interested in everyone's thoughts on possible directions:

 (1) Crib the WinSTL/UNIXSTL sequence types, and D-ise them into separate
 types (e.g. D.win32.filesys.readdir, D.linux.filesys.readdir).
 (2) Option (1), but abstract the common features, and D-ise into a single
 type (e.g. D.filesys.readdir).
 (3) Use the recls D mapping, as a part of the (outer) standard library


(e.g.

 D.c.(filesys.)recls)
 (4) Use the recls D mapping, as a SynSoft offering (e.g. synsoft.recls).
 (5) P**s off and leave it to the Java-ites. :)

 Technically speaking, I'd prefer option (3). If people preferred recls


(and,

 importantly, Walter approved it!), I guess I could put some effort into it
 soonish, and have a Win32 version working within the next couple of weeks.
 Options (1) & (2) will take longer, since there'll be a lot more work
 involved (and it's work I had not planned to do, unlike the recls D
 mapping). Option (4) would be much less likely to get used in the main,


and

 will therefore still require a standard library system file-searcher, so


one

 wonders what the point would be. Option (5) is easiest (for me). :)
Oct 07 2003
parent reply "Matthew Wilson" <matthew stlsoft.org> writes:
Good to hear. :)

Option 2 will take longer, and will not do the recursive stuff, but will be
100% pure D. (I don't know whether this issue is going to be important to
the anal, not to say ludicrous, degree it is with the 100% pure Java
movement. I hope not.)
Option 3 will be sooner, and will recurse, but will be a D+C lib. (However,
since we'll always have dependencies on C libs in D, I personally don't
think it's an issue).

My suspicion is that this will come down to a Walter-decision, and that
he'll want option 2.

I think the way to go is for me to do the recls-D mapping, and advertise it
here, and people can decide from use, rather than conversation, what they
feel about it.


"Sean L. Palmer" <palmer.sean verizon.net> wrote in message
news:blts2r$2g3n$1 digitaldaemon.com...

 That's really cool Matthew.  I vote for option 2 (but 3 sounds nice too)

 Sean

 "Matthew Wilson" <matthew stlsoft.org> wrote in message
 news:blsn7k$2v3p$1 digitaldaemon.com...

 At this point, I'm stupidly busy as usual, so would rather not be doing


 any

 more D until the registry library is into Phobos and happy, but would be
 interested in everyone's opinions on this issue. My ideal is to wait for


 the

 D recls mapping (which I'll be doing in a couple of weeks), since once
 that's done there's not really a need for any other search components:




it

 can do recursive or non-recursive searching, handles patterns, can




search

 for files and/or directories (and/or links), and will be
 platform-independent, so no-one would have to care about


 platform-dependent

 client code hassles.All you do is specify a root directory to search, a
 search pattern (it will eventually support multiple search patterns,




e.g.

 "*.cpp;r*.d;*.xl?"), and search flags. I even have plans for letting it
 handle searching FTP sites, if I can wrangle the URI syntax in a simple


 and

 unambiguous fashion.

 I'd be interested in everyone's thoughts on possible directions:

 (1) Crib the WinSTL/UNIXSTL sequence types, and D-ise them into separate
 types (e.g. D.win32.filesys.readdir, D.linux.filesys.readdir).
 (2) Option (1), but abstract the common features, and D-ise into a




single

 type (e.g. D.filesys.readdir).
 (3) Use the recls D mapping, as a part of the (outer) standard library


 (e.g.

 D.c.(filesys.)recls)
 (4) Use the recls D mapping, as a SynSoft offering (e.g. synsoft.recls).
 (5) P**s off and leave it to the Java-ites. :)

 Technically speaking, I'd prefer option (3). If people preferred recls


 (and,

 importantly, Walter approved it!), I guess I could put some effort into




it

 soonish, and have a Win32 version working within the next couple of




weeks.

 Options (1) & (2) will take longer, since there'll be a lot more work
 involved (and it's work I had not planned to do, unlike the recls D
 mapping). Option (4) would be much less likely to get used in the main,


 and

 will therefore still require a standard library system file-searcher, so


 one

 wonders what the point would be. Option (5) is easiest (for me). :)
Oct 07 2003
parent reply Charles Hixson <charleshixsn earthlink.net> writes:
Matthew Wilson wrote:

 Good to hear. :)
 
 Option 2 will take longer, and will not do the recursive stuff, but will be
 100% pure D. (I don't know whether this issue is going to be important to
 the anal, not to say ludicrous, degree it is with the 100% pure Java
 movement. I hope not.)
 Option 3 will be sooner, and will recurse, but will be a D+C lib. (However,
 since we'll always have dependencies on C libs in D, I personally don't
 think it's an issue).
 
 My suspicion is that this will come down to a Walter-decision, and that
 he'll want option 2.
 
 I think the way to go is for me to do the recls-D mapping, and advertise it
 here, and people can decide from use, rather than conversation, what they
 feel about it.
 ...
 


Walter, or whoever wrote the D language specs, has already indicated 
that (paraphrased) "D will link easily with C, as it is rediculous 
to think of translating all of the C libraries".  So those who are 
purists should probably think again.

That said, I believe that it is beneficial that as much as feasible 
of the language features and tools be implemented in D.  D doesn't 
have the same reasons that Java has for being purist about it.  It's 
not aiming at full portability for compiled code.  But code in C 
often makes use of things like doubly indirect pointers, 
inintelligible macros, lots and lots of casts, etc., which makes it 
difficult to read and understand.  Of course, I'm sure that 
obfuscated code could be written in D, even without dropping into 
assembler.  But the language doesn't positively *encourage* the 
practice.
Oct 07 2003
parent "Matthew Wilson" <matthew stlsoft.org> writes:
"Charles Hixson" <charleshixsn earthlink.net> wrote in message
news:blvjdn$1r94$1 digitaldaemon.com...

 Matthew Wilson wrote:

 Good to hear. :)

 Option 2 will take longer, and will not do the recursive stuff, but will




be

 100% pure D. (I don't know whether this issue is going to be important




to

 the anal, not to say ludicrous, degree it is with the 100% pure Java
 movement. I hope not.)
 Option 3 will be sooner, and will recurse, but will be a D+C lib.




(However,

 since we'll always have dependencies on C libs in D, I personally don't
 think it's an issue).

 My suspicion is that this will come down to a Walter-decision, and that
 he'll want option 2.

 I think the way to go is for me to do the recls-D mapping, and advertise




it

 here, and people can decide from use, rather than conversation, what




they

 feel about it.
 ...


 Walter, or whoever wrote the D language specs, has already indicated
 that (paraphrased) "D will link easily with C, as it is rediculous
 to think of translating all of the C libraries".  So those who are
 purists should probably think again.


Agreed. It makes perfect sense to me to use a C library that is available
just about now, rather than wait for one that may take a long time just to
get pure D.


 That said, I believe that it is beneficial that as much as feasible
 of the language features and tools be implemented in D.  D doesn't
 have the same reasons that Java has for being purist about it.  It's
 not aiming at full portability for compiled code.  But code in C
 often makes use of things like doubly indirect pointers,
 inintelligible macros, lots and lots of casts, etc., which makes it
 difficult to read and understand.  Of course, I'm sure that
 obfuscated code could be written in D, even without dropping into
 assembler.  But the language doesn't positively *encourage* the
 practice.


You've not made your preference clear. Option (2) or (3)? (Or (5)?)

:)

btw, I've just started playing around with the recls-D mapping, and I reckon
I can have it done in a very short space of time indeed.

Matthew
Oct 07 2003