• Kar (3/3) Nov 22 2007 Im trying to index a very large document collections. I need to know whi...
• Daniel Keep (8/13) Nov 22 2007 Well, assuming the machine you're going to be running on doesn't
• Janice Caron (2/2) Nov 23 2007 Halfway between the two extremes is the BufferedFile, which keeps a
• Christopher Wright (3/8) Nov 23 2007 std.mmfile for the win.
• Daniel Keep (4/17) Nov 23 2007 I haven't used memory-mapping before, so excuse my ignorance, but how
• Christopher Wright (2/21) Nov 23 2007 Um...good point. Allow me to blush.
• Kar (2/12) Nov 25 2007 Im developing the program under win machine but it will be run as a linu...

Kar <akmalxxx gmail.com> writes:

Im trying to index a very large document collections. I need to know which file
access method is faster, either std.file read all data into buffer or use
stream.

i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.

The index file could be more than 10GBs, so which method is better.

Daniel Keep <daniel.keep.lists gmail.com> writes:

Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.

Well, assuming the machine you're going to be running on doesn't
actually have 10 GB of memory, you probably don't want to read it all in
at once.

Streams work by reading in only what you actually use as you use it.  A
stream remembers "where" it is, so it doesn't need to read the whole
file into memory.

	-- Daniel

"Janice Caron" <caron800 googlemail.com> writes:

Halfway between the two extremes is the BufferedFile, which keeps a
bufferful of data in memory, and the rest of the file on disc.

Christopher Wright <dhasenan gmail.com> writes:

Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.

std.mmfile for the win.

Try the higher-level interface in std.stream.MmFileStream.

Daniel Keep <daniel.keep.lists gmail.com> writes:

Christopher Wright wrote:
 Kar wrote:
 Im trying to index a very large document collections. I need to know
 which file access method is faster, either std.file read all data into
 buffer or use stream.

 i didnt understand stream very well, does it store file strings in
 memory or just a pointer to exact disc location.

 The index file could be more than 10GBs, so which method is better.

 
 std.mmfile for the win.
 
 Try the higher-level interface in std.stream.MmFileStream.

I haven't used memory-mapping before, so excuse my ignorance, but how
would you map a 10GB file into the address space on a 32-bit machine?!

	-- Daniel

Christopher Wright <dhasenan gmail.com> writes:

Daniel Keep wrote:
 
 Christopher Wright wrote:
 Kar wrote:
 Im trying to index a very large document collections. I need to know
 which file access method is faster, either std.file read all data into
 buffer or use stream.

 i didnt understand stream very well, does it store file strings in
 memory or just a pointer to exact disc location.

 The index file could be more than 10GBs, so which method is better.

 std.mmfile for the win.

 Try the higher-level interface in std.stream.MmFileStream.

 
 I haven't used memory-mapping before, so excuse my ignorance, but how
 would you map a 10GB file into the address space on a 32-bit machine?!
 
 	-- Daniel

Um...good point. Allow me to blush.

Kar <akmalxxx gmail.com> writes:

Christopher Wright Wrote:

 Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.

 
 std.mmfile for the win.
 
 Try the higher-level interface in std.stream.MmFileStream.

Im developing the program under win machine but it will be run as a linux
daemon. So mmfile is not an option for me. But i think i understand more about
stream now, thanks alot. btw, any other option for buffer file other than
mmfile?