www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How does one read file line by line / upto a specific delimeter of an

reply Adnan <relay.public.adnan outlook.com> writes:
https://dlang.org/library/std/mmfile/mm_file.html doesn't seem to 
specify anything similar to lines() or byLine() or byLineCopy() 
etc.
Mar 14 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Mar 14, 2020 at 10:37:37PM +0000, Adnan via Digitalmars-d-learn wrote:

 https://dlang.org/library/std/mmfile/mm_file.html doesn't seem to
 specify anything similar to lines() or byLine() or byLineCopy() etc.


That's because a memory-mapped file appears directly in your program's
memory address space as if it was an array of bytes (ubyte[]).  No
interpretation is imposed upon the contents.  If you want lines out of
it, try casting the memory to const(char)[] and using
std.algorithm.splitter to get a range of lines. For example:

	auto mmfile = new MmFile("myfile.txt");
	auto data = cast(const(char)[]) mmfile[];
	auto lines = data.splitter("\n");
	foreach (line; lines) {
		...
	}


T

-- 
"You are a very disagreeable person." "NO."
Mar 14 2020
parent reply Adnan <relay.public.adnan outlook.com> writes:
On Sunday, 15 March 2020 at 00:37:35 UTC, H. S. Teoh wrote:

 On Sat, Mar 14, 2020 at 10:37:37PM +0000, Adnan via 
 Digitalmars-d-learn wrote:

 https://dlang.org/library/std/mmfile/mm_file.html doesn't seem 
 to
 specify anything similar to lines() or byLine() or 
 byLineCopy() etc.


 That's because a memory-mapped file appears directly in your 
 program's memory address space as if it was an array of bytes 
 (ubyte[]).  No interpretation is imposed upon the contents.  If 
 you want lines out of it, try casting the memory to 
 const(char)[] and using std.algorithm.splitter to get a range 
 of lines. For example:

 	auto mmfile = new MmFile("myfile.txt");
 	auto data = cast(const(char)[]) mmfile[];
 	auto lines = data.splitter("\n");
 	foreach (line; lines) {
 		...
 	}


 T


Would it be wasteful to cast the entire content into a const 
string? Can a memory mapped file be read with a buffer?
Mar 16 2020
next sibling parent Adnan <relay.public.adnan outlook.com> writes:
On Monday, 16 March 2020 at 13:09:08 UTC, Adnan wrote:

 On Sunday, 15 March 2020 at 00:37:35 UTC, H. S. Teoh wrote:

 On Sat, Mar 14, 2020 at 10:37:37PM +0000, Adnan via 
 Digitalmars-d-learn wrote:

 https://dlang.org/library/std/mmfile/mm_file.html doesn't 
 seem to
 specify anything similar to lines() or byLine() or 
 byLineCopy() etc.


 That's because a memory-mapped file appears directly in your 
 program's memory address space as if it was an array of bytes 
 (ubyte[]).  No interpretation is imposed upon the contents.  
 If you want lines out of it, try casting the memory to 
 const(char)[] and using std.algorithm.splitter to get a range 
 of lines. For example:

 	auto mmfile = new MmFile("myfile.txt");
 	auto data = cast(const(char)[]) mmfile[];
 	auto lines = data.splitter("\n");
 	foreach (line; lines) {
 		...
 	}


 T


 Would it be wasteful to cast the entire content into a const 
 string? Can a memory mapped file be read with a buffer?


for more context here's the program

string lexHash(scope const ref string arg) {
	auto repeated = arg ~ arg;
	string result = arg;
	for (auto i = 1; i < arg.length; ++i) {
		const slice = repeated[i .. i + arg.length];
		if (slice < result)
			result = slice;
	}
	return result;
}

unittest {
	assert("cba".smallestRepr() == "acb");
}

void main(const string[] args) {
	import std.stdio : writeln, lines, File;
	import std.algorithm : splitter;
	import std.mmfile : MmFile;

	string[][const string] wordTable;
	scope auto mmfile = new MmFile(args[1]);
	auto data = cast(const string) mmfile[];

	foreach (string word; data.splitter()) {
		const string key = word.lexHash();
		wordTable.require(key, []) ~= word;
		if (wordTable[key].length == 4) {
			writeln(wordTable[key]);
			break;
		}
	}
}
Mar 16 2020
prev sibling next sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 16 March 2020 at 13:09:08 UTC, Adnan wrote:

 On Sunday, 15 March 2020 at 00:37:35 UTC, H. S. Teoh wrote:

 On Sat, Mar 14, 2020 at 10:37:37PM +0000, Adnan via 
 Digitalmars-d-learn wrote:

 [...]


 That's because a memory-mapped file appears directly in your 
 program's memory address space as if it was an array of bytes 
 (ubyte[]).  No interpretation is imposed upon the contents.  
 If you want lines out of it, try casting the memory to 
 const(char)[] and using std.algorithm.splitter to get a range 
 of lines. For example:

 	auto mmfile = new MmFile("myfile.txt");
 	auto data = cast(const(char)[]) mmfile[];
 	auto lines = data.splitter("\n");
 	foreach (line; lines) {
 		...
 	}


 T


 Would it be wasteful to cast the entire content into a const 
 string? Can a memory mapped file be read with a buffer?


a string is the same thing as immutable(char)[] . It would make 
no difference with the example above.
Mar 16 2020
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Mar 16, 2020 at 01:09:08PM +0000, Adnan via Digitalmars-d-learn wrote:
[...]

 Would it be wasteful to cast the entire content into a const string?


Why would it be?  It's just reinterpreting a pointer.



 Can a memory mapped file be read with a buffer?


That totally defeats the purpose of memory-mapping.  The whole idea of
memory-mapping is that the OS takes care of buffering the file data for
you.  As far as your program is concerned, the entire file is actually
"in memory" (it isn't really, but since the OS takes care of paging
parts of it in and out as necessary, and this is completely transparent
to your program, you might as well treat it as a huge consecutive array
that resides, for all intents and purposes, "in memory" -- hence the
term "virtual memory").


T

-- 
There are three kinds of people in the world: those who can count, and those
who can't.
Mar 16 2020