www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Can we do compile time reading part of a file using import?

reply data pulverizer <data.pulverizer gmail.com> writes:
Hi all,

the `import` function allows a file to be read at compile time, 
which opens up great opportunities for (mostly binary) file IO, 
where data types can be coded into files - the user doesn't need 
to know data types ahead of time. As specified in my old blog 
article: 
https://www.active-analytics.com/blog/reading-idx-files-in-d/.

Binary files can be very large and reading the whole file at 
compile time is often unnecessary.

This isn't exactly the intended use for the function, but there 
it is. Since compile time read capability already exists and is 
useful, adding capability to be able to read only a portion of 
the file at compile time in a given location is advantageous and 
has utility.

For me it's not make-or-break, it just something very useful and 
I think has clear use case. Please let me know if there are 
aspects or alternatives I am missing.

Thanks
Oct 23 2020
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-10-23 18:42, data pulverizer wrote:


 For me it's not make-or-break, it just something very useful and I think 
 has clear use case. Please let me know if there are aspects or 
 alternatives I am missing.


You could always have the build tool split up the file in multiple 
smaller files and read one of the smaller files with the import expression.

-- 
/Jacob Carlborg
Oct 25 2020
next sibling parent data pulverizer <data.pulverizer gmail.com> writes:
On Sunday, 25 October 2020 at 12:16:36 UTC, Jacob Carlborg wrote:

 On 2020-10-23 18:42, data pulverizer wrote:
 You could always have the build tool split up the file in 
 multiple smaller files and read one of the smaller files with 
 the import expression.


Thanks. My current solution is to have a metadata file where all 
the type information is stored and then the data file separately.

I should have been clearer when I asked if there was something I 
was missing, like whether there is another way to do compile time 
read without reading the whole file. I think the feature I 
suggested is a useful one to bear in mind for future versions of 
D. It would save on preprocessing large files before reading 
them. I don't know how `import` for file reads works at the 
moment, and correct me if I'm wrong but it doesn't seem to me 
that it would be a big deal to make the change.
Oct 25 2020
prev sibling parent reply Jack <jckj33 gmail.com> writes:
On Sunday, 25 October 2020 at 12:16:36 UTC, Jacob Carlborg wrote:

 On 2020-10-23 18:42, data pulverizer wrote:


 For me it's not make-or-break, it just something very useful 
 and I think has clear use case. Please let me know if there 
 are aspects or alternatives I am missing.


 You could always have the build tool split up the file in 
 multiple smaller files and read one of the smaller files with 
 the import expression.


Which build tool are you refering to? an existing one or build 
one oneself to do this job?
Oct 25 2020
parent Jacob Carlborg <doob me.com> writes:
On Sunday, 25 October 2020 at 16:50:09 UTC, Jack wrote:


 Which build tool are you refering to? an existing one or build 
 one oneself to do this job?


It should work with any build tool that has hooks to execute 
arbitrary commands.

--
/Jacob Carlborg
Oct 26 2020
prev sibling next sibling parent Jack <jckj33 gmail.com> writes:
On Friday, 23 October 2020 at 16:42:00 UTC, data pulverizer wrote:

 Hi all,

 the `import` function allows a file to be read at compile time, 
 which opens up great opportunities for (mostly binary) file IO, 
 where data types can be coded into files - the user doesn't 
 need to know data types ahead of time. As specified in my old 
 blog article: 
 https://www.active-analytics.com/blog/reading-idx-files-in-d/.

 Binary files can be very large and reading the whole file at 
 compile time is often unnecessary.

 This isn't exactly the intended use for the function, but there 
 it is. Since compile time read capability already exists and is 
 useful, adding capability to be able to read only a portion of 
 the file at compile time in a given location is advantageous 
 and has utility.

 For me it's not make-or-break, it just something very useful 
 and I think has clear use case. Please let me know if there are 
 aspects or alternatives I am missing.

 Thanks


That's a really good approach, I always wanted to do things like 

Could be good if we find a out a way to read part of the file and 
lazy-reading.Sure an external tool could do the first job, I mean 
if we could find a way in pure D.
Oct 25 2020
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/23/20 12:42 PM, data pulverizer wrote:

 Hi all,
 
 the `import` function allows a file to be read at compile time, which 
 opens up great opportunities for (mostly binary) file IO, where data 
 types can be coded into files - the user doesn't need to know data types 
 ahead of time. As specified in my old blog article: 
 https://www.active-analytics.com/blog/reading-idx-files-in-d/.
 
 Binary files can be very large and reading the whole file at compile 
 time is often unnecessary.
 
 This isn't exactly the intended use for the function, but there it is. 
 Since compile time read capability already exists and is useful, adding 
 capability to be able to read only a portion of the file at compile time 
 in a given location is advantageous and has utility.
 
 For me it's not make-or-break, it just something very useful and I think 
 has clear use case. Please let me know if there are aspects or 
 alternatives I am missing.


I see no reason why the compiler can't do something special with:

enum slicedFile = import("foo.bin")[0 .. 100];

One thing that comes to mind -- most OSes support memory-mapped files. 
If import simply did a memory mapped file internally, and duplicated the 
string once actually used somewhere, then you can avoid full reading of 
the data until it's necessary.

-Steve
Oct 26 2020