www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Phobos v3 idea: File interface

reply solidstate1991 <laszloszeremi outlook.com> writes:
In a nutshell:

Create either a base class or an interface, which defines basic 
file accessing capabilities, then let that be used for not just 
files on the disk, but for things like the network, memory mapped 
files (would be great for compressed files without writing them 
to the disk first, etc.).

Current workaround is to read the file at once, and while that 
has its own uses, also has its own downsides, especially at 
bigger filesizes.

I kind of did something like that with VFile, trying to copy the 
API of `std.stdio.File` as closely as possible, although I didn't 
end up using it as much as I wanted due to having to work on 
other libraries of mine instead of my data packaging one 
(Datapak). At least I'm pretty close to creating a minimalistic D 
native replacement of SDL, also I'm yet again working on my 
SDLang/KDL/XDL parser (latter being my own modification of 
SDLang, with ISO date/time and hexadecimal numbers).
Sep 19
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Thursday, 19 September 2024 at 20:39:33 UTC, solidstate1991 
wrote:

 In a nutshell:

 Create either a base class or an interface, which defines basic 
 file accessing capabilities, then let that be used for not just 
 files on the disk, but for things like the network, memory 
 mapped files (would be great for compressed files without 
 writing them to the disk first, etc.).


In Phobos v2, `File` is a wrapper around a C `FILE*`, which means 
that its feature are limited by what the C library supports.

In Phobos v3, we probably want our file abstraction to directly 
wrap the platform's native file descriptor/handle, so that we can 
offer access to all of the features that the platforms themselves 
support.
Sep 19
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 20/09/2024 9:18 AM, Paul Backus wrote:

 On Thursday, 19 September 2024 at 20:39:33 UTC, solidstate1991 wrote:

 In a nutshell:

 Create either a base class or an interface, which defines basic file 
 accessing capabilities, then let that be used for not just files on 
 the disk, but for things like the network, memory mapped files (would 
 be great for compressed files without writing them to the disk first, 
 etc.).


 
 In Phobos v2, `File` is a wrapper around a C `FILE*`, which means that 
 its feature are limited by what the C library supports.


Worth noting is that ``File`` is a reference counted struct.

Unfortunately this is where we enter hill to die on territory for me.

System handles must be deterministically cleaned up. If you do not do 
this, you can run out of handles and then crash.

This is very easy especially for the 100k req/s target, think 10 
seconds, not 2 minutes to run out of handles.

I am very willing to say, that RC types cannot go into GC memory (the 
memory itself can be GC, but the cleanup must be done via RC).

Right now RC is blocked by Walter on owner escape analysis. This is why 
I'm pushing so hard on this design wise currently.

I did ask Steven a couple months ago, and a new GC design will not lift 
this restriction. This is a hard requirement.
Sep 19
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Thursday, 19 September 2024 at 21:43:42 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 System handles must be deterministically cleaned up. If you do 
 not do this, you can run out of handles and then crash.

 This is very easy especially for the 100k req/s target, think 
 10 seconds, not 2 minutes to run out of handles.

 I am very willing to say, that RC types cannot go into GC 
 memory (the memory itself can be GC, but the cleanup must be 
 done via RC).

 Right now RC is blocked by Walter on owner escape analysis. 
 This is why I'm pushing so hard on this design wise currently.


I don't see why this has to be a hard blocker. D is already 
perfectly capable of using RC for the object lifetime 
(construction and destruction) and GC for the memory lifetime 
(allocation and deallocation).
Sep 20
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 21/09/2024 5:34 AM, Paul Backus wrote:

 On Thursday, 19 September 2024 at 21:43:42 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 System handles must be deterministically cleaned up. If you do not do 
 this, you can run out of handles and then crash.

 This is very easy especially for the 100k req/s target, think 10 
 seconds, not 2 minutes to run out of handles.

 I am very willing to say, that RC types cannot go into GC memory (the 
 memory itself can be GC, but the cleanup must be done via RC).

 Right now RC is blocked by Walter on owner escape analysis. This is 
 why I'm pushing so hard on this design wise currently.


 
 I don't see why this has to be a hard blocker. D is already perfectly 
 capable of using RC for the object lifetime (construction and 
 destruction) and GC for the memory lifetime (allocation and deallocation).


For structs yes.

But this thread is all about system handles being owned by classes, 
which don't offer this capability (currently).
Sep 20
parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 20 September 2024 at 18:03:28 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 For structs yes.

 But this thread is all about system handles being owned by 
 classes, which don't offer this capability (currently).


The original poster in this thread, solidstate1991, is talking 
about classes. My post said nothing about classes, and in fact I 
do not think using classes for Phobos v3's file abstraction is 
either necessary or desirable.
Sep 20
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 21/09/2024 6:57 AM, Paul Backus wrote:

 On Friday, 20 September 2024 at 18:03:28 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 For structs yes.

 But this thread is all about system handles being owned by classes, 
 which don't offer this capability (currently).


 
 The original poster in this thread, solidstate1991, is talking about 
 classes. My post said nothing about classes, and in fact I do not think 
 using classes for Phobos v3's file abstraction is either necessary or 
 desirable.


We seem to be talking a bit past each other.

Regardless we both agree for different reasons that classes are out at 
the API level.
Sep 20
prev sibling parent Adam Wilson <flyboynw gmail.com> writes:
On Thursday, 19 September 2024 at 21:43:42 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 System handles must be deterministically cleaned up. If you do 
 not do this, you can run out of handles and then crash.


I don't see this as a blocker. This can be handled by calling 
.close() in a scope(exit) block. I'll accept that manual resource 
management is more difficult to get right, but for now it works 
just fine. It's also pretty much the state of things right now, 
so we won't be going backwards.
Sep 22
prev sibling parent user1234 <user1234 12.de> writes:
On Thursday, 19 September 2024 at 20:39:33 UTC, solidstate1991 
wrote:

 In a nutshell:

 Create either a base class or an interface, which defines basic 
 file accessing capabilities, then let that be used for not just 
 files on the disk, but for things like the network, memory 
 mapped files (would be great for compressed files without 
 writing them to the disk first, etc.).


Given the latest developement of the discussion, I assume that 
what you suggest is really "no-DbI", right ? Or do you think that 
a static interface would do it ?

I dont know why but when you talk about bigger files that reminds 
me the Stream class... removed. For example, immediate thought is 
that, buffering of a big file could be hidden behind an 
InputRange.
Sep 20