www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - temporary files - what is the resolution?

reply Marco Leise <Marco.Leise gmx.de> writes:

537 fixed (http://d.puremagic.com/issues/show_bug.cgi?id=3D7537), which I h=
ad to work around before.
Seeing as this is a basic C library function, I was wondering if D could ha=
ve a modern version. Looking at the MSDN it seems, like Microsoft added qui=
te a few functions to the Visual Studio C Runtime over the years:

           directory         generates  security enhanced=C2=B9

tmpnam_s   current working=C2=B2  filename   yes
tmpnam     current working=C2=B2  filename   no
_tempnam   TMP/user defined  filename   no
tmpfile_s  system root       FILE ptr=C2=B3  yes
tmpfile    system root       FILE ptr=C2=B3  no

=C2=B9 =3D http://msdn.microsoft.com/en-us/library/8ef0s5kh(v=3Dvs.80).aspx
=C2=B2 =3D functions that use the current working directory require that yo=
u set it to the desired location first, which makes them difficult to use i=
n a library, especially since the cwd is attached to the process and other =
user threads may not be prepared for sudden changes
=C2=B3 =3D these files are automagically deleted when the process exists

It looks like "GetTempFileName" (http://msdn.microsoft.com/en-us/library/wi=
ndows/desktop/aa364991(v=3Dvs.85).aspx) is the actual operating system func=
tion that is used by the C Runtime. I'd like to see a cross-platform soluti=
on that uses GetTempFileName on Windows. The result should be usable as a D=
 File. It is currently impossible to create a std.stream wrapped around a F=
ile.tmpfile, because it returns a C FILE pointer (and until 2.060, because =
of the administrator rights issue on Windows).
On Unix mkstemp (http://pubs.opengroup.org/onlinepubs/009695399/functions/m=
kstemp.html) looks like a pretty similar function as well. Unlike the Windo=
ws equivalent it returns an open file descriptor/handle in addition to the =
file name. In both cases the user has to delete the files after use. To me =
it looks they are flexible enough to write up set of cross-platform (Window=
s/MacOS X/Linux/FreeBSD) functions for temporary files.

The question is: Is the problem solved with the fixed bug, or is there inte=
rest in D functions for temporary files?

--=20
Marco
Jul 25 2012
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, July 26, 2012 03:49:27 Marco Leise wrote:

 The question is: Is the problem solved with the fixed bug, or is there
 interest in D functions for temporary files?


Of course there is. tmpfile is a horrible function. And actually, pretty much 
every C function for creating a temp file or even just a temp file name sucks. 
Most of them are marked as "don't ever use" and the like, and the ones that 
aren't are still quite poor (e.g. on some systems tmpnam, can only generate 26 
unique file names with a given prefix). As bad as tmpfile is, at least it _sort 
of_ works. But it doesn't give you a file name, and it deletes the file when 
it's closed, both which make it unusable for a lot of use cases.

I was working on a solution but ran into problems on Windows due to missing C 
function declarations which were required in order to be able to create a 
temporary file without introducing a race condition, and I temporarily tabled 
it. I need to get back to it.

I did create a nice, cross-platform function for generating a random file name 
which used D's random number generators and created a pull request for it ( 
https://github.com/D-Programming-Language/phobos/pull/691 ). By default, It 
puts the file in the directory returned by std.file.tempDir, but you can give
it 
a different directory if you want to.

But there's technically a race condition if you check for the file's existence 
and then create it if it didn't exist (and generate a new name if it did, 
check that one for existence, etc.). So, doing

auto file = File(std.file.tempFile());

would have technically had the potential to cause problems (though the file 
name had enough random characters in it to be as good or better than a UUID 
for uniqueness, so practically speaking, it was probably okay even if it was 
theoretically a problem). So, it wasn't merged. If we're going to add 
something, it will be a function which creates a file with a random name but 
which doesn't expose the function for generating the random name.

I intend to create std.stdio.File.tempFile to replace std.stdio.File.tmpfile. 
It will create a temporary file using a name generated with the implementation 
that I put together for generating a random file name and put it in the 
directory returned by std.file.tempDir (or wherever you tell it), and the File 
that it returns will be like any other File save for the fact that it 
generated the name for you. So, it won't get deleted on you or anything 
annoying like that.

I have a fully working implementation on Linux. I just need to sort out the 
Windows C function declaration problem before I can create a pull request for 
it. I expect that it'll be in 2.061.

- Jonathan M Davis
Jul 25 2012
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 25 Jul 2012 19:24:21 -0700
schrieb Jonathan M Davis <jmdavisProg gmx.com>:


 [=E2=80=A6] As bad as tmpfile is, at least it _sort=20
 of_ works. But it doesn't give you a file name, and it deletes the file w=


hen=20

 it's closed, both which make it unusable for a lot of use cases.
=20
 I was working on a solution but ran into problems on Windows due to missi=


ng C=20

 function declarations which were required in order to be able to create a=


=20

 temporary file without introducing a race condition, and I temporarily ta=


bled=20

 it. I need to get back to it.
=20
 I did create a nice, cross-platform function for generating a random file=


 name=20

 which used D's random number generators and created a pull request for it=


 (=20

 https://github.com/D-Programming-Language/phobos/pull/691 ). By default, =


It=20

 puts the file in the directory returned by std.file.tempDir, but you can =


give it=20

 a different directory if you want to.
=20
 But there's technically a race condition if you check for the file's exis=


tence=20

 and then create it if it didn't exist (and generate a new name if it did,=


=20

 check that one for existence, etc.).
=20
 [=E2=80=A6]

 I have a fully working implementation on Linux. I just need to sort out t=


he=20

 Windows C function declaration problem before I can create a pull request=


 for=20

 it. I expect that it'll be in 2.061.
=20
 - Jonathan M Davis


I've had to write something up myself, too. But with the closed bug in the =
Digital Mars C Runtime, I was wondering if there were actual operating syst=
em calls that cater for all our needs instead of writing our own functions.=
 And that's where I stumbled upon the similarities between "GetTempFileName=
" on Windows and "mkstemp" on Posix. What makes me uneasy are the limits of=
 the Windows API. A common wrapper around both would give us:

* free choice of base directory
* optional prefix string (only first 3 letters on Windows)
* can generate unique names (only up to 65,535 on Windows)
* avoids race conditions
* doesn't delete file after program termination

Actually I'd think mkstemp alone with it's random naming scheme and huge li=
mits is what we want. It returns both the name and an open file descriptor =
with access for the current user only. In other words, my version(Posix) wo=
uld be a one-liner ;). It's unfortunate that the Windows API doesn't offer =
something similarly secure and flexible.

Good luck with the race condition check on Windows!

--=20
Marco
Jul 27 2012
parent reply Marco Leise <Marco.Leise gmx.de> writes:
P.S.: There is a _mktemp_s in MS CRT, but it allows for only 26 unique names
per calling thread, directory and prefix :p

-- 
Marco
Jul 27 2012
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, July 27, 2012 17:37:20 Marco Leise wrote:

 P.S.: There is a _mktemp_s in MS CRT, but it allows for only 26 unique names
 per calling thread, directory and prefix :p


Yes. And some POSIX systems have exactly the same problem with mkstemp. And 
given how easy it is to write a function which just generates a random file 
name, I see no reason to deal with nonsense like that. As far as I can tell, 
_every_ C function for generating either a random file or a random file name
has 
a caveat of some sort.


 Good luck with the race condition check on Windows!


It's easily done with the right function calls. It's just that their 
declarations are missing. I'll get back to sorting that out soon.

- Jonathan M Davis
Jul 27 2012
prev sibling parent Brad Roberts <braddr puremagic.com> writes:
On Fri, 27 Jul 2012, Jonathan M Davis wrote:


 On Friday, July 27, 2012 17:37:20 Marco Leise wrote:

 P.S.: There is a _mktemp_s in MS CRT, but it allows for only 26 unique names
 per calling thread, directory and prefix :p


 
 Yes. And some POSIX systems have exactly the same problem with mkstemp. And 
 given how easy it is to write a function which just generates a random file 
 name, I see no reason to deal with nonsense like that. As far as I can tell, 
 _every_ C function for generating either a random file or a random file name
has 
 a caveat of some sort.


Which has that limit?  I haven't found one yet.
Jul 27 2012
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, July 27, 2012 12:44:40 Brad Roberts wrote:

 On Fri, 27 Jul 2012, Jonathan M Davis wrote:

 On Friday, July 27, 2012 17:37:20 Marco Leise wrote:

 P.S.: There is a _mktemp_s in MS CRT, but it allows for only 26 unique
 names per calling thread, directory and prefix :p


 
 Yes. And some POSIX systems have exactly the same problem with mkstemp.
 And
 given how easy it is to write a function which just generates a random
 file
 name, I see no reason to deal with nonsense like that. As far as I can
 tell, _every_ C function for generating either a random file or a random
 file name has a caveat of some sort.


 
 Which has that limit?  I haven't found one yet.


Here's one page that talks about it, but I've seen other pages mentioning it 
when searching previously:

http://www.gnu.org/software/gnulib/manual/html_node/mkstemp.html

I do not believe that any of the platforms that we currently support have that 
particular problem, but I'm not about to trust an OS function that varies that 
much from platform to platform when I can easily create a solution which is 
guaranteed to work across all of the platforms that we support and will do so 
consistently.

- Jonathan M Davis
Jul 27 2012