www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - [windows] Can't delete a closed file?

reply Cym13 <cpicard openmailbox.org> writes:
Hi,

this is likely not related to D itself but hopefully someone can 
help me with this since I'm rather new to windows programming, I 
mainly work on linux. I'm trying to bundle a DLL in a binary, 
write it in a temp folder, use it and remove the dangling file.

So far I have the following file:

     import std;

     void main(string[] args) {
         import core.runtime;
         static immutable libcurl = import("libcurl.dll");

         import std.file: write;
         auto libpath = tempDir.buildPath("libcurl.dll");
         libpath.write(libcurl);

         auto libcurlMem = rt_loadLibrary(libpath.toStringz);

         import std.net.curl;
         "https://dlang.org/".byLine.count.writeln;

         rt_unloadLibrary(libcurlMem);
         remove(libpath);
     }

Compiled with:  dmd.exe -Jlibdir test.d

It almost work, I can write, load and use the library, but when 
it comes to
removing it nothing works.

std.file.FileException std\file.d(1045): 
C:\users\cym13\Temp\libcurl.dll: Access denied.
----------------
0x00402377 in EntryPoint
0x00413BC7 in EntryPoint
0x00413B49 in EntryPoint
0x004139E3 in EntryPoint
0x0040B77F in EntryPoint
0x7B4754C2 in call_process_entry
0x7B477FC6 in ExitProcess
0x7B4754CE in call_process_entry

I tried using an explicit File handle to explicitely close the 
file after
writing to it but that doesn't change anything.

I'm pretty sure I'm missing something basic about the way windows 
handles
open files but I don't know what, could someone explain why this 
doesn't work
the way I expect it to?
May 09 2019
next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 9 May 2019 at 10:09:23 UTC, Cym13 wrote:

 Hi,

 this is likely not related to D itself but hopefully someone 
 can help me with this since I'm rather new to windows 
 programming, I mainly work on linux. I'm trying to bundle a DLL 
 in a binary, write it in a temp folder, use it and remove the 
 dangling file.

 [...]


I can't explain the behaviour, but you could store the temp file 
name in a string array and remove these file in the module 
destructor. That's the way how dub handles temp files/folders.

https://github.com/dlang/dub/blob/master/source/dub/internal/utils.d#L98

Kind regards
Andre
May 09 2019
parent reply Cym13 <cpicard openmailbox.org> writes:
On Thursday, 9 May 2019 at 11:07:53 UTC, Andre Pany wrote:

 On Thursday, 9 May 2019 at 10:09:23 UTC, Cym13 wrote:

 Hi,

 this is likely not related to D itself but hopefully someone 
 can help me with this since I'm rather new to windows 
 programming, I mainly work on linux. I'm trying to bundle a 
 DLL in a binary, write it in a temp folder, use it and remove 
 the dangling file.

 [...]


 I can't explain the behaviour, but you could store the temp 
 file name in a string array and remove these file in the module 
 destructor. That's the way how dub handles temp files/folders.

 https://github.com/dlang/dub/blob/master/source/dub/internal/utils.d#L98

 Kind regards
 Andre


No luck there.
May 09 2019
parent reply Cym13 <cpicard openmailbox.org> writes:
On Thursday, 9 May 2019 at 11:31:20 UTC, Cym13 wrote:

 ...


To dismiss any doubt about AV or other processes coming into play 
I took the binary and ran it with wine on linux with the exact 
same end result.
For reference my windows system is a 64b windows 10.
May 09 2019
parent reply Rene Zwanenburg <renezwanenburg gmail.com> writes:
On Thursday, 9 May 2019 at 12:33:37 UTC, Cym13 wrote:

 On Thursday, 9 May 2019 at 11:31:20 UTC, Cym13 wrote:

 ...


 To dismiss any doubt about AV or other processes coming into 
 play I took the binary and ran it with wine on linux with the 
 exact same end result.
 For reference my windows system is a 64b windows 10.


You could try to use the find handle function in Process Explorer 
to figure out what process has the file open:

https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer
May 09 2019
parent reply Cym13 <cpicard openmailbox.org> writes:
On Thursday, 9 May 2019 at 13:02:51 UTC, Rene Zwanenburg wrote:

 On Thursday, 9 May 2019 at 12:33:37 UTC, Cym13 wrote:

 On Thursday, 9 May 2019 at 11:31:20 UTC, Cym13 wrote:

 ...


 To dismiss any doubt about AV or other processes coming into 
 play I took the binary and ran it with wine on linux with the 
 exact same end result.
 For reference my windows system is a 64b windows 10.


 You could try to use the find handle function in Process 
 Explorer to figure out what process has the file open:

 https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer


I did just that and my test program truly is the only process on 
the system having that file open. Unless I'm missing something 
that process-explorer doesn't see and is even true in wine's 
ultra light environment comprised of a single process, this is 
definitely not the issue.
May 09 2019
next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 9 May 2019 at 13:18:44 UTC, Cym13 wrote:

 On Thursday, 9 May 2019 at 13:02:51 UTC, Rene Zwanenburg wrote:

 On Thursday, 9 May 2019 at 12:33:37 UTC, Cym13 wrote:

 On Thursday, 9 May 2019 at 11:31:20 UTC, Cym13 wrote:

 ...


 To dismiss any doubt about AV or other processes coming into 
 play I took the binary and ran it with wine on linux with the 
 exact same end result.
 For reference my windows system is a 64b windows 10.


 You could try to use the find handle function in Process 
 Explorer to figure out what process has the file open:

 https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer


 I did just that and my test program truly is the only process 
 on the system having that file open. Unless I'm missing 
 something that process-explorer doesn't see and is even true in 
 wine's ultra light environment comprised of a single process, 
 this is definitely not the issue.


Can you reproduce the issue with other Dlls or is it only 
reproducible with curl dll? Does the issue with curl dll also 
exists if you do not call the curl function?

Kind regards
Andre
May 09 2019
parent Cym13 <cpicard purrfect.fr> writes:
On Thursday, 9 May 2019 at 15:05:10 UTC, Andre Pany wrote:

 Can you reproduce the issue with other Dlls or is it only 
 reproducible with curl dll? Does the issue with curl dll also 
 exists if you do not call the curl function?

 Kind regards
 Andre


I didn't have the time to test with another dll just yet but in 
that specific case the issue disappears if I don't call its 
functions.

That's interesting.
May 10 2019
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 9 May 2019 at 13:18:44 UTC, Cym13 wrote:

 On Thursday, 9 May 2019 at 13:02:51 UTC, Rene Zwanenburg wrote:

 On Thursday, 9 May 2019 at 12:33:37 UTC, Cym13 wrote:

 [...]


 You could try to use the find handle function in Process 
 Explorer to figure out what process has the file open:

 https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer


 I did just that and my test program truly is the only process 
 on the system having that file open. Unless I'm missing 
 something that process-explorer doesn't see and is even true in 
 wine's ultra light environment comprised of a single process, 
 this is definitely not the issue.


Which C runtime are you using?

The old and buggy DigitalMars one or the official MS one?
May 10 2019
parent reply Cym13 <cpicard purrfect.fr> writes:
On Friday, 10 May 2019 at 07:09:45 UTC, Seb wrote:

 On Thursday, 9 May 2019 at 13:18:44 UTC, Cym13 wrote:

 On Thursday, 9 May 2019 at 13:02:51 UTC, Rene Zwanenburg wrote:

 On Thursday, 9 May 2019 at 12:33:37 UTC, Cym13 wrote:

 [...]


 You could try to use the find handle function in Process 
 Explorer to figure out what process has the file open:

 https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer


 I did just that and my test program truly is the only process 
 on the system having that file open. Unless I'm missing 
 something that process-explorer doesn't see and is even true 
 in wine's ultra light environment comprised of a single 
 process, this is definitely not the issue.


 Which C runtime are you using?

 The old and buggy DigitalMars one or the official MS one?


I really don't know, how can I find out? I litterally just used 
dmd on the script above with no special options then ran the 
resulting exe.
May 10 2019
parent Seb <seb wilzba.ch> writes:
On Friday, 10 May 2019 at 08:07:32 UTC, Cym13 wrote:

 Which C runtime are you using?

 The old and buggy DigitalMars one or the official MS one?


 I really don't know, how can I find out? I litterally just used 
 dmd on the script above with no special options then ran the 
 resulting exe.


Yeah, so you're using the old and buggy DigitalMars C runtime.
So you can use:

- use LDC
- use -m64
- use -ms32mscoff

and there's a high chance that your problem will go away ;-)

They all use the universal C runtime.
May 10 2019
prev sibling next sibling parent reply Rumbu <rumbu rumbu.ro> writes:
On Thursday, 9 May 2019 at 10:09:23 UTC, Cym13 wrote:

 Hi,

 this is likely not related to D itself but hopefully someone 
 can help me with this since I'm rather new to windows 
 programming, I mainly work on linux. I'm trying to bundle a DLL 
 in a binary, write it in a temp folder, use it and remove the 
 dangling file.

 So far I have the following file:

     import std;

     void main(string[] args) {
         import core.runtime;
         static immutable libcurl = import("libcurl.dll");

         import std.file: write;
         auto libpath = tempDir.buildPath("libcurl.dll");
         libpath.write(libcurl);

         auto libcurlMem = rt_loadLibrary(libpath.toStringz);

         import std.net.curl;
         "https://dlang.org/".byLine.count.writeln;

         rt_unloadLibrary(libcurlMem);
         remove(libpath);
     }

 Compiled with:  dmd.exe -Jlibdir test.d

 It almost work, I can write, load and use the library, but when 
 it comes to
 removing it nothing works.

 std.file.FileException std\file.d(1045): 
 C:\users\cym13\Temp\libcurl.dll: Access denied.
 ----------------
 0x00402377 in EntryPoint
 0x00413BC7 in EntryPoint
 0x00413B49 in EntryPoint
 0x004139E3 in EntryPoint
 0x0040B77F in EntryPoint
 0x7B4754C2 in call_process_entry
 0x7B477FC6 in ExitProcess
 0x7B4754CE in call_process_entry

 I tried using an explicit File handle to explicitely close the 
 file after
 writing to it but that doesn't change anything.

 I'm pretty sure I'm missing something basic about the way 
 windows handles
 open files but I don't know what, could someone explain why 
 this doesn't work
 the way I expect it to?


Since deploying a dll is a suspect behaviour outside a normal 
installation process, most probably you have a lock on the file 
put by windows defender or an antivirus if installed.

FreeLibrary doesn't guarantee that the dll file is closed, maybe 
other windows processes are accessing it (like prefetch).

Anyway, you are deploying a dll just to read a file, It's over 
engineering, use WinAPI UrlDownloadToFile instead or any other 
winapi functions.
May 09 2019
parent Cym13 <cpicard openmailbox.org> writes:
On Thursday, 9 May 2019 at 11:11:56 UTC, Rumbu wrote:

 Since deploying a dll is a suspect behaviour outside a normal 
 installation process, most probably you have a lock on the file 
 put by windows defender or an antivirus if installed.


Thanks for your input but I'm absolutely certain that it's not 
related to AV since I made sure to test in an AV-free environment 
(and I'd expect AV to kick in before loading the DLL, but I may 
be wrong on that).


 FreeLibrary doesn't guarantee that the dll file is closed, 
 maybe other windows processes are accessing it (like prefetch).

 Anyway, you are deploying a dll just to read a file, It's over 
 engineering, use WinAPI UrlDownloadToFile instead or any other 
 winapi functions.


Well, that's just a test script, it's feature or DLL doesn't 
matter.
May 09 2019
prev sibling next sibling parent reply wjoe <invalid example.com> writes:
On Thursday, 9 May 2019 at 10:09:23 UTC, Cym13 wrote:

 Hi,

 this is likely not related to D itself but hopefully someone 
 can help me with this since I'm rather new to windows 
 programming, I mainly work on linux. I'm trying to bundle a DLL 
 in a binary, write it in a temp folder, use it and remove the 
 dangling file.

 [...]


That's a windows "feature". You can't delete files that are in 
use. The fact that you close the file in your program isn't a 
guarantee that there's no other open handles around.

For the same reason you have to restart windows after every 
update to apply it.
May 10 2019
parent reply Temtaime <temtaime gmail.com> writes:
Lol, you don't have to load and unload the curl dll.
std.net.curl have its own lazy libcurl loader. But i'm not sure 
if it tries to find the dll in the temp directory. If it is the 
case, then it simply doesn't unload the dll when you have called 
some function from it.
May 10 2019
parent Cym13 <cpicard openmailbox.org> writes:
On Friday, 10 May 2019 at 15:06:46 UTC, Temtaime wrote:

 Lol, you don't have to load and unload the curl dll.
 std.net.curl have its own lazy libcurl loader. But i'm not sure 
 if it tries to find the dll in the temp directory. If it is the 
 case, then it simply doesn't unload the dll when you have 
 called some function from it.


As I said it's more about the general idea of bundling DLLs in 
than libcurl itself, but even in that case it doesn't search the 
temp directory for it. Besides, if it did, it would be a pretty 
important vulnerability.
May 10 2019
prev sibling parent reply Machine Code <jckj33 gmail.com> writes:
On Thursday, 9 May 2019 at 10:09:23 UTC, Cym13 wrote:

 Hi,

 this is likely not related to D itself but hopefully someone 
 can help me with this since I'm rather new to windows 
 programming, I mainly work on linux. I'm trying to bundle a DLL 
 in a binary, write it in a temp folder, use it and remove the 
 dangling file.

 So far I have the following file:

     import std;

     void main(string[] args) {
         import core.runtime;
         static immutable libcurl = import("libcurl.dll");

         import std.file: write;
         auto libpath = tempDir.buildPath("libcurl.dll");
         libpath.write(libcurl);

         auto libcurlMem = rt_loadLibrary(libpath.toStringz);

         import std.net.curl;
         "https://dlang.org/".byLine.count.writeln;

         rt_unloadLibrary(libcurlMem);
         remove(libpath);
     }

 Compiled with:  dmd.exe -Jlibdir test.d

 It almost work, I can write, load and use the library, but when 
 it comes to
 removing it nothing works.

 std.file.FileException std\file.d(1045): 
 C:\users\cym13\Temp\libcurl.dll: Access denied.
 ----------------
 0x00402377 in EntryPoint
 0x00413BC7 in EntryPoint
 0x00413B49 in EntryPoint
 0x004139E3 in EntryPoint
 0x0040B77F in EntryPoint
 0x7B4754C2 in call_process_entry
 0x7B477FC6 in ExitProcess
 0x7B4754CE in call_process_entry

 I tried using an explicit File handle to explicitely close the 
 file after
 writing to it but that doesn't change anything.

 I'm pretty sure I'm missing something basic about the way 
 windows handles
 open files but I don't know what, could someone explain why 
 this doesn't work
 the way I expect it to?


Well, I've had similar issue. The error message says "access 
denied" which I believe refers to the tmp directory; i.e, the 
user that is running your executable has no permissions to delete 
that file. To be honest, I find all the permissions issues on 
Windows a pain the pass so I did a simple tmp files manager. My 
application create a tmp folder in same folder as in the 
executable's directory then delete it when the application 
finishs.
I also did write my own function to geneate a random string meant 
to be used as tmp filename.

To delete the file, it might be finishing something and the OS 
may hold it for some while; so try to delete the file a couple of 
times, with some interval between the attemps, I do something 
like this:

void deleteFile(string filename)
{
	import std.file : remove, FileException;

	enum int nAttempts = 5;
	enum int interval = 200; // value in ms

	int n = nAttempts;
	do
	{
		try
		{
			remove(filename);
		}
		catch(FileException)
		{
			// the file may be temporary busy or something, so we try 
again after a while, few times.
			sleep(interval);
		}
	} while(n --> 0);
}
May 10 2019
parent Rumbu <rumbu rumbu.ro> writes:
On Friday, 10 May 2019 at 19:10:05 UTC, Machine Code wrote:

 Well, I've had similar issue. The error message says "access 
 denied" which I believe refers to the tmp directory; i.e, the 
 user that is running your executable has no permissions to 
 delete that file.


Well, this has nothing to do with access rights, the user have 
full access to his own temp folder even in the most restrictive 
windows configuration.

According to winapi docs, FreeLibrary doesn't guarantee file 
close op, and that's why you cannot delete it. Internally, 
windows maps a memory handle directly into the dll file and that 
pointer remains active until the application exits.

Actually, I ran your example with several sequential calls to 
FreeLibrary, all returning  success, even if it's clear that the 
internal reference count touched zero after the first call.

The lock of the file disappears effectively after the application 
has exited and the dll can be safely deleted afterwards.

If you want this dll deploying mechanism (which I consider a bad 
approach, contributing to the well known dll hell) check at 
startup if the dll already exists, delete it and redeploy if the 
existing one doesn't satisfy your version requirements.

To avoid dll hell, dlls must be deployed in the system dir, or at 
least in the application folder through a normal installation 
process. In our network environment governed by a restrictive 
GPO, your application will never run.
May 10 2019