• Jay Norwood (29/29) Apr 15 2012 I'm trying to figure out how to achieve folder deletion times
• Kagamin (1/1) Apr 16 2012 Do you use FILE_FLAG_SEQUENTIAL_SCAN too?
• Jay Norwood (32/33) Apr 16 2012 The std.file.write does use FILE_FLAG_SEQUENTIAL_SCAN

"Jay Norwood" <jayn prismnet.com> writes:

I'm  trying to figure out how to achieve folder deletion times
close to the times achieved with the parallel rmd after myDefrag
sortByName on a folder.  It takes less than 3.5 secs for a 2G
layout that has been sorted, and with the rmd configured so that
it also works on a sorted list.  This is a huge improvement over
standard folder delete times for a folder of this size, which can
be up in the 1.5 to 3 minute range.  However the myDefrag takes
around 15 minutes to run on the 2G folder.

I dumped the lcn info for the layout sorted with myDefrag
sortByName, and I see that one of the things done is the
directory entries have been sorted in order among the files.
Directories grow as subfiles and subdirectories are added, and
since there doesn't seem to be a way to reserve room for
directory entries in a newly created directory (is there a way?),
it looks like defrag movements would be necessary to achieve the
same ordered layout.

Also, I assumed that ntfs would create files in sequential lcn
addresses as new files are added, and this doesn't seem to be the
case, at least when using  the std.file.write().  When writing
full files with std.file.write(), in sorted order, the resulting
folders still take over 1.5 min to delete.

  I replaced std.file.write() call with a function that creates an 
empty file of a the specified length using SetFilePointerEx and 
SetEndOfFile.  I sent the sorted filenames and their sizes to it 
from the zip file,  and
deleting the resulting 2G folder takes on the order of 15 secs.
So, I'm thinking there may be some problem with the order ntfs is
writing out buffered files to disk when using std.file.write().
Does anyone have some insight into this issue?

"Kagamin" <spam here.lot> writes:

Do you use FILE_FLAG_SEQUENTIAL_SCAN too?

"Jay Norwood" <jayn prismnet.com> writes:

On Monday, 16 April 2012 at 09:16:09 UTC, Kagamin wrote:
 Do you use FILE_FLAG_SEQUENTIAL_SCAN too?

The std.file.write does use FILE_FLAG_SEQUENTIAL_SCAN
void write(in char[] name, const void[] buffer)

my experimental code to create the empty file also uses it, but 
it doesn't write any buffers, just seeks to a position and calls 
SetEndOfFile.

void createEmptyFile(in char[] name, ulong length)
{
     version(Windows)
     {
         alias TypeTuple!(GENERIC_WRITE, 0, null, CREATE_ALWAYS,
						 FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN,
						 HANDLE.init)
             defaults;
         auto h = useWfuncs
             ? CreateFileW(std.utf.toUTF16z(name), defaults)
             : CreateFileA(toMBSz(name), defaults);

         cenforce(h != INVALID_HANDLE_VALUE, name);
         scope(exit) cenforce(CloseHandle(h), name);

		 SetFilePointerEx (h,  *cast(long *) &length,null, 0);
		 SetEndOfFile(h);
     }
     //else version(Posix)  not implemented
     //    return writeImpl(name, buffer, O_CREAT | O_WRONLY | 
O_TRUNC);
}

I tried a quick test of writing without the buffering, to see if 
that would provide results similar to the createEmptyFile() 
experiment. Unfortunately the return buffers from the uncompress  
call (I'm experimenting with unzip) are not compatible with 
unbuffered writing requirements, so the uncompressed data would 
need to be realigned and/or resized  ... not clear at the moment.