digitalmars.D.learn - Coping files and folders
- Joel (12/12) Jan 23 2013 How does one copy folders (with files in them) including sub
- Jacob Carlborg (6/17) Jan 23 2013 I don't think Phobos currently has any functions for this. Someone
- monarch_dodra (64/83) Jan 24 2013 Mustn't be very hard to manually write "copyDir".
- Joel (6/90) Jan 24 2013 Brilliant! Thanks.
- Jay Norwood (12/14) Jan 24 2013 I posted a parallel implementation a while back, and also put it
- Joel (9/23) Jan 24 2013 Sounds cool. I've bookmarked your github hub link.
- =?UTF-8?B?QWxpIMOHZWhyZWxp?= (14/17) Jan 25 2013 It is the work unit size:
- Jacob Carlborg (4/12) Jan 24 2013 Any change you could turn this into a pull request and submit to Phobos?
- Jay Norwood (15/26) Jan 25 2013 Looks like I never promoted the write through code on github. I
How does one copy folders (with files in them) including sub folders, and creating any needed folders? Like: land\house\cat.d land\house\rat.exe land\house\bedroom\ants.txt to root\island\house\cat.d root\island\house\rat.exe root\island\house\bedroom\ants.txt One work around is to use 'system' (under std.process). -joelcnz
Jan 23 2013
On 2013-01-24 07:28, Joel wrote:How does one copy folders (with files in them) including sub folders, and creating any needed folders? Like: land\house\cat.d land\house\rat.exe land\house\bedroom\ants.txt to root\island\house\cat.d root\island\house\rat.exe root\island\house\bedroom\ants.txt One work around is to use 'system' (under std.process).I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove. -- /Jacob Carlborg
Jan 23 2013
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg wrote:On 2013-01-24 07:28, Joel wrote:Mustn't be very hard to manually write "copyDir". The problem with "copyDir" is its transactional behavior: What should it do in case of a failure mid copy? Bail? Cleanup? Continue? Anyways, I just threw this together. The first version bails on first error. The second version keeps going as much as it can, and returns true on success. The caller is then free to (or not to) call rmdirRecurse in case of failure. //Throws exception on first error. void copyDir(string inDir, string outDir) { if (!exists(outDir)) mkdir(outDir); else if (!isDir(outDir)) throw new FileException(format("Destination path %s is not a folder.", outDir)); foreach (entry; dirEntries(inDir.idup, SpanMode.shallow)) { auto fileName = baseName(entry.name); auto destName = buildPath(outDir, fileName); if (entry.isDir()) copyDir(entry.name, destName); else copy(entry.name, destName); } } //Silently keeps going as much as it can, then returns true on success, //or false if an error occured. bool copyDirSilent(string inDir, string outDir) { if (!exists(outDir)) { auto e = collectException(mkdir(outDir)); if (e !is null) return false; } else if (!isDir(outDir)) return false; foreach (entry; dirEntries(inDir, SpanMode.shallow)) { auto fileName = baseName(entry.name); auto destName = buildPath(outDir, fileName); if (entry.isDir()) { bool b = copyDirSilent(entry.name, destName); if (b == false) return false; } else { auto e = collectException(mkdir(outDir)); if (e !is null) return false; } } return true; }How does one copy folders (with files in them) including sub folders, and creating any needed folders? Like: land\house\cat.d land\house\rat.exe land\house\bedroom\ants.txt to root\island\house\cat.d root\island\house\rat.exe root\island\house\bedroom\ants.txt One work around is to use 'system' (under std.process).I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove.
Jan 24 2013
"Joel" <joelcnz gmail.com> Brilliant! Thanks. I think the silent one has errors. But I've used the first function in my program now. With mine, strait after doing the copying is was supposed then update a file, but didn't. Your one seems to work though. On Thursday, 24 January 2013 at 08:58:06 UTC, monarch_dodra wrote:On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg wrote:On 2013-01-24 07:28, Joel wrote:Mustn't be very hard to manually write "copyDir". The problem with "copyDir" is its transactional behavior: What should it do in case of a failure mid copy? Bail? Cleanup? Continue? Anyways, I just threw this together. The first version bails on first error. The second version keeps going as much as it can, and returns true on success. The caller is then free to (or not to) call rmdirRecurse in case of failure. //Throws exception on first error. void copyDir(string inDir, string outDir) { if (!exists(outDir)) mkdir(outDir); else if (!isDir(outDir)) throw new FileException(format("Destination path %s is not a folder.", outDir)); foreach (entry; dirEntries(inDir.idup, SpanMode.shallow)) { auto fileName = baseName(entry.name); auto destName = buildPath(outDir, fileName); if (entry.isDir()) copyDir(entry.name, destName); else copy(entry.name, destName); } } //Silently keeps going as much as it can, then returns true on success, //or false if an error occured. bool copyDirSilent(string inDir, string outDir) { if (!exists(outDir)) { auto e = collectException(mkdir(outDir)); if (e !is null) return false; } else if (!isDir(outDir)) return false; foreach (entry; dirEntries(inDir, SpanMode.shallow)) { auto fileName = baseName(entry.name); auto destName = buildPath(outDir, fileName); if (entry.isDir()) { bool b = copyDirSilent(entry.name, destName); if (b == false) return false; } else { auto e = collectException(mkdir(outDir)); if (e !is null) return false; } } return true; }How does one copy folders (with files in them) including sub folders, and creating any needed folders? Like: land\house\cat.d land\house\rat.exe land\house\bedroom\ants.txt to root\island\house\cat.d root\island\house\rat.exe root\island\house\bedroom\ants.txt One work around is to use 'system' (under std.process).I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove.
Jan 24 2013
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg wrote:Someone posted code in these newsgroups of a parallel implementation of copy and remove.I posted a parallel implementation a while back, and also put it on github. The parallel trick is to create the folder structure first, then populate the folders with the files. Best for ssd drives. I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it. This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd. https://github.com/jnorwood
Jan 24 2013
Sounds cool. I've bookmarked your github hub link.
Just wondering with this:
// parallel foreach for regular files
foreach(fn ; taskPool.parallel(files,100)) {
string dfn = destRoot ~ fn[srcLen..$];
copy(fn,dfn);
}
What is the 100 number for?
On Friday, 25 January 2013 at 05:30:11 UTC, Jay Norwood wrote:
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg
wrote:
Someone posted code in these newsgroups of a parallel
implementation of copy and remove.
I posted a parallel implementation a while back, and also put
it on github.
The parallel trick is to create the folder structure first,
then populate the folders with the files. Best for ssd drives.
I also wrote a copy version that orders file sequence on disk
efficiently, using write through, and posted it. This speeds
up any subsequent file system operations done in the directory
order as if you have done a defrag. Great for hard drives, but
not needed for ssd.
https://github.com/jnorwood
Jan 24 2013
On 01/24/2013 10:23 PM, Joel wrote:// parallel foreach for regular files foreach(fn ; taskPool.parallel(files,100)) {What is the 100 number for?It is the work unit size: Quoting: The number of elements processed per work unit is controlled by the workUnitSize parameter. Smaller work units provide better load balancing, but larger work units avoid the overhead of communicating with other threads frequently to fetch the next work unit. Large work units also avoid false sharing in cases where the range is being modified. The less time a single iteration of the loop takes, the larger workUnitSize should be. For very expensive loop bodies, workUnitSize should be 1. An overload that chooses a default work unit size is also available. Ali
Jan 25 2013
On 2013-01-25 06:30, Jay Norwood wrote:I posted a parallel implementation a while back, and also put it on github. The parallel trick is to create the folder structure first, then populate the folders with the files. Best for ssd drives. I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it. This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd. https://github.com/jnorwoodAny change you could turn this into a pull request and submit to Phobos? -- /Jacob Carlborg
Jan 24 2013
Looks like I never promoted the write through code on github. I just posted it on the forum.dlang.org, along with some timings. I was only experimenting on ntfs, and so the extension I posted doesn't force write-through on linux. http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj forum.dlang.org After re-reading my post, I don't think I experimented with write through in copydir. I show the timings in the post for ntfs operations on a 2GB layout after unzipping with a write through version of unzip. I suspect you'd see similar improvement following a copydir that uses write through. The problem I saw with ntfs was that the target directory was very fragmented when allowing ntfs to do its lazy flushes. I tried several experiments, and using the write through was the only solution that resulted in a reasonably defragged target immediately as the result of the unzip.I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it. This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd. https://github.com/jnorwoodAny change you could turn this into a pull request and submit to Phobos?
Jan 25 2013