www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - phobo's std.file is completely broke!

reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
For very long file names it is broke and every command fails. 
These paths are not all that long but over 256 limit. (For 
windows)

The problem this causes can be disastrous. If some check fails 
and runs code that isn't mean to run if the file exists, it could 
destroy data.


I replaced many of the std.file functions with executeShell(...) 
and using command line functions and they work. But then my code 
was still failing and it was because exist was returning false 
even though the file exists.

I'm sure this is not a big deal to some...
Sep 14 2018
next sibling parent reply bachmeier <no spam.net> writes:
On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)


Please file a bug report with reproducible examples if you 
believe it's a bug.
Sep 14 2018
next sibling parent reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo 
 wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)


 Please file a bug report with reproducible examples if you 
 believe it's a bug.


It's a bug, but how the hell can I reproduce examples when it 
depends on the file system? Do you want me to make a VM image and 
upload that too?


dirEntries(elem, SpanMode.shallow))

also fails. I've changed all my code over to using executeShell 
and everything works except that now I can't even iterate over 
directories that are too long.


This is ridiculous!


"It doesn't matter. When I compile a program or DLL in C/C++ and 
many other languages, I use the Windows headers. These headers 
define MAX_PATH to 260.
So the program will have the limit set by compile time, no matter 
what you do in the OS later on. There is no advantage with this 
setting for now, except that you now *may* pass long paths to API 
functions w/o the infamous \\?\ prefix, but you have no guarantee 
for it.

So to make this have the advantage that some people think it will 
provide, we'd need:

     wait until the Windows headers (in Visual Studio, or the DDK, 
or the Platform SDK or whatever) is properly updated, i.e. the 
runtime libs may take advantage of it
     wait until MS forces this setting to be enabled all the time, 
otherwise it's useless when it stays optional
     not checking any more for the target OS in the application, 
assuming Win 10 with the mentioned update

The latter will only be the case when an app can't target an 
Windows version below 10. So all in all it will take years until 
we get the advantage for standalone applications, and for TC 
these things don't matter for now anyway.
"

enum size_t MAX_PATH = 260;

!!


Seems this is the problem! What a great failure scenario! How 
bout at least check if the filename is larger than max path?

Seesh, the more I use D the more I regret it.
Sep 14 2018
next sibling parent Josphe Brigmo <JospheBrigmo gmail.com> writes:
https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maxpath

But because MAX_PATH is an enum, it can't be redefined without 
recompiling phobos, which means it will break later on...

It does say that this is not a problem with unicode... but...
Sep 14 2018
prev sibling next sibling parent Neia Neutuladh <neia ikeran.org> writes:
On Friday, 14 September 2018 at 19:42:39 UTC, Josphe Brigmo wrote:

 It's a bug, but how the hell can I reproduce examples when it 
 depends on the file system?


Something like this (though I don't know much about Windows, so 
this might be wrong):

auto path = getcwd;
auto dir = `0123456789abcdef`;
foreach (i; 0..20)
{
   path = buildPath(path, dir);
}
system("mkdir " ~ path);
dirEntries(path, SpanMode.shallow);

And describe what filesystem you're using (presumably NTFS), what 
OS and version, 64-bit vs 32-bit, etc.
Sep 14 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 14 September 2018 at 19:42:39 UTC, Josphe Brigmo wrote:

 "It doesn't matter. When I compile a program or DLL in C/C++ 
 and many other languages, I use the Windows headers. These 
 headers define MAX_PATH to 260.
 So the program will have the limit set by compile time, no 
 matter what you do in the OS later on. There is no advantage 
 with this setting for now, except that you now *may* pass long 
 paths to API functions w/o the infamous \\?\ prefix, but you 
 have no guarantee for it.

 So to make this have the advantage that some people think it 
 will provide, we'd need:

     wait until the Windows headers (in Visual Studio, or the 
 DDK, or the Platform SDK or whatever) is properly updated, i.e. 
 the runtime libs may take advantage of it
     wait until MS forces this setting to be enabled all the 
 time, otherwise it's useless when it stays optional
     not checking any more for the target OS in the application, 
 assuming Win 10 with the mentioned update

 The latter will only be the case when an app can't target an 
 Windows version below 10. So all in all it will take years 
 until we get the advantage for standalone applications, and for 
 TC these things don't matter for now anyway.
 "

 enum size_t MAX_PATH = 260;


MAX_PATH only applies to static arrays (usually on the stack), 
i.e. code which does this:

void doSomething()
{
     char fileName[MAX_PATH];
     ...
     SomeWindowsAPI(fileName);
}

D almost never does this - instead, it uses dynamically sized 
buffers which fit the entire file name. The only times MAX_PATH 
is used in Phobos is:

- thisExePath (return path to current .exe file)
- tempDir (return path to temporary directory)

Obviously neither of these is related to the problems you're 
seeing.
Sep 15 2018
parent reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
On Saturday, 15 September 2018 at 10:48:10 UTC, Vladimir 
Panteleev wrote:

 On Friday, 14 September 2018 at 19:42:39 UTC, Josphe Brigmo 
 wrote:

 "It doesn't matter. When I compile a program or DLL in C/C++ 
 and many other languages, I use the Windows headers. These 
 headers define MAX_PATH to 260.
 So the program will have the limit set by compile time, no 
 matter what you do in the OS later on. There is no advantage 
 with this setting for now, except that you now *may* pass long 
 paths to API functions w/o the infamous \\?\ prefix, but you 
 have no guarantee for it.

 So to make this have the advantage that some people think it 
 will provide, we'd need:

     wait until the Windows headers (in Visual Studio, or the 
 DDK, or the Platform SDK or whatever) is properly updated, 
 i.e. the runtime libs may take advantage of it
     wait until MS forces this setting to be enabled all the 
 time, otherwise it's useless when it stays optional
     not checking any more for the target OS in the 
 application, assuming Win 10 with the mentioned update

 The latter will only be the case when an app can't target an 
 Windows version below 10. So all in all it will take years 
 until we get the advantage for standalone applications, and 
 for TC these things don't matter for now anyway.
 "

 enum size_t MAX_PATH = 260;


 MAX_PATH only applies to static arrays (usually on the stack), 
 i.e. code which does this:

 void doSomething()
 {
     char fileName[MAX_PATH];
     ...
     SomeWindowsAPI(fileName);
 }

 D almost never does this - instead, it uses dynamically sized 
 buffers which fit the entire file name. The only times MAX_PATH 
 is used in Phobos is:

 - thisExePath (return path to current .exe file)
 - tempDir (return path to temporary directory)

 Obviously neither of these is related to the problems you're 
 seeing.


You are missing the point, MAX_PATH is more than just phobos. 
It's built in to the windows design. Windows enforces it.

All ansi api calls are limited by MAX_PATH.

The way to fix it is to use the wide api calls which are not 
limited or to use other tricks.

Phobos *NEEDS* to be modified to work with these newer OS's.

You wouldn't like it if phobos limit something that it didn't 
need that you would use, would you?

File operations are so common that this stuff is relevant to all 
that use windows(and some that don't).
Sep 15 2018
next sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 15 September 2018 at 10:57:56 UTC, Josphe Brigmo 
wrote:

 All ansi api calls are limited by MAX_PATH.

 The way to fix it is to use the wide api calls which are not 
 limited or to use other tricks.

 Phobos *NEEDS* to be modified to work with these newer OS's.


Phobos already uses the wide APIs where it can.

Sometimes it uses the C APIs, though, for C compatibility. One 
example of this is std.stdio.File, which is based on C FILE*.

We can change std.file to use Windows APIs on Windows, no 
problem. That also will help with Unicode compatibility. However, 
as far as I can see, this has already been done for the cases 
you're having problems with (remove, and dirEntries which doesn't 
even have a C API).

So, it looks like all you need to do is prepend \\?\ to your 
paths. If it still doesn't work, the problem is with Windows. 
(Well, the problem was with Windows to begin with...)
Sep 15 2018
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 15 September 2018 at 10:57:56 UTC, Josphe Brigmo 
wrote:

 Phobos *NEEDS* to be modified to work with these newer OS's.


You need to look at the source code before posting. The code for 
remove is literally

DeleteFileW(name);

it is a one-line wrapper, and obviously uses the unicode version.

https://github.com/dlang/phobos/blob/master/std/file.d#L1047
Sep 15 2018
parent reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
On Saturday, 15 September 2018 at 12:38:41 UTC, Adam D. Ruppe 
wrote:

 On Saturday, 15 September 2018 at 10:57:56 UTC, Josphe Brigmo 
 wrote:

 Phobos *NEEDS* to be modified to work with these newer OS's.


 You need to look at the source code before posting. The code 
 for remove is literally

 DeleteFileW(name);

 it is a one-line wrapper, and obviously uses the unicode 
 version.

 https://github.com/dlang/phobos/blob/master/std/file.d#L1047



It doesn't matter, the fact is that something in phobos is broke. 
Do you really expect me to do all the work? The fact that using 
executeShell or "\\?\" solves 90% of the problems(maybe all of 
them) proves that phobos is not up to par.

It's simple as that.

just because it uses unicode does not mean squat if it doesn't 
work. The docs from microsoft said that the unicode functions are 
suppose to not have the limitation, but that doesn't mean that 
using them is the only fix.

The fact is, I write a directory parser and it failed, then I 
spend the next day trying to get something relatively easy to be 
fixed that should already have been fixed.

I'm also not the only one with the problem.

Maybe you should spend some time on windows and using std.file 
with long paths. I bet you have actually never done it so you are 
obviously to the problems.

There may be "simple" fixes, but the fact is that something is 
wrong or else my code would work(you can blame me all you want 
but the facts are the facts and the code only breaks on long file 
names).

https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation

It is a bit more complicated than just saying that phobos is 
doing it's job. Of course, it could be some windows issue but 
given that


https://issues.dlang.org/show_bug.cgi?id=8967

The point is that these things should be thoroughly tested before 
blaming the end user. Whatever is going on, it should. File IO is 
very basic and common and for code to break silently or in ways 
that does not make sense is bad, regardless of if I existed in 
this universe or not.

How hard is it for phobos to detect long files and absolute files 
and such and prepend if necessary?
Sep 15 2018
next sibling parent reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
For example,

https://issues.dlang.org/show_bug.cgi?id=8967

`
  Jay Norwood 2014-03-18 18:01:59 UTC

More surprising is  attempting to remove a long directory path 
and having an exception occur.

The libraries are already copying the user's string and adding 
the 0 termination prior to calling the windows api, so it seems 
to me to be a reasonable place to make other modifications if 
they are needed to accomplish the intended operation.
`

Which is almost 5 years old and exactly the same problem I'm 
having...

Yet it is somehow my fault for not reading the source of phobos 
to make sure it is using unicode api? Which it is and it's still 
failing!

and, of course, no followup on that bug has been done...

again, this is why I generally end up regretting using D. I 
thought it would be easier to write a small utility in a few 
hours and it's turned in to a few days. Any other sane language 
and I would have been done with 1/10th of the frustration and I'd 
actually have working code(which I still don't) because rmdir is 
till failing for some reason.

This is the typical mindset with D. There are all these "minor" 
problems that people(the D community pretends are all that big a 
deal but when you couple all these problems together it results 
in a very unpleasant programming experience(out of all the 
languages I've programmed in D is about the worse in this regard).
Sep 15 2018
next sibling parent reply Rubn <ifoorget damn.it> writes:
On Saturday, 15 September 2018 at 12:59:25 UTC, Josphe Brigmo 
wrote:

 This is the typical mindset with D. There are all these "minor" 
 problems that people(the D community pretends are all that big 
 a deal but when you couple all these problems together it 
 results in a very unpleasant programming experience(out of all 
 the languages I've programmed in D is about the worse in this 
 regard).


You keep saying you regret using D, well let's go to C++ then. 
How are you going to solve this problem with C++?

It's a problem that can be worked around, if you are using the 
latest version of Windows it can be fixed by simply setting a 
registry entry. Then **all** your applications on your system 
will work with long file names.
Sep 15 2018
parent Josphe Brigmo <JospheBrigmo gmail.com> writes:
On Saturday, 15 September 2018 at 13:23:34 UTC, Rubn wrote:

 On Saturday, 15 September 2018 at 12:59:25 UTC, Josphe Brigmo 
 wrote:

 This is the typical mindset with D. There are all these 
 "minor" problems that people(the D community pretends are all 
 that big a deal but when you couple all these problems 
 together it results in a very unpleasant programming 
 experience(out of all the languages I've programmed in D is 
 about the worse in this regard).


 You keep saying you regret using D, well let's go to C++ then. 
 How are you going to solve this problem with C++?

 It's a problem that can be worked around, if you are using the 
 latest version of Windows it can be fixed by simply setting a 
 registry entry. Then **all** your applications on your system 
 will work with long file names.



is consistent in nature and runtime.

But, the damn problem, which you seem to not understand, is that 
once one chooses D to do something in and puts in time, then only 
do they find out how poor it works and then the choice is made to 
continue using it hoping for the best or start over from scratch.

The problem is that I'm an optimist at heart but every time that 
is tested with D. It works for something amazingly well, for 
other things amazingly poor.

But with peoples attitudes like yours it seems this will always 
be the case for D. It would be nice if people did what was 
required to make D as great as it could be rather than having the 
attitude "If you don't like it leave".

Usually when someone comes in to bitch about a problem with 
D(which is usually legit), there is a wall of "it's not our 
problem" attitudes.
Sep 15 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 15 September 2018 at 12:59:25 UTC, Josphe Brigmo 
wrote:

 The libraries are already copying the user's string and adding 
 the 0 termination prior to calling the windows api, so it seems 
 to me to be a reasonable place to make other modifications if 
 they are needed to accomplish the intended operation.


That only works for absolute paths.


 Yet it is somehow my fault for not reading the source of phobos 
 to make sure it is using unicode api? Which it is and it's 
 still failing!


Right. The problem is on the OS side.


 again, this is why I generally end up regretting using D.


Can you list some programming languages that achieve this task in 
a way you approve of?


 This is the typical mindset with D. There are all these "minor" 
 problems that people(the D community pretends are all that big 
 a deal but when you couple all these problems together it 
 results in a very unpleasant programming experience(out of all 
 the languages I've programmed in D is about the worse in this 
 regard).


Please drop this tone, it will make you no allies.
Sep 15 2018
next sibling parent reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
On Saturday, 15 September 2018 at 13:37:29 UTC, Vladimir 
Panteleev wrote:

 On Saturday, 15 September 2018 at 12:59:25 UTC, Josphe Brigmo 
 wrote:

 The libraries are already copying the user's string and adding 
 the 0 termination prior to calling the windows api, so it 
 seems to me to be a reasonable place to make other 
 modifications if they are needed to accomplish the intended 
 operation.


 That only works for absolute paths.


And that is all I'm using...


 Yet it is somehow my fault for not reading the source of 
 phobos to make sure it is using unicode api? Which it is and 
 it's still failing!


 Right. The problem is on the OS side.


 again, this is why I generally end up regretting using D.


 Can you list some programming languages that achieve this task 
 in a way you approve of?



python, perl, C++(yes, c++, we are not talking about language 
features but usability). The simple fact is that C++ can be used 
to do anything almost 100% correct while D can fail. D is only a 
better language, not a better compiler(except it's speed).

You know, there is so many problems with D but you do not care to 
see them.

Take the way the library is designed. strip? trim is the 
standard. Then you get things like exists. Instead of fileExists 
or dirExists. Not all that bad though, but what about slurp and 
others that are just odd and for people that are not professional 
D programmers requires one to look up the name of what they are 
trying to do.

I don't know how many times I have typed in a name of a function 
that is "standard" only to find out that is not what D used but 
some odd ball name... or worse, it differs by a character or two.

You can pretend all you want about how great D is, but D is not 
great, it is just great at some things. That goes for all 
languages, but at least some languages let you enjoy the 
experience.

With D, it seems the hard core users seem not to care about the 
experience or think it's great because they don't know how much 
better it can be.



 This is the typical mindset with D. There are all these 
 "minor" problems that people(the D community pretends are all 
 that big a deal but when you couple all these problems 
 together it results in a very unpleasant programming 
 experience(out of all the languages I've programmed in D is 
 about the worse in this regard).


 Please drop this tone, it will make you no allies.


I'm not here to make allies. Truth is not subjective and it 
doesn't depend on how many friends you have that believe in the 
same BS.
Sep 15 2018
next sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 15 September 2018 at 18:21:43 UTC, Josphe Brigmo 
wrote:

 Can you list some programming languages that achieve this task 
 in a way you approve of?



 python, perl, C++(yes, c++, we are not talking about language 
 features but usability).


I had a quick look at the implementations of some of the 
languages you mentioned. They use the C API or use the Windows 
API without the UNC prefix.

You are lying.
Sep 15 2018
prev sibling parent tide <tide tide.tide> writes:
On Saturday, 15 September 2018 at 18:21:43 UTC, Josphe Brigmo 
wrote:

 On Saturday, 15 September 2018 at 13:37:29 UTC, Vladimir 
 Panteleev wrote:

 Can you list some programming languages that achieve this task 
 in a way you approve of?



 python, perl, C++(yes, c++, we are not talking about language 
 features but usability). The simple fact is that C++ can be 
 used to do anything almost 100% correct while D can fail. D is 
 only a better language, not a better compiler(except it's 
 speed).


See you are just talking out of your ass right now. I just tried 
C++, it doesn't work. You can't use std::fstream with a path that 
is larger than 260. I also moved an executable to that large 
path. And guess what I couldn't even get Windows to run it. Not 
through powershell nor explorer. I can't even run applications 
like "ls" with powershell. Let alone "cd" into the folder. oddly 
enough the only thing that came close was git bash, which gave me 
an error message. While powerhshell just said couldn't find path.

```
Error: Current working directory has a path longer than allowed 
for a
Win32 working directory.
Can't start native Windows application from here.
```

I don't know how you can complain about D when Windows is so 
fundamentally broken for large files path, that even it's set of 
tools don't support it.

Another one, Python: 
https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation

It does not support long paths, you have to use \\?\ or enable it 
with the registry (only on newer Windows 10 versions).

If you are going to make claims -- Do Your Research.
Sep 15 2018
prev sibling parent reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
and the biggest problem is that I don't see any motivation in the 
D community to make things better. Anyone with the abilities to 
make it better in the right way simply does not care about having 
a proper plan to get D to where it needs to be. Hence, it gives 
me no hope that D will ever reach a status of usability that it 
should have. What's the point of all the fancy meta language 
capabilities if ultimately they are ineffective due to the 
ecosystem of D sucking?

What I have learned from D:

1. It has an amazing meta programming language. Of course most 
functional languages have even better but D allows systems 
programming and is also fast.

2. I am the most unproductive in D. While I can write meta code 
that does things 10x easier, it takes me 10x longer to code, 
debug, and revision... not because the meta programming is 
difficult but because the errors are pathetic(most of the time a 
huge amount of noise is created in the errors making me sift 
through a bunch of junk just to figure out what is going on) and 
usually not even related to the real problem. Just miss a 
semicolon and your program can explode with errors having nothing 
to do with that.

3. No one seems to care about fixing these deficits of D but only 
on making it more "attractive" on paper.

4. One can't expect a program to work properly, EVER! In other 
languages, when I write something, debug it, and test it, I am 
sure that it will generally work fine after and not have to worry 
in any way shape or form... and when it does break, it is usually 
obvious and easy to fix.

Be it some flaw in the language(such as what I have experienced 
with the paths and other things) or updating compilers or library 
features and it breaking something, etc.

All these problems, which are well known, and I'd bet you that D 
has the most forum posts of problems dealing with these types of 
things than any other language, are rarely addressed.

I mean, look at bugzillia... how many bugs have 5+ years and no 
one has done a damn thing? And these bugs being critical in many 
cases.

Again, this is the whole attitude "It's not our problem"...

Of course, once one of the hardcore D guys run in to the bug 
after actually doing some real programing of real applications, 
they might have to fix it and then it will get fixed.

It's not that D is terrible, it's that it's not great and it has 
the potential to be great but either no one realizes it or gives 
a shit. All you guys put in your life energy in to making D what 
it is but won't put in the energy to strengthen the weaknesses.

When you get tired of D, say like Dicebot, you'll leave and some 
other *idiot* will come along and have the same attitude and the 
process repeats. This is why it's called "kicking the can down 
the road" and why I used the term idiot(not meaning a lack of 
intelligence but a lack of foresight).


And this is why after 10+ years of D not really much has 
changed(even though a lot has changed). After another 20 years, 
same thing. A lot will change, but the same all problems(the 
weaknesses) will exist.
Sep 15 2018
parent bachmeier <no spam.net> writes:
On Saturday, 15 September 2018 at 18:33:36 UTC, Josphe Brigmo 
wrote:

 and the biggest problem is that I don't see any motivation in 
 the D community to make things better.


This is an open source project. If you're hoping that you can 
report that something doesn't work the way you want it to and a 
manager will assign a couple developers to fix it to do what you 
want, D is not for you.

That's reality even if you (and the many other new users posting 
these things) don't like reality. If you've got a million dollars 
to donate, that can be changed.
Sep 15 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, September 15, 2018 6:54:50 AM MDT Josphe Brigmo via 
Digitalmars-d wrote:

 On Saturday, 15 September 2018 at 12:38:41 UTC, Adam D. Ruppe

 wrote:

 On Saturday, 15 September 2018 at 10:57:56 UTC, Josphe Brigmo

 wrote:

 Phobos *NEEDS* to be modified to work with these newer OS's.


 You need to look at the source code before posting. The code
 for remove is literally

 DeleteFileW(name);

 it is a one-line wrapper, and obviously uses the unicode
 version.

 https://github.com/dlang/phobos/blob/master/std/file.d#L1047


 It doesn't matter, the fact is that something in phobos is broke.
 Do you really expect me to do all the work? The fact that using
 executeShell or "\\?\" solves 90% of the problems(maybe all of
 them) proves that phobos is not up to par.


Using std.file should be on par with using the Windows API from C or C++. It
doesn't try to fix the arguably broken behavior of the Windows API with
regards to long paths but requires that the programmer deal with them just
like they would in C/C++. The main differences are that the std.file
functions in question use D strings rather than C strings, and they
translate them to the UTF-16 C strings for you rather than requiring you to
do it. But they don't do anything like add "\\?\" for you any more than the
Windows API itself does that.

If you consider that to be broken, then sorry. For better or worse, it was
decided that it was better to let the programmer deal with those intricacies
rather than trying to tweak the input to make it work based on the idea that
that could have undesirable consequences in some circumstances. On some
level, that does suck, but the Windows API does not make it easy to make
this work like it would on a *nix system without introducing subtle bugs.

If you find that the std.file functions don't work whereas using the same
input to the Windows API functions in C/C++ would have, then there's a bug
in the D code, and it needs to be fixed, but if it acts the same as the
C/C++ code, then it's working as intended.

- Jonathan M Davis
Sep 15 2018
next sibling parent reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
On Saturday, 15 September 2018 at 23:06:57 UTC, Jonathan M Davis 
wrote:

 On Saturday, September 15, 2018 6:54:50 AM MDT Josphe Brigmo 
 via Digitalmars-d wrote:

 On Saturday, 15 September 2018 at 12:38:41 UTC, Adam D. Ruppe

 wrote:

 On Saturday, 15 September 2018 at 10:57:56 UTC, Josphe Brigmo

 wrote:

 Phobos *NEEDS* to be modified to work with these newer OS's.


 You need to look at the source code before posting. The code 
 for remove is literally

 DeleteFileW(name);

 it is a one-line wrapper, and obviously uses the unicode 
 version.

 https://github.com/dlang/phobos/blob/master/std/file.d#L1047


 It doesn't matter, the fact is that something in phobos is 
 broke. Do you really expect me to do all the work? The fact 
 that using executeShell or "\\?\" solves 90% of the 
 problems(maybe all of them) proves that phobos is not up to 
 par.


 Using std.file should be on par with using the Windows API from 
 C or C++. It doesn't try to fix the arguably broken behavior of 
 the Windows API with regards to long paths but requires that 
 the programmer deal with them just like they would in C/C++. 
 The main differences are that the std.file functions in 
 question use D strings rather than C strings, and they 
 translate them to the UTF-16 C strings for you rather than 
 requiring you to do it. But they don't do anything like add 
 "\\?\" for you any more than the Windows API itself does that.

 If you consider that to be broken, then sorry. For better or 
 worse, it was decided that it was better to let the programmer 
 deal with those intricacies rather than trying to tweak the 
 input to make it work based on the idea that that could have 
 undesirable consequences in some circumstances. On some level, 
 that does suck, but the Windows API does not make it easy to 
 make this work like it would on a *nix system without 
 introducing subtle bugs.


Do you not realize how moronic that is though?  You are expecting 
each user to know the correct behavior and to compensate for it. 
With that approach all you are doing is creating a whole mess of 
problems.

See, there is only one phobos but many users of phobos. You are 
expecting every single user to deal with it instead of dealing 
with it in one place.

Your reason is because "It's a problem with windows".

Both are "We'll just kick the can down the road cause the other 
guy kicked it to us".

Now, this is great if you want repeat customers and don't care 
about their sanity but terrible to make progress.


 If you find that the std.file functions don't work whereas 
 using the same input to the Windows API functions in C/C++ 
 would have, then there's a bug in the D code, and it needs to 
 be fixed, but if it acts the same as the C/C++ code, then it's 
 working as intended.


And that is precisely the problem. For some reason you don't get 
that because it is broke in windows, and you following windows, 
you are just perpetuating the "brokeness".  This is why D sucks 
the way it does, because of these types of mentalities of "It's 
not our problem".

You basically expect the customers, before eating to wash the 
dishes, cook the meal, set the table, etc. All you do is provide 
the food, and not all that great food... and when they are done 
you expect them to wash the dishes against, clean up their mess, 
and pay the bill.

I'm sure you'll never realize how wrong your mentality is, but at 
least you could do everyone a favor and think about what you are 
actually saying. Your logic might have a valid reason, but it is 
not producing the results that make the world a better place.

D won't ever get any traction with the wrong mentalities and I 
don't even know how it has made it this far.

Trust me, if you expect the user to know everything and also 
solve all the problems over and over and over and over, you will 
have very few users.

But, keep on doing what your doing, it's worked so well, we will 
see how far it gets D.

If D is as perfect as you think it is, why isn't everyone jumping 
on board? I know, you have answers for that one too!
Sep 15 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 15 September 2018 at 23:50:43 UTC, Josphe Brigmo 
wrote:

 [...]


D is generally described as a system programming language. There 
is value in favoring a simple and obvious implementation ("do 
what I say") over going out of one's way to make usage simpler 
("do what I mean"). The tradeoff is performance and complexity. 
Performance is generally an important factor for users of system 
programming languages, and complexity is a source of unforeseen 
problems in non-trivial use cases.

Consider, for example, how integers are treated in D and Python. 
D's integers are fixed-length and roll over on overflow. Python 
integers are bigints, which makes them slower, but can handle 
numbers of any size.

 From your posts, it sounds like you're looking for a programming 
language closer to Python than to D.
Sep 15 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/15/2018 08:09 PM, Vladimir Panteleev wrote:

 On Saturday, 15 September 2018 at 23:50:43 UTC, Josphe Brigmo wrote:

 [...]


 
 D is generally described as a system programming language. There is 
 value in favoring a simple and obvious implementation ("do what I say") 
 over going out of one's way to make usage simpler ("do what I mean"). 
 The tradeoff is performance and complexity. Performance is generally an 
 important factor for users of system programming languages, and 
 complexity is a source of unforeseen problems in non-trivial use cases.
 
 Consider, for example, how integers are treated in D and Python. D's 
 integers are fixed-length and roll over on overflow. Python integers are 
 bigints, which makes them slower, but can handle numbers of any size.
 
  From your posts, it sounds like you're looking for a programming 
 language closer to Python than to D.
 


D's philosophy is (or at least is supposed to be) "Whenever possible, 
the right thing should be default, alternatives should still be 
possible." (And if I'm mistaken on that, then I've very much chosen the 
wrong language to use.)

Note that this does not contradict D's integer behaviour, because 
changing that would be *prohibitively* expensive and cause the need for 
workarounds to be common yet very difficult (if even possible). 
Therefore, it is a justifiable exception to the rule.

By contrast, I find it difficult to believe that a fix to the OP's 
problem would be prohibitively expensive. Of course, maybe I'm wrong. 
And maybe, as some have vaguely suggested, the WinAPI makes a correct 
fix impossible. **BUT**, unfortunately, I (as well as the OP) have no 
way to know because there appears to be a distressingly blatant LACK of 
technical discussion on the specific OP problem itself. Instead, all I'm 
seeing is a bunch of high-level design-philosophy bickering left 
completely divorced from the specifics of the original problem in question.

Yes, the OP needs to file a bug report (and if he's already done so, 
then please post a link here for our reference). But sheesh, people, 
it's *no wonder* he's gotten upset about it. I would too, and so would 
most of the rest of you (I've seen it happen).
Sep 18 2018
parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Tuesday, 18 September 2018 at 19:57:09 UTC, Nick Sabalausky 
(Abscissa) wrote:

 Yes, the OP needs to file a bug report (and if he's already 
 done so, then please post a link here for our reference).


It’s an old issue, and the OP posted the link a bit further up in 
this thread: 
https://forum.dlang.org/post/gyaswfyzwoofaozkizcu forum.dlang.org
Sep 18 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/18/2018 06:14 PM, Bastiaan Veelo wrote:

 On Tuesday, 18 September 2018 at 19:57:09 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 Yes, the OP needs to file a bug report (and if he's already done so, 
 then please post a link here for our reference).


 
 It’s an old issue, and the OP posted the link a bit further up in this 
 thread: https://forum.dlang.org/post/gyaswfyzwoofaozkizcu forum.dlang.org
 


Ah, don't know how I managed to miss that. Thanks.
Sep 18 2018
prev sibling parent reply Ecstatic Coder <ecstatic.coder gmail.com> writes:
On Saturday, 15 September 2018 at 23:06:57 UTC, Jonathan M Davis 
wrote:

 On Saturday, September 15, 2018 6:54:50 AM MDT Josphe Brigmo 
 via Digitalmars-d wrote:

 On Saturday, 15 September 2018 at 12:38:41 UTC, Adam D. Ruppe

 wrote:

 On Saturday, 15 September 2018 at 10:57:56 UTC, Josphe Brigmo

 wrote:

 Phobos *NEEDS* to be modified to work with these newer OS's.


 You need to look at the source code before posting. The code 
 for remove is literally

 DeleteFileW(name);

 it is a one-line wrapper, and obviously uses the unicode 
 version.

 https://github.com/dlang/phobos/blob/master/std/file.d#L1047


 It doesn't matter, the fact is that something in phobos is 
 broke. Do you really expect me to do all the work? The fact 
 that using executeShell or "\\?\" solves 90% of the 
 problems(maybe all of them) proves that phobos is not up to 
 par.


 Using std.file should be on par with using the Windows API from 
 C or C++. It doesn't try to fix the arguably broken behavior of 
 the Windows API with regards to long paths but requires that 
 the programmer deal with them just like they would in C/C++. 
 The main differences are that the std.file functions in 
 question use D strings rather than C strings, and they 
 translate them to the UTF-16 C strings for you rather than 
 requiring you to do it. But they don't do anything like add 
 "\\?\" for you any more than the Windows API itself does that.

 If you consider that to be broken, then sorry. For better or 
 worse, it was decided that it was better to let the programmer 
 deal with those intricacies rather than trying to tweak the 
 input to make it work based on the idea that that could have 
 undesirable consequences in some circumstances. On some level, 
 that does suck, but the Windows API does not make it easy to 
 make this work like it would on a *nix system without 
 introducing subtle bugs.

 If you find that the std.file functions don't work whereas 
 using the same input to the Windows API functions in C/C++ 
 would have, then there's a bug in the D code, and it needs to 
 be fixed, but if it acts the same as the C/C++ code, then it's 
 working as intended.

 - Jonathan M Davis


This attitude is unfortunately the cause of a lot frustration 
among cross-platform developers like me.

I chose D for my file scripting needs because it's a 
cross-platform language.

I expect that calling the function F on system X will work the 
same as calling that same function on system Y.

That's the contract in cross-platform programming.

Unfortunately D fails at being consistent.

I recently learned this lesson with my Resync tool.

No everybody wants the cross-platform to behave inconsistently.

For example, in the past I've implemented a proprietary 
cross-platform C++ game engine for Windows, PS2, Xbox and 
GameCube.

The games needed some tuning for the graphics, etc.

But code-wise, the engine made the games behave consistently 
across the different platforms.

This was all about making each method of each class behaving the 
same. As simple as that.

Indeed, on some platforms, the game engine also provided extra 
classes and/or methods to add some functionalities specific to 
these platforms.

But the common trunc was implemented (!) to behave the same. That 
was what our game developers expected...
Sep 17 2018
next sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 18 September 2018 at 06:16:50 UTC, Ecstatic Coder 
wrote:

 I expect that calling the function F on system X will work the 
 same as calling that same function on system Y.

 That's the contract in cross-platform programming.


Heh, I remember working around a filesystem that doesn't support 
unicode, I used bullet • as utf8 prefix and heuristically 
detected that names starting with that prefix are utf8 encoded 
and decoded from that.
Sep 18 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Tuesday, 18 September 2018 at 06:16:50 UTC, Ecstatic Coder 
wrote:

 This attitude is unfortunately the cause of a lot frustration 
 among cross-platform developers like me.

 I chose D for my file scripting needs because it's a 
 cross-platform language.

 I expect that calling the function F on system X will work the 
 same as calling that same function on system Y.


You ask for the impossible.

How do you expect the following to "work the same" across all 
platforms:

	import std.stdio;
	import std.file;
	auto fn = "a.txt";
	auto f = File(fn, "w");
	remove(fn);

or this:

	import std.file;
	write("a", "a");
	write("A", "A");
	assert(readText("a") == "a");
	assert(readText("A") == "A");

There will always be inherent differences between platforms, 
because they are wildly different. Using this sentiment as an 
argument for Phobos to smooth over this one particular difference 
that you care about, and for which incidentally an apparent 
solution appears to be available, is fallacious.
Sep 18 2018
next sibling parent reply Ecstatic Coder <ecstatic.coder gmail.com> writes:
 There will always be inherent differences between platforms, 
 because they are wildly different.


Right.

Technically the PS2 console, the GameCube and the Xbox console 
were very different from each other, so I had no choice but to 
implement low-level abstraction function (GetPhysicalPath() etc) 
to make the file system classes work similarly across all four 
systems.

That wasn't an easy task, but it made the life so much easier for 
the game programmers that it was obvious this was "the right 
thing" to do.

The fact that D's standard library has already bitten me several 
time with its platform specific problem clearly shows that you 
have chosen another path.

That's your right, but don't expect those who develop 
cross-platform tools in D to be happy to HAVE to put ugly 
"version ( ... )" stuff in their code when their software 
suddenly break on some platforms for unknown (= undocumented) 
reasons...
Sep 18 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Tuesday, 18 September 2018 at 18:04:19 UTC, Ecstatic Coder 
wrote:

 There will always be inherent differences between platforms, 
 because they are wildly different.


 Right.

 Technically the PS2 console, the GameCube and the Xbox console 
 were very different from each other, so I had no choice but to 
 implement low-level abstraction function (GetPhysicalPath() 
 etc) to make the file system classes work similarly across all 
 four systems.


Do the PS2, GameCube and Xbox filesystems all have identical file 
path limits?

And, did any of the paths in your game exceed 260 characters in 
length?

These comparisons are not helpful.
Sep 18 2018
parent reply Ecstatic Coder <ecstatic.coder gmail.com> writes:
 Do the PS2, GameCube and Xbox filesystems all have identical 
 file path limits?


Guess ;)


 And, did any of the paths in your game exceed 260 characters in 
 length?


No. But the suggested GetPhysicalPath() solution would also work 
equally well in this case.


 These comparisons are not helpful.


None would ever be, considering you obviously have decided to 
ignore such a simple solution to the 260 character limit...
Sep 18 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 05:24:24 UTC, Ecstatic Coder 
wrote:

 None would ever be, considering you obviously have decided to 
 ignore such a simple solution to the 260 character limit...


Add "ad hominem" to your pile of fallacies, I guess. I've 
addressed it twice in this thread already - it is problematic for 
technical reasons. It seems you are the one ignoring the problems 
with it...
Sep 18 2018
parent reply Ecstatic Coder <ecstatic.coder gmail.com> writes:
On Wednesday, 19 September 2018 at 05:32:47 UTC, Vladimir 
Panteleev wrote:

 On Wednesday, 19 September 2018 at 05:24:24 UTC, Ecstatic Coder 
 wrote:

 None would ever be, considering you obviously have decided to 
 ignore such a simple solution to the 260 character limit...


 Add "ad hominem" to your pile of fallacies, I guess.


Now I will, thanks :)

Once again, this forum proves to be very effective at removing 
any motivation from D users to get involved and contribute to the 
D language.

That's probably one of the keys of its success...
Sep 19 2018
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 08:13:31 UTC, Ecstatic Coder 
wrote:

 On Wednesday, 19 September 2018 at 05:32:47 UTC, Vladimir 
 Panteleev wrote:

 On Wednesday, 19 September 2018 at 05:24:24 UTC, Ecstatic 
 Coder wrote:

 None would ever be, considering you obviously have decided to 
 ignore such a simple solution to the 260 character limit...


 Add "ad hominem" to your pile of fallacies, I guess.


 Now I will, thanks :)

 Once again, this forum proves to be very effective at removing 
 any motivation from D users to get involved and contribute to 
 the D language.


Sorry, I didn't intend to make you feel unwelcome.

But, you did say I was ignoring something, after I've addressed 
it twice in the same thread. That wasn't nice of you either.

In any case, the solution in question won't work:
https://forum.dlang.org/post/riuqfngyzfumehsmontk forum.dlang.org
Sep 19 2018
prev sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/18/2018 05:25 AM, Vladimir Panteleev wrote:

 On Tuesday, 18 September 2018 at 06:16:50 UTC, Ecstatic Coder wrote:

 I expect that calling the function F on system X will work the same as 
 calling that same function on system Y.


 
 You ask for the impossible.


I think it's safe to assume a "...to the extent reasonably possible." 
suffixed to his statement.
Sep 18 2018
prev sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/15/2018 06:57 AM, Josphe Brigmo wrote:

 
 You are missing the point, MAX_PATH is more than just phobos. It's built 
 in to the windows design. Windows enforces it.
 
 All ansi api calls are limited by MAX_PATH.
 
 The way to fix it is to use the wide api calls which are not limited or 
 to use other tricks.
 
 Phobos *NEEDS* to be modified to work with these newer OS's.
 
 You wouldn't like it if phobos limit something that it didn't need that 
 you would use, would you?
 
 File operations are so common that this stuff is relevant to all that 
 use windows(and some that don't).
 


As has already been said, *file a bug report*.

When told this before, instead of submitting a proper bug report, you 
simply moved your goalposts and began complaining about very vague, 
general things instead.

You appear to be less interested in getting this, or any other specific 
issue fixed than you are in simply ranting, griping and berating.
Sep 17 2018
prev sibling next sibling parent reply tide <tide tide.tide> writes:
On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo 
 wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)


 Please file a bug report with reproducible examples if you 
 believe it's a bug.


I feel people need to stop saying this. It feels like people are 
just being told to say this if there is a bug. There is a larger 
issue, Bugzilla doesn't and isn't working. Someone will probably 
throw up some stats about how many bugs are filed and how many 
are resolved. Those exist because someone working on Dlang comes 
across a bug that affects them, creates a patch for it first, 
then goes and creates a bugzilla entry and marks it resolved. 
Issues are rarely resolved by anyone other than the person that 
created the bug report to begin with. Or issues created by a team 
member is resolved by another team member.
Sep 15 2018
next sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 15 September 2018 at 13:54:45 UTC, tide wrote:

 I feel people need to stop saying this. It feels like people 
 are just being told to say this if there is a bug. There is a 
 larger issue, Bugzilla doesn't and isn't working. Someone will 
 probably throw up some stats about how many bugs are filed and 
 how many are resolved. Those exist because someone working on 
 Dlang comes across a bug that affects them, creates a patch for 
 it first, then goes and creates a bugzilla entry and marks it 
 resolved. Issues are rarely resolved by anyone other than the 
 person that created the bug report to begin with. Or issues 
 created by a team member is resolved by another team member.


- A reproducible test case removes any room for miscommunication, 
ensures that a fix will fix the problem the submitter is having, 
and saves time for the developer working on fixing the bug.

- Not filing a bug will essentially guarantee that it's not fixed.

- Sometimes, people do look at random bugs and fix them. 
Regressions especially are tracked closely.

- Having an issue provides a central place to discuss the bug and 
link to it.

I have some experience working with and curating D's Bugzilla 
(see e.g. https://github.com/CyberShadow/DBugTests), so I think 
the above are authoritatively true.
Sep 15 2018
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Saturday, 15 September 2018 at 13:54:45 UTC, tide wrote:

 On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo 
 wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)


 Please file a bug report with reproducible examples if you 
 believe it's a bug.


 I feel people need to stop saying this.


That's how things are done in this project (and most projects). 
Anyone not liking that is out of luck. I don't use any open 
source projects that use vague posts to a forum to report bugs.
Sep 15 2018
parent reply tide <tide tide.tide> writes:
On Saturday, 15 September 2018 at 18:33:52 UTC, bachmeier wrote:

 On Saturday, 15 September 2018 at 13:54:45 UTC, tide wrote:

 On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo 
 wrote:

 For very long file names it is broke and every command 
 fails. These paths are not all that long but over 256 limit. 
 (For windows)


 Please file a bug report with reproducible examples if you 
 believe it's a bug.


 I feel people need to stop saying this.


 That's how things are done in this project (and most projects). 
 Anyone not liking that is out of luck. I don't use any open 
 source projects that use vague posts to a forum to report bugs.


That's how they are, but if you are going to say silly things. At 
least take the initiative on yourself to go see if the bug 
already exists. Odds are it has probably already been reported, 
someone making a forums post about it is just able the next step 
after the bug report has been sitting in the queue for 4+ years.

The problem isn't reporting the bug, it's that for D, no one is 
managing bugzilla. It seems to be the conclusion was reached that 
this is not a bug and won't be fixed. That could have been done a 
long time ago and closed the bug. Or at least I think Jonathan is 
part of the Dlang organization. Not sure, hard to tell who is and 
isn't on these boards.
Sep 15 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, September 15, 2018 5:47:08 PM MDT tide via Digitalmars-d wrote:

 On Saturday, 15 September 2018 at 18:33:52 UTC, bachmeier wrote:

 On Saturday, 15 September 2018 at 13:54:45 UTC, tide wrote:

 On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo

 wrote:

 For very long file names it is broke and every command
 fails. These paths are not all that long but over 256 limit.
 (For windows)


 Please file a bug report with reproducible examples if you
 believe it's a bug.


 I feel people need to stop saying this.


 That's how things are done in this project (and most projects).
 Anyone not liking that is out of luck. I don't use any open
 source projects that use vague posts to a forum to report bugs.


 That's how they are, but if you are going to say silly things. At
 least take the initiative on yourself to go see if the bug
 already exists. Odds are it has probably already been reported,
 someone making a forums post about it is just able the next step
 after the bug report has been sitting in the queue for 4+ years.

 The problem isn't reporting the bug, it's that for D, no one is
 managing bugzilla. It seems to be the conclusion was reached that
 this is not a bug and won't be fixed. That could have been done a
 long time ago and closed the bug. Or at least I think Jonathan is
 part of the Dlang organization. Not sure, hard to tell who is and
 isn't on these boards.


The issue was reported in bugzilla quite some time ago.

https://issues.dlang.org/show_bug.cgi?id=8967

However, while Walter's response on it basically indicates that we should
just close it as "won't fix," we never actually did - which is something
that probably happens too often. There was further discussion in there that
never went anywhere. What we should probably do is ensure that the std.file
documentation is clear about the issue and count that as the fix. Either
way, I think that it's pretty clear that the documentation needs to be
improved.

As for figuring out who is "officially" part of the dlang org (or at least
has the rights to merge PRs from at least one dlang repo), you can look here

https://github.com/orgs/dlang/people

though it's possible to hide your membership, so while I see 59 members when
I look at it while signed in, if I look at it while not signed in, I see
only 40.

- Jonathan M Davis
Sep 15 2018
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Sunday, 16 September 2018 at 00:14:12 UTC, Jonathan M Davis 
wrote:

 As for figuring out who is "officially" part of the dlang org 
 (or at least has the rights to merge PRs from at least one 
 dlang repo), you can look here

 https://github.com/orgs/dlang/people

 though it's possible to hide your membership, so while I see 59 
 members when I look at it while signed in, if I look at it 
 while not signed in, I see only 40.


I think it's worth clarifying: Only Walter and Andrei have the 
final say on things. Everyone else is a contributor (with a small 
few financially rewarded for their work). So, the above list of 
people isn't a good metric for much other than knowing who can 
merge your PR.
Sep 15 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, September 15, 2018 6:28:20 PM MDT Vladimir Panteleev via 
Digitalmars-d wrote:

 On Sunday, 16 September 2018 at 00:14:12 UTC, Jonathan M Davis

 wrote:

 As for figuring out who is "officially" part of the dlang org
 (or at least has the rights to merge PRs from at least one
 dlang repo), you can look here

 https://github.com/orgs/dlang/people

 though it's possible to hide your membership, so while I see 59
 members when I look at it while signed in, if I look at it
 while not signed in, I see only 40.


 I think it's worth clarifying: Only Walter and Andrei have the
 final say on things. Everyone else is a contributor (with a small
 few financially rewarded for their work). So, the above list of
 people isn't a good metric for much other than knowing who can
 merge your PR.


Yeah, the list is mostly of folks who have contributed significantly enough
to have been given merge permissions at some point. Some of us may have some
influence based on how long we've been with the project and the level of
knowledge and expertise that we've shown in the process, but as far as
deciding the direction of the project or anything like that, it's all Walter
and Andrei. The primary influence that any of us have is simply by the work
we contribute via PRs and the feedback we give when reviewing PRs. It's not
like there's a hierarchy of authority or anything like that. For the most
part, the only authority that Walter and Andrei have given others is the
authority to merge PRs.

- Jonathan M Davis
Sep 15 2018
parent reply tide <tide tide.tide> writes:
On Sunday, 16 September 2018 at 00:53:45 UTC, Jonathan M Davis 
wrote:

 On Saturday, September 15, 2018 6:28:20 PM MDT Vladimir 
 Panteleev via Digitalmars-d wrote:

 On Sunday, 16 September 2018 at 00:14:12 UTC, Jonathan M Davis

 wrote:

 As for figuring out who is "officially" part of the dlang 
 org (or at least has the rights to merge PRs from at least 
 one dlang repo), you can look here

 https://github.com/orgs/dlang/people

 though it's possible to hide your membership, so while I see 
 59 members when I look at it while signed in, if I look at 
 it while not signed in, I see only 40.


 I think it's worth clarifying: Only Walter and Andrei have the 
 final say on things. Everyone else is a contributor (with a 
 small few financially rewarded for their work). So, the above 
 list of people isn't a good metric for much other than knowing 
 who can merge your PR.


 Yeah, the list is mostly of folks who have contributed 
 significantly enough to have been given merge permissions at 
 some point. Some of us may have some influence based on how 
 long we've been with the project and the level of knowledge and 
 expertise that we've shown in the process, but as far as 
 deciding the direction of the project or anything like that, 
 it's all Walter and Andrei. The primary influence that any of 
 us have is simply by the work we contribute via PRs and the 
 feedback we give when reviewing PRs. It's not like there's a 
 hierarchy of authority or anything like that. For the most 
 part, the only authority that Walter and Andrei have given 
 others is the authority to merge PRs.

 - Jonathan M Davis


I guess that's why Bugzilla is a complete disaster. No one, at 
all, is maintaining it. As there are only 2 people that can 
really maintain it, and I don't see either of them commenting on 
bugs to provide direction, at least very often.
Sep 15 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, September 15, 2018 7:19:46 PM MDT tide via Digitalmars-d wrote:

 On Sunday, 16 September 2018 at 00:53:45 UTC, Jonathan M Davis

 wrote:

 On Saturday, September 15, 2018 6:28:20 PM MDT Vladimir

 Panteleev via Digitalmars-d wrote:

 On Sunday, 16 September 2018 at 00:14:12 UTC, Jonathan M Davis

 wrote:

 As for figuring out who is "officially" part of the dlang
 org (or at least has the rights to merge PRs from at least
 one dlang repo), you can look here

 https://github.com/orgs/dlang/people

 though it's possible to hide your membership, so while I see
 59 members when I look at it while signed in, if I look at
 it while not signed in, I see only 40.


 I think it's worth clarifying: Only Walter and Andrei have the
 final say on things. Everyone else is a contributor (with a
 small few financially rewarded for their work). So, the above
 list of people isn't a good metric for much other than knowing
 who can merge your PR.


 Yeah, the list is mostly of folks who have contributed
 significantly enough to have been given merge permissions at
 some point. Some of us may have some influence based on how
 long we've been with the project and the level of knowledge and
 expertise that we've shown in the process, but as far as
 deciding the direction of the project or anything like that,
 it's all Walter and Andrei. The primary influence that any of
 us have is simply by the work we contribute via PRs and the
 feedback we give when reviewing PRs. It's not like there's a
 hierarchy of authority or anything like that. For the most
 part, the only authority that Walter and Andrei have given
 others is the authority to merge PRs.


 I guess that's why Bugzilla is a complete disaster. No one, at
 all, is maintaining it. As there are only 2 people that can
 really maintain it, and I don't see either of them commenting on
 bugs to provide direction, at least very often.


Pretty much everything done around here is done by volunteers. There are
contributors who sometimes go through bugzilla to update or close bugs, and
there are contributors who go looking through bugzilla for bugs to fix, but
there is no one whose job it is to manage the list of bugs. I don't know of
any open source project that works that way. Usually, at most, you end up
with bugs being automatically assigned to people based on what they're for.

- Jonathan M Davis
Sep 15 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Sunday, 16 September 2018 at 01:19:46 UTC, tide wrote:

 I guess that's why Bugzilla is a complete disaster. No one, at 
 all, is maintaining it. As there are only 2 people that can 
 really maintain it, and I don't see either of them commenting 
 on bugs to provide direction, at least very often.


Well, I think that's looking at the situation from the wrong 
angle.

Most of D's code was written by volunteer contributors, and 
usually the code's author ends up maintaining that code, at least 
for a while. So, when you find a bug in some part of D and can't 
fix it yourself, looking at who wrote or last maintained the 
relevant code and pinging them would be the first step.

There are some things we can improve, like upgrading the platform 
or improving the categorization so people can receive 
notifications when someone files a bug in a Phobos module they 
care about. I've been slowly working on that front 
(https://github.com/CyberShadow/bugzilla-meta), but it doesn't 
change the underlying facts that bugs are most likely to be fixed 
by people who work on the code, not Andrei or Walter or any one 
person "in charge" of Bugzilla.
Sep 15 2018
parent reply tide <tide tide.tide> writes:
On Sunday, 16 September 2018 at 01:33:52 UTC, Vladimir Panteleev 
wrote:

 On Sunday, 16 September 2018 at 01:19:46 UTC, tide wrote:

 I guess that's why Bugzilla is a complete disaster. No one, at 
 all, is maintaining it. As there are only 2 people that can 
 really maintain it, and I don't see either of them commenting 
 on bugs to provide direction, at least very often.


 Well, I think that's looking at the situation from the wrong 
 angle.

 Most of D's code was written by volunteer contributors, and 
 usually the code's author ends up maintaining that code, at 
 least for a while. So, when you find a bug in some part of D 
 and can't fix it yourself, looking at who wrote or last 
 maintained the relevant code and pinging them would be the 
 first step.

 There are some things we can improve, like upgrading the 
 platform or improving the categorization so people can receive 
 notifications when someone files a bug in a Phobos module they 
 care about. I've been slowly working on that front 
 (https://github.com/CyberShadow/bugzilla-meta), but it doesn't 
 change the underlying facts that bugs are most likely to be 
 fixed by people who work on the code, not Andrei or Walter or 
 any one person "in charge" of Bugzilla.


There are a lot of issues that aren't simple bugs that just 
anyone can fix. They are issues that are locked behind 
management. One's that are 4 years old for example, they are 
probably some bug locked behind management. That's why they get 
so old. From the comments it is not clear that a pull request 
wouldn't be accepted to fix the issue. Personally I think phobos 
should not exception for long file names.

Walters concern is that the path will change unexpected for the 
user. Where does that matter for something like rmDir ? The user 
passes a long path, and rmDir swallows it, never to be seen again 
by the user. What does it matter if the path gets corrected if it 
is too long?

As for any stored path, they can remain the same, as in DirEntry. 
The length of the path is what determines if it needs to use the 
special syntax or not. The user won't see any difference at all. 

version, you might be able to see how they implemented it there. 
Parts of it are open source iirc.

Anyways there's only so many issues one person can chase to hell 
for someone as stubborn as ./.
Sep 15 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Sunday, 16 September 2018 at 02:58:30 UTC, tide wrote:

 There are a lot of issues that aren't simple bugs that just 
 anyone can fix. They are issues that are locked behind 
 management. One's that are 4 years old for example, they are 
 probably some bug locked behind management. That's why they get 
 so old. From the comments it is not clear that a pull request 
 wouldn't be accepted to fix the issue. Personally I think 
 phobos should not exception for long file names.


Nothing is "locked behind management". If you feel that some 
issue important to you is stalled, you can create a forum thread, 
or email Walter/Andrei to ask for a resolution.


 Walters concern is that the path will change unexpected for the 
 user. Where does that matter for something like rmDir ? The 
 user passes a long path, and rmDir swallows it, never to be 
 seen again by the user. What does it matter if the path gets 
 corrected if it is too long?


It's more than that.

The path needs to be normalized, which means that \.\ and \..\ 
fragments need to be removed away first. Depending on your 
interpretation of symlinks/junctions/etc., "foo/bar/../" might 
mean something else than "foo/" if "bar" is a reparse point.

The path also needs to be absolute, so it has to be expanded to a 
full path before it can be prefixed with the UNC prefix. Given 
how the current directory is state tied to the process (not 
thread), you get bonus race condition issues. There's also issues 
like the current directory object being renamed/moved; then a 
relative path will no longer correspond to what the program 
thinks the absolute paths is.

All things considered, this goes well into the territory of "not 
D's problem". My personal recommendation: if you care about long 
path names, use an operating system which implements them right.
Sep 15 2018
next sibling parent reply tide <tide tide.tide> writes:
On Sunday, 16 September 2018 at 03:19:12 UTC, Vladimir Panteleev 
wrote:

 On Sunday, 16 September 2018 at 02:58:30 UTC, tide wrote:

 There are a lot of issues that aren't simple bugs that just 
 anyone can fix. They are issues that are locked behind 
 management. One's that are 4 years old for example, they are 
 probably some bug locked behind management. That's why they 
 get so old. From the comments it is not clear that a pull 
 request wouldn't be accepted to fix the issue. Personally I 
 think phobos should not exception for long file names.


 Nothing is "locked behind management". If you feel that some 
 issue important to you is stalled, you can create a forum 
 thread, or email Walter/Andrei to ask for a resolution.


Funny the other guy was saying to create a bugzilla issue.


 Walters concern is that the path will change unexpected for 
 the user. Where does that matter for something like rmDir ? 
 The user passes a long path, and rmDir swallows it, never to 
 be seen again by the user. What does it matter if the path 
 gets corrected if it is too long?


 It's more than that.

 The path needs to be normalized, which means that \.\ and \..\ 
 fragments need to be removed away first. Depending on your 
 interpretation of symlinks/junctions/etc., "foo/bar/../" might 
 mean something else than "foo/" if "bar" is a reparse point.


All these issues yet for some reason this function was included 
in the lot: https://dlang.org/phobos/std_path.html#absolutePath


 The path also needs to be absolute, so it has to be expanded to 
 a full path before it can be prefixed with the UNC prefix. 
 Given how the current directory is state tied to the process 
 (not thread), you get bonus race condition issues. There's also 
 issues like the current directory object being renamed/moved; 
 then a relative path will no longer correspond to what the 
 program thinks the absolute paths is.


This issue exists anyways, you'd only expand the path when it 
need to be used. If the file changes within milliseconds, I don't 
see that happening often and if it does there's a flaw in your 
design that'd happen even if the path didn't have to be 
constructed first.


 All things considered, this goes well into the territory of 
 "not D's problem". My personal recommendation: if you care 
 about long path names, use an operating system which implements 
 them right.


So you pass a valid path (selected by a user through a UI) to 
rmDir, and it doesn't remove the directory. You think this is 
acceptable behavior?
Sep 16 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Sunday, 16 September 2018 at 16:17:21 UTC, tide wrote:

 Nothing is "locked behind management". If you feel that some 
 issue important to you is stalled, you can create a forum 
 thread, or email Walter/Andrei to ask for a resolution.


 Funny the other guy was saying to create a bugzilla issue.


Do that *first*.


 The path needs to be normalized, which means that \.\ and \..\ 
 fragments need to be removed away first. Depending on your 
 interpretation of symlinks/junctions/etc., "foo/bar/../" might 
 mean something else than "foo/" if "bar" is a reparse point.


 All these issues yet for some reason this function was included 
 in the lot: https://dlang.org/phobos/std_path.html#absolutePath
 [...]
 This issue exists anyways, you'd only expand the path when it 
 need to be used. If the file changes within milliseconds, I 
 don't see that happening often and if it does there's a flaw in 
 your design that'd happen even if the path didn't have to be 
 constructed first.


You've missed the point. Complexity breeds bugs and unexpected 
behavior. The expectation is that D's function to delete a file 
should do little else than call the OS function.

If *YOU* are OK with the consequences of complexity, implement 
this in YOUR code, but do not enforce it upon others.


 So you pass a valid path (selected by a user through a UI) to 
 rmDir, and it doesn't remove the directory. You think this is 
 acceptable behavior?


It is absolutely not acceptable behavior. Complain to Microsoft. 
The OS should not allow users to create or select paths that 
programs cannot operate on without jumping through crazy hoops.
Sep 16 2018
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
To elaborate:

On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir Panteleev 
wrote:

 If *YOU* are OK with the consequences of complexity, implement 
 this in YOUR code, but do not enforce it upon others.


This is much better done in user code anyway, because you only 
need to expand / normalize the path and prepend the prefix only 
once (root of the directory tree you're operating on), instead of 
once per directory call.

We could add a `string longPath(string)` function in Phobos 
(no-op on POSIX, expands and prepends prefix on Windows). I 
believe I suggested the same in the bug report years ago when we 
discussed it.


 It is absolutely not acceptable behavior. Complain to 
 Microsoft. The OS should not allow users to create or select 
 paths that programs cannot operate on without jumping through 
 crazy hoops.


Microsoft could have solved this easily enough:

extern(System) void AllowLongPaths();

Programs (or programming language runtimes) which can handle 
paths longer than MAX_PATH could call that function. It can also 
be used as a hint to the OS that file/directory selection 
dialogs, as you mentioned, are allowed to select paths longer 
than MAX_PATH.
Sep 16 2018
parent reply Temtaime <temtaime gmail.com> writes:
On Sunday, 16 September 2018 at 22:49:26 UTC, Vladimir Panteleev 
wrote:

 To elaborate:

 On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir 
 Panteleev wrote:

 If *YOU* are OK with the consequences of complexity, implement 
 this in YOUR code, but do not enforce it upon others.


 This is much better done in user code anyway, because you only 
 need to expand / normalize the path and prepend the prefix only 
 once (root of the directory tree you're operating on), instead 
 of once per directory call.

 We could add a `string longPath(string)` function in Phobos 
 (no-op on POSIX, expands and prepends prefix on Windows). I 
 believe I suggested the same in the bug report years ago when 
 we discussed it.


 It is absolutely not acceptable behavior. Complain to 
 Microsoft. The OS should not allow users to create or select 
 paths that programs cannot operate on without jumping through 
 crazy hoops.


 Microsoft could have solved this easily enough:

 extern(System) void AllowLongPaths();

 Programs (or programming language runtimes) which can handle 
 paths longer than MAX_PATH could call that function. It can 
 also be used as a hint to the OS that file/directory selection 
 dialogs, as you mentioned, are allowed to select paths longer 
 than MAX_PATH.


It's problem with phobos.
It should be able handle all the paths whatever length they have, 
on all the platforms without noising the user.

Even with performance penalty, but it should.
Sep 17 2018
parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 17 September 2018 at 12:37:13 UTC, Temtaime wrote:

 On Sunday, 16 September 2018 at 22:49:26 UTC, Vladimir 
 Panteleev wrote:

 To elaborate:

 On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir 
 Panteleev wrote:

 If *YOU* are OK with the consequences of complexity, 
 implement this in YOUR code, but do not enforce it upon 
 others.


 This is much better done in user code anyway, because you only 
 need to expand / normalize the path and prepend the prefix 
 only once (root of the directory tree you're operating on), 
 instead of once per directory call.

 We could add a `string longPath(string)` function in Phobos 
 (no-op on POSIX, expands and prepends prefix on Windows). I 
 believe I suggested the same in the bug report years ago when 
 we discussed it.


 It is absolutely not acceptable behavior. Complain to 
 Microsoft. The OS should not allow users to create or select 
 paths that programs cannot operate on without jumping through 
 crazy hoops.


 Microsoft could have solved this easily enough:

 extern(System) void AllowLongPaths();

 Programs (or programming language runtimes) which can handle 
 paths longer than MAX_PATH could call that function. It can 
 also be used as a hint to the OS that file/directory selection 
 dialogs, as you mentioned, are allowed to select paths longer 
 than MAX_PATH.


 It's problem with phobos.
 It should be able handle all the paths whatever length they 
 have, on all the platforms without noising the user.

 Even with performance penalty, but it should.


No, that's completely nuts!
A library, especially a standard library, should not introduce 
new limitations, but pampering over the limitations of the 
platform is not the right thing to do. If the platforms API is 
piling POS, there's nothing a sane library can do about.
If your app writes to a FAT12 formatted floppy disk you don't 
expect the library to implement code to alleviate its limitation, 
like 8+3 filenames or fixed number of files in the root directory.
Sep 17 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/17/2018 11:27 AM, Patrick Schluter wrote:

 On Monday, 17 September 2018 at 12:37:13 UTC, Temtaime wrote:

 It's problem with phobos.
 It should be able handle all the paths whatever length they have, on 
 all the platforms without noising the user.

 Even with performance penalty, but it should.




This actually leads to an interesting point. Let's change gears for a 
moment to "API Design Theory"...

Suppose you implement API function X which takes Y as an argument. 
Question: Should X accept values (or types) of Y for which X fails? 
Answer: Ideally, no. At least to the extent realistically possible.

So what should a well-designed function X do when faced with known-to-be 
unsupported input Y?[1] Here are the possibilities[2]:

([1] "Unsupported" is defined here as "for the given input, the function 
does not correctly and successfully perform its intended goal".)

([2] Again, we're assuming "to the extent realistically possible" is 
still in play here.)

1. Add support for input Y to function X.
2. Reject input Y (via either: abort, throw or static compile error)
3. Accept input Y and allow one of the following to occur[3]: Silently 
fail to perform the expected task, silently do the wrong thing, or 
trigger an unclear error (assert/exception/compile) from deeper in the 
callstack. ([3] Note that which one of these three possibilities occurs 
is dependent on function X's exact implementation details.)

Of these three possibilities for a function X faced with unsupported 
input Y, the first two options are (in theory) acceptable[4]. The third 
possibility is absolutely not acceptable [to the extent realistically 
possible.]

([4] The library containing function X may impose additional 
restrictions on what is/isn't acceptable.)

So, what does this mean in our specific situation?

If good API design is followed, then any filesystem-based API, when 
running on Windows and faced with a path exceeding the local system's 
size limit, must do one of the following:

1. Modify Phobos to support long filepaths even on Windows versions 
below Win10 v1607.
2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 bytes[5].

sufficiently unrealistic/problematic.


since they do not involve passing unsupported input to the given 
function, but they could still be optionally determined unacceptable if 
desired: A. Rejecting all paths longer than MAX_PATH-12 bytes, even if 
\\?\-based. B. Rejecting too-long paths even on Win10 v1607+ with 
LongPathsEnabled set in the registry.)



 No, that's completely nuts!
 A library, especially a standard library, should not introduce new 
 limitations, but pampering over the limitations of the platform is not 
 the right thing to do.


This is debatable. Why, exactly, is pampering over pre-Win10-v1607's 
maximum non-\\?\ filepath length a bad thing? Exactly what problems does 
it cause?


 If the platforms API is piling POS, there's 
 nothing a sane library can do about.


That is patently untrue. It might be true in specific circumstances, but 
it is not generally true. If you believe it to be true in this specific 
case, then please explain *how*/*why* there is nothing the library can 
do about it.


 If your app writes to a FAT12 formatted floppy disk you don't expect the 
 library to implement code to alleviate its limitation, like 8+3 
 filenames or fixed number of files in the root directory.


The general rule of thumb is: "Typical situations should work as 
expected, atypical situations should be possible." Therefore, please 
explain *ANY* one of the following:

1. How writing to a FAT12 formatted floppy disk qualifies as a typical 
situation.

2. How abstracting over the MAX_PATH limitation makes writing to a FAT12 
formatted floppy impossible.

3. How my claim of "Typical situations should work as expected, atypical 
situations should be possible" is fundamentally wrong for D.
Sep 18 2018
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
(Abscissa) wrote:

 This actually leads to an interesting point. Let's change gears 
 for a moment to "API Design Theory"...

 Suppose you implement API function X which takes Y as an 
 argument. Question: Should X accept values (or types) of Y for 
 which X fails? Answer: Ideally, no. At least to the extent 
 realistically possible.

 So what should a well-designed function X do when faced with 
 known-to-be unsupported input Y?[1] Here are the 
 possibilities[2]:

 ([1] "Unsupported" is defined here as "for the given input, the 
 function does not correctly and successfully perform its 
 intended goal".)


I don't think that accurately describes the situation. 
Considering how there is now a toggle which changes the API 
implementation's behavior, the limitation is no longer really a 
part of the API, but a part of the implementation.

Consider, also, other implementations of the Win32 API (Wine and 
ReactOS), as well as possible future versions of Windows (which 
may do away with this limitation entirely).


 1. Add support for input Y to function X.
 2. Reject input Y (via either: abort, throw or static compile 
 error)
 3. Accept input Y and allow one of the following to occur[3]: 
 Silently fail to perform the expected task, silently do the 
 wrong thing, or trigger an unclear error 
 (assert/exception/compile) from deeper in the callstack. ([3] 
 Note that which one of these three possibilities occurs is 
 dependent on function X's exact implementation details.)

 Of these three possibilities for a function X faced with 
 unsupported input Y, the first two options are (in theory) 
 acceptable[4]. The third possibility is absolutely not 
 acceptable [to the extent realistically possible.]


This is, again, flawed logic. It is unreasonable to expect what 
the implementation of the underlying API will do, because it is 
outside of our control. Consider that this discussion would have 
occurred just before Microsoft added the registry switch to allow 
long path names. The solution would have been "obvious"; the 
reality is that it is an implementation we do not control and can 
change in the future at whim. At best, we can treat the API at 
face value and don't attempt to work around implementation quirks.

Consider also the case of special filenames like "con" or "prn". 
Creating these using the standard API is not allowed, but this 
can also be bypassed with UNC paths. All the arguments in favor 
of making Phobos support paths longer than MAX_PATH (using the 
UNC prefix) seem to also favor detecting and supporting these 
reserved file names. But is doing so really Phobos' burden?

We simply cannot define our API in terms of what we think the 
underlying implementation supports or doesn't.


 No, that's completely nuts!
 A library, especially a standard library, should not introduce 
 new limitations, but pampering over the limitations of the 
 platform is not the right thing to do.


 This is debatable. Why, exactly, is pampering over 
 pre-Win10-v1607's maximum non-\\?\ filepath length a bad thing? 
 Exactly what problems does it cause?


https://forum.dlang.org/post/bqsjebjxuljlqusaobst forum.dlang.org


 If the platforms API is piling POS, there's nothing a sane 
 library can do about.


 That is patently untrue. It might be true in specific 
 circumstances, but it is not generally true. If you believe it 
 to be true in this specific case, then please explain 
 *how*/*why* there is nothing the library can do about it.


It is not Phobos' job to work around quirks in implementations 
beyond our control which can change at any moment.


 The general rule of thumb is: "Typical situations should work 
 as expected, atypical situations should be possible."


Operating on paths longer than MAX_PATH is not a typical 
situation.
Sep 18 2018
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 06:05:38 UTC, Vladimir 
Panteleev wrote:

 [...]


One more thing:

There is the argument that the expected behavior of Phobos 
functions creating filesystems objects with long paths is to 
succeed and create those files. However, this results in 
filesystem objects that most software will fail to access 
(everyone needs to also use the long paths workaround).

One point of view is that the expected behavior is that the 
functions succeed. Another point of view is that Phobos should 
not allow programs to create files and directories with invalid 
paths. Consider, e.g. that a user writes a program that creates a 
large tree of deeply nested filesystem objects. When they are 
done and wish to delete them, their file manager fails and 
displays an error. The user's conclusion? D sucks because it 
corrupts the filesystem and creates objects they can't operate 
with.
Sep 18 2018
next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir 
Panteleev wrote:

 On Wednesday, 19 September 2018 at 06:05:38 UTC, Vladimir 
 Panteleev wrote:

 [...]


 One more thing:

 There is the argument that the expected behavior of Phobos 
 functions creating filesystems objects with long paths is to 
 succeed and create those files. However, this results in 
 filesystem objects that most software will fail to access 
 (everyone needs to also use the long paths workaround).

 One point of view is that the expected behavior is that the 
 functions succeed. Another point of view is that Phobos should 
 not allow programs to create files and directories with invalid 
 paths. Consider, e.g. that a user writes a program that creates 
 a large tree of deeply nested filesystem objects. When they are 
 done and wish to delete them, their file manager fails and 
 displays an error. The user's conclusion? D sucks because it 
 corrupts the filesystem and creates objects they can't operate 
 with.


I was wanting to reply with something similar:)

My 2 cents..whatever it's worth.  Vladimir has expressed most if 
not all the points I would have brought up.  Abscissa did bring 
up a good idea to help users support long filenames, but I agree 
with Vladimir that this should be "opt-in".  Provide a function 
in phobos for it, plus, it lets them cache the result AND 
infinitely better, the developer knows what's going on.  What 
drives me mad is when you have library writers who try to 
"protect" you from the underlying system by translating 
everything you do into what they "think" you're trying to do.  
This will inevitably result in large complex adaptation layers as 
both the underlying system and the front-facing API change over 
time with unwieldy maintenance burden.  An opt-in solution 
doesn't have this problem because you've kept each solution 
orthogonal rather than developing a translation layer that needs 
to be able to determine what the underlying system does or does 
not support.  This is a fundamental example of encapsulation, the 
filesystem library should be it's own component with the windows 
filesystem workaround being an optional "add-on" that the 
filesystem library doesn't need to know about.  This workaround 
could look like an extra function in phobos...or you could even 
write a module that wraps std.file and does the translation on a 
per-call basis.
Sep 18 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 02:33 AM, Jonathan Marler wrote:

 
 What drives me mad is when you have library writers 
 who try to "protect" you from the underlying system by translating 
 everything you do into what they "think" you're trying to do.


What drives me mad is when allegedly cross-platform tools deliberately 
propagate non-cross-platform quirks that could easily be abstracted away 
and pretend that's somehow "helping" me instead of making a complete 
wreck of the whole point of cross-platform. Bonus points if they're 
doing it mainly to help with my C++-standard premature optimizations.

If I actually want to deal with platform-specific quirks, then I'll use 
the platform's API directly. (And then I'll beat myself with a brick, 
just for fun.)
Sep 19 2018
next sibling parent Ecstatic Coder <ecstatic.coder gmail.com> writes:
On Thursday, 20 September 2018 at 02:48:06 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 09/19/2018 02:33 AM, Jonathan Marler wrote:

 
 What drives me mad is when you have library writers who try to 
 "protect" you from the underlying system by translating 
 everything you do into what they "think" you're trying to do.


 What drives me mad is when allegedly cross-platform tools 
 deliberately propagate non-cross-platform quirks that could 
 easily be abstracted away and pretend that's somehow "helping" 
 me instead of making a complete wreck of the whole point of 
 cross-platform. Bonus points if they're doing it mainly to help 
 with my C++-standard premature optimizations.

 If I actually want to deal with platform-specific quirks, then 
 I'll use the platform's API directly. (And then I'll beat 
 myself with a brick, just for fun.)


+1

A cross-platform library has to be designed to operate in the 
same way on each supported platform, even if this means that it's 
harder to implement on some platform, or that some platforms will 
need more complicated implementations.

That's the whole point of this "HAL" approach.
Sep 19 2018
prev sibling parent reply Kagamin <spam here.lot> writes:
On Thursday, 20 September 2018 at 02:48:06 UTC, Nick Sabalausky 
(Abscissa) wrote:

 What drives me mad is when allegedly cross-platform tools 
 deliberately propagate non-cross-platform quirks that could 
 easily be abstracted away and pretend that's somehow "helping" 
 me instead of making a complete wreck of the whole point of 
 cross-platform.


http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html

A strictly conforming application must not require a larger 
value for correct operation.


{_POSIX_NAME_MAX}
     Maximum number of bytes in a filename (not including 
terminating null).
     Value: 14
{_POSIX_PATH_MAX}
     Maximum number of bytes in a pathname.
     Value: 256
Sep 20 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/20/2018 03:38 AM, Kagamin wrote:

 On Thursday, 20 September 2018 at 02:48:06 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 What drives me mad is when allegedly cross-platform tools deliberately 
 propagate non-cross-platform quirks that could easily be abstracted 
 away and pretend that's somehow "helping" me instead of making a 
 complete wreck of the whole point of cross-platform.


 
 http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html

 A strictly conforming application must not require a larger value for 
 correct operation.


 {_POSIX_NAME_MAX}
      Maximum number of bytes in a filename (not including terminating 
 null).
      Value: 14
 {_POSIX_PATH_MAX}
      Maximum number of bytes in a pathname.
      Value: 256


I generally omit obvious disclaimers such as "to a reasonable extent" 
and "within reason".

I've yet to see a current, non-esoteric *nix that has such limitations. 
Particularly that first one. If it were a realistic thing, like it is 
with Windows, then I'd be all for a similarly simple workaround if such 
exists. But to my knowledge, that is more theoretical/historical than it 
is realistic.
Sep 20 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir 
Panteleev wrote:

 One point of view is that the expected behavior is that the 
 functions succeed. Another point of view is that Phobos should 
 not allow programs to create files and directories with invalid 
 paths. Consider, e.g. that a user writes a program that creates 
 a large tree of deeply nested filesystem objects. When they are 
 done and wish to delete them, their file manager fails and 
 displays an error. The user's conclusion? D sucks because it 
 corrupts the filesystem and creates objects they can't operate 
 with.


You don't even need to use crazy third-party software.

Try this program:

	mkdir(`\\?\C:\ a \`);
	write(`\\?\C:\ a \a.txt`, "Hello");

Then, try doing the following:

- Double-click the created text file.

- Try deleting the directory from Explorer (by sending it to the 
recycle bin).

- Try permanently deleting it (Shift+Delete).

- Try renaming it.

All of these fail for me. Deleting the directory doesn't even 
show an error - nothing at all happens.

When the OS itself fails to properly deal with such files, I 
don't think D has any business in *facilitating* their creation 
by default.
Sep 19 2018
next sibling parent Neia Neutuladh <neia ikeran.org> writes:
On Thursday, 20 September 2018 at 03:15:20 UTC, Vladimir 
Panteleev wrote:

 When the OS itself fails to properly deal with such files, I 
 don't think D has any business in *facilitating* their creation 
 by default.


Dear lord Windows is terrible. Can we just deprecate it?
Sep 19 2018
prev sibling next sibling parent Ecstatic Coder <ecstatic.coder gmail.com> writes:
On Thursday, 20 September 2018 at 03:15:20 UTC, Vladimir 
Panteleev wrote:

 On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir 
 Panteleev wrote:

 One point of view is that the expected behavior is that the 
 functions succeed. Another point of view is that Phobos should 
 not allow programs to create files and directories with 
 invalid paths. Consider, e.g. that a user writes a program 
 that creates a large tree of deeply nested filesystem objects. 
 When they are done and wish to delete them, their file manager 
 fails and displays an error. The user's conclusion? D sucks 
 because it corrupts the filesystem and creates objects they 
 can't operate with.


 You don't even need to use crazy third-party software.

 Try this program:

 	mkdir(`\\?\C:\ a \`);
 	write(`\\?\C:\ a \a.txt`, "Hello");

 Then, try doing the following:

 - Double-click the created text file.

 - Try deleting the directory from Explorer (by sending it to 
 the recycle bin).

 - Try permanently deleting it (Shift+Delete).

 - Try renaming it.

 All of these fail for me. Deleting the directory doesn't even 
 show an error - nothing at all happens.

 When the OS itself fails to properly deal with such files, I 
 don't think D has any business in *facilitating* their creation 
 by default.


*Windows Explorer* prevents you from creating a folder or file 
whose name STARTS with spaces. It trims them automatically, 
whether you want it or not.

So it's NOT a surprise that *Windows Explorer* (!) has problems 
if you use it on such files which were created manually.

But obviously, *Windows* OS doesn't prevent you to create them 
through scripts and applications...
Sep 19 2018
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 11:15 PM, Vladimir Panteleev wrote:

 On Wednesday, 19 September 2018 at 06:11:22 UTC, Vladimir Panteleev wrote:

 One point of view is that the expected behavior is that the functions 
 succeed. Another point of view is that Phobos should not allow 
 programs to create files and directories with invalid paths. Consider, 
 e.g. that a user writes a program that creates a large tree of deeply 
 nested filesystem objects. When they are done and wish to delete them, 
 their file manager fails and displays an error. The user's conclusion? 
 D sucks because it corrupts the filesystem and creates objects they 
 can't operate with.


 
 You don't even need to use crazy third-party software.
 
 Try this program:
 
      mkdir(`\\?\C:\ a \`);
      write(`\\?\C:\ a \a.txt`, "Hello");
 
 Then, try doing the following:
 
 - Double-click the created text file.
 
 - Try deleting the directory from Explorer (by sending it to the recycle 
 bin).
 
 - Try permanently deleting it (Shift+Delete).
 
 - Try renaming it.
 
 All of these fail for me. Deleting the directory doesn't even show an 
 error - nothing at all happens.
 
 When the OS itself fails to properly deal with such files, I don't think 
 D has any business in *facilitating* their creation by default.
 


I used to be a pure Windows user for a long, long time. Trust me, if 
you're using Windows, you run into PLENTY of ways to generate files that 
explorer can't handle. Even the system's own commandline makes their 
creation trivial (it also makes it trivial remove/rename such files, 
too). Very little else on Windows has the same bizarre, 
inconsistent-with-the-OS-itself limitations that explorer has. It's a 
jungle. Welcome to Windows. *shrug*
Sep 20 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/20/2018 03:59 AM, Nick Sabalausky (Abscissa) wrote:

 On 09/19/2018 11:15 PM, Vladimir Panteleev wrote:

 When the OS itself fails to properly deal with such files, I don't 
 think D has any business in *facilitating* their creation by default.


 
 I used to be a pure Windows user for a long, long time. Trust me, if 
 you're using Windows, you run into PLENTY of ways to generate files that 
 explorer can't handle. Even the system's own commandline makes their 
 creation trivial (it also makes it trivial remove/rename such files, 
 too). Very little else on Windows has the same bizarre, 
 inconsistent-with-the-OS-itself limitations that explorer has. It's a 
 jungle. Welcome to Windows. *shrug*


In any case, the last thing Windows users need is more software that 
gives them the same "It's not even compatible with its own freaking 
native filesystem" headaches explorer already gives them.

(Not that I don't have opposite beefs with Linux - Linux could use a lot 
LESS software that happily supports file/dir names with forward slashes 
and hidden control codes in them. Ugh, ya can't win anywhere in 
computing...)
Sep 20 2018
prev sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Wednesday, 19 September 2018 at 06:05:38 UTC, Vladimir 
Panteleev wrote:


 Operating on paths longer than MAX_PATH is not a typical 
 situation.


https://forum.rejectedsoftware.com/groups/rejectedsoftware.dub/thread/1499/

The worst situation was node.js on Windows,  anyway...
Sep 18 2018
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 06:16:21 UTC, Paolo Invernizzi 
wrote:

 On Wednesday, 19 September 2018 at 06:05:38 UTC, Vladimir 
 Panteleev wrote:


 Operating on paths longer than MAX_PATH is not a typical 
 situation.


 https://forum.rejectedsoftware.com/groups/rejectedsoftware.dub/thread/1499/

 The worst situation was node.js on Windows,  anyway...


Not sure that's actually MAX_PATH related... cmd has a very low 
limit on command line length. Ran into this myself:

https://github.com/VerySleepy/tests/blob/721e52fcb14d8134394264586a4fe92e73574059/scripts/toolchains_download.cmd#L8
Sep 18 2018
prev sibling next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
(Abscissa) wrote:

 [...]


Someone mentioned in this thread that .NET runtime does do the 
long-path workaround automatically. One thing we could do is copy 


The rationale being that:
- .NET is made by Microsoft
- The Windows API's filesystem implementation is made by Microsoft
- Given that these two are made by the same party, it's 
reasonable to assume that the .NET authors authoritatively "knew 
what they were doing" when implementing the workaround.
- The algorithm used by .NET is very likely to be supported by 
the API (even future implementations), as well as third-party 
implementations of the API.

However, there are still drawbacks to this:

- There is still the matter of overhead (one OS API call 
(GetCurrentDirectory) and at least one GC allocation (for the 
current directory buffer)).
- Using paths longer than MAX_PATH is an exceptional situation. 
Putting the workaround in the main code path penalizes 99.9% of 
use cases.
- The registry switch in newer Windows versions removes the need 
for this workaround, so systems with it enabled are penalized as 
well.
- There is still the matter regarding special filenames, as well 
as whether the expected behavior is really to succeed and create 
paths inaccessible to most software, instead of failing.
Sep 18 2018
next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 02:26 AM, Vladimir Panteleev wrote:

 On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 [...]


 
 Someone mentioned in this thread that .NET runtime does do the long-path 

 is doing.
 


This is a complete textbook example of the "appeal to authority" fallacy.

If an approach is valid, then it stands on its own merits regardless of 
whether or not Microsoft implemented it.

If an approach in invalid, then it fails on its own demerits regardless 
of whether or not Microsoft implemented it.

What MS has or hasn't implemented and released is completely irrelevant 
WRT validity and correctness.

What it *might* be useful for is as a starting point for further 
exploration. But that is all.


 However, there are still drawbacks to this:
 
 - There is still the matter of overhead (one OS API call 
 (GetCurrentDirectory) and at least one GC allocation (for the current 
 directory buffer)).


If one extra OS API call + allocation per std.file API call is 
unacceptable, then explain how it is unacceptable. I disagree that it is 
significant enough to be unacceptable.

If a user needs to optimize their 
already-working-for-all-accepted-inputs application, then they are free 
to do so. I argue that building this into the standard library's default 
behaviour amounts to mandatory premature optimization, prioritizing 
premature optimization over correctness. Prove me wrong.


 - Using paths longer than MAX_PATH is an exceptional situation. Putting 
 the workaround in the main code path penalizes 99.9% of use cases.


I have many filepaths on my system right now which exceed MAX_PATH in 
total length. I submit that this "penalty" you speak of is nothing more 
than a trivial performance boost at the expense of correctness. 
Furthermore, I submit that long paths which need extra optimization are 
MORE exceptional than long paths which do NOT need extra optimization.


 - The registry switch in newer Windows versions removes the need for 
 this workaround, so systems with it enabled are penalized as well.


Using the Phobos-based workaround on a system WITH the longpath setting 
supported and enabled results in slightly reduced performance (which can 
be overridden and optimized when necessary). Note that it is possible 
(and very simple) to detect this situation and handle it optimally by 
skipping the workaround.

OTOH, NOT using the Phobos-based workaround on a system where the 
longpath setting is NOT supported *OR* NOT enabled results in erroneous 
behavior. Not something as trivial as slightly-degraded performance on 
IO access.

The superior default is clear: Use the workaround except where the 
workaround in known to be safe to omit.


 - There is still the matter regarding special filenames,


If you're referring to NUL, COM1, COM2, etc, then this is completely 
orthogonal.


 as well as 
 whet her the expected behavior is really to succeed and create paths 
 inaccessible to most software, instead of failing.


Ok, suppose we decide "Sure, we have reason to believe there may be a 
significant amount of software on Windows which fails to handle long 
paths and we want to ensure maximum compatibility with those admittedly 
broken programs." That's fine. I can get behind that. HOWEVER, that does 
NOT mean we should leave our APIs as they are, because currently, our 
APIs fail at that goal. Instead, what it really means is that our APIs 
should be designed to *REJECT* long paths with an appropriately 
meaningful error message - and a reasonable workaround - and NOT to 
blindly just pass them along as they currently do.

Either way, Phobos needs changed:

Do you believe D should prevent its own software from being broken on 
long paths? Then Phobos should be modified to detect and fix long paths.

Do you believe D should permit breakage on long paths and encourage its 
programs to play nicely with other non-D Windows software that is *also* 
broken on long paths? Then Phobos should be modified to detect and 
*reject* long paths.

Either way, the current Phobos behavior is clearly the worst of both 
worlds and needs modification.
Sep 19 2018
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 08:18:38 UTC, Nick Sabalausky 
(Abscissa) wrote:

 Someone mentioned in this thread that .NET runtime does do the 
 long-path workaround automatically. One thing we could do is 

 


 This is a complete textbook example of the "appeal to 
 authority" fallacy.

 If an approach is valid, then it stands on its own merits 
 regardless of whether or not Microsoft implemented it.

 If an approach in invalid, then it fails on its own demerits 
 regardless of whether or not Microsoft implemented it.

 What MS has or hasn't implemented and released is completely 
 irrelevant WRT validity and correctness.

 What it *might* be useful for is as a starting point for 
 further exploration. But that is all.


No, absolutely not.

Microsoft is in charge of the implementation. We can't know that 
any deviation from Microsoft's algorithm will work in all 
situations, or all past/future implementations of the API. There 
is the considerable possibility that there are situations which 
we cannot foresee from our limited knowledge of the problem and 
Windows API implementation; on the other hand, Microsoft not only 
has complete knowledge of the implementation, but also controls 
its future. They have an incentive to keep the .NET algorithm 
working.


it is our fault.


it breaks, it's Microsoft's fault.

You cannot evaluate any intrinsic merit here because the result 
is beyond your control.


 If one extra OS API call + allocation per std.file API call is 
 unacceptable, then explain how it is unacceptable. I disagree 
 that it is significant enough to be unacceptable.


It is not unacceptable, but it is a drawback.


 If a user needs to optimize their 
 already-working-for-all-accepted-inputs application, then they 
 are free to do so. I argue that building this into the standard 
 library's default behaviour amounts to mandatory premature 
 optimization, prioritizing premature optimization over 
 correctness. Prove me wrong.


You could extend this argument to any severity of workarounds. 
Where do you draw the line?


 - Using paths longer than MAX_PATH is an exceptional 
 situation. Putting the workaround in the main code path 
 penalizes 99.9% of use cases.


 I have many filepaths on my system right now which exceed 
 MAX_PATH in total length. I submit that this "penalty" you 
 speak of is nothing more than a trivial performance boost at 
 the expense of correctness. Furthermore, I submit that long 
 paths which need extra optimization are MORE exceptional than 
 long paths which do NOT need extra optimization.


Optimization is the least concern.


 - The registry switch in newer Windows versions removes the 
 need for this workaround, so systems with it enabled are 
 penalized as well.


 Using the Phobos-based workaround on a system WITH the longpath 
 setting supported and enabled results in slightly reduced 
 performance (which can be overridden and optimized when 
 necessary).


I'm more concerned about differences in behavior.


 OTOH, NOT using the Phobos-based workaround on a system where 
 the longpath setting is NOT supported *OR* NOT enabled results 
 in erroneous behavior.


I disagree that failure on paths exceeding MAX_PATH is 
necessarily erroneous behavior. The API reports an error given 
the user's path, so should Phobos.


 The superior default is clear: Use the workaround except where 
 the workaround in known to be safe to omit.


We don't even have an algorithm for determining for sure when the 
workaround is needed.


 - There is still the matter regarding special filenames,


 If you're referring to NUL, COM1, COM2, etc, then this is 
 completely orthogonal.


Yes. How so? It is the same issue: paths with certain properties 
are valid on all platforms except on Windows. Phobos errors out 
when attempting to access/create them. A simple workaround is 
available: expand/normalize the path, prepend the UNC prefix, and 
use Unicode APIs.


 as well as whet her the expected behavior is really to succeed 
 and create paths inaccessible to most software, instead of 
 failing.


 Ok, suppose we decide "Sure, we have reason to believe there 
 may be a significant amount of software on Windows which fails 
 to handle long paths and we want to ensure maximum 
 compatibility with those admittedly broken programs." That's 
 fine. I can get behind that. HOWEVER, that does NOT mean we 
 should leave our APIs as they are, because currently, our APIs 
 fail at that goal. Instead, what it really means is that our 
 APIs should be designed to *REJECT* long paths with an 
 appropriately meaningful error message - and a reasonable 
 workaround - and NOT to blindly just pass them along as they 
 currently do.


This is not possible, because you need to precisely know how the 
implementation will handle the path. Considering the 
implementation's behavior can be configured by the user, I don't 
think this is feasible.


 Either way, Phobos needs changed:

 Do you believe D should prevent its own software from being 
 broken on long paths? Then Phobos should be modified to detect 
 and fix long paths.

 Do you believe D should permit breakage on long paths and 
 encourage its programs to play nicely with other non-D Windows 
 software that is *also* broken on long paths? Then Phobos 
 should be modified to detect and *reject* long paths.

 Either way, the current Phobos behavior is clearly the worst of 
 both worlds and needs modification.


Sorry, I don't see how you're reaching that conclusion. Looks 
like a false dichotomy.
Sep 19 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 08:36:35 UTC, Vladimir 
Panteleev wrote:

 If you're referring to NUL, COM1, COM2, etc, then this is 
 completely orthogonal.


 Yes. How so? It is the same issue: paths with certain 
 properties are valid on all platforms except on Windows. Phobos 
 errors out when attempting to access/create them. A simple 
 workaround is available: expand/normalize the path, prepend the 
 UNC prefix, and use Unicode APIs.


I just remembered, there is a third class of paths with these 
properties: paths containing directory components that begin or 
end with spaces.

There are probably more... I think some special characters are 
also valid only in UNC paths.
Sep 19 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 08:46:13 UTC, Vladimir 
Panteleev wrote:

 On Wednesday, 19 September 2018 at 08:36:35 UTC, Vladimir 
 Panteleev wrote:

 If you're referring to NUL, COM1, COM2, etc, then this is 
 completely orthogonal.


 Yes. How so? It is the same issue: paths with certain 
 properties are valid on all platforms except on Windows. 
 Phobos errors out when attempting to access/create them. A 
 simple workaround is available: expand/normalize the path, 
 prepend the UNC prefix, and use Unicode APIs.


 I just remembered, there is a third class of paths with these 
 properties: paths containing directory components that begin or 
 end with spaces.

 There are probably more... I think some special characters are 
 also valid only in UNC paths.


BTW, something follows from the above:

write(`C:\` ~ (short path) ~  `con`) will fail

but:

write(`C:\` ~ (long path) ~ `con`) will succeed.

This is just one issue I've noticed... there's probably more 
lurking. This is why I think the whole idea is bankrupt.
Sep 19 2018
parent reply Neia Neutuladh <neia ikeran.org> writes:
On Wednesday, 19 September 2018 at 08:54:42 UTC, Vladimir 
Panteleev wrote:

 BTW, something follows from the above:

 write(`C:\` ~ (short path) ~  `con`) will fail

 but:

 write(`C:\` ~ (long path) ~ `con`) will succeed.

 This is just one issue I've noticed... there's probably more 
 lurking.


Also, according to the internet:

write(chainPath(shortDirectory, "A "), "Win32 API strips trailing 
space");
readText(chainPath(shortDirectory, "A")); // Win32 API strips 
trailing space

But:
write(chainPath(longDirectory, "A "), "Win32 API strips trailing 
space");
readText(chainPath(longDirectory, "A")); // File not found

write(chainPath(shortDirectory, "A."));  // fails
write(chainPath(longDirectory, "A."));  // succeeds


 This is why I think the whole idea is bankrupt.


This is why we should use the exact same behavior in all cases. 
Always use `\\?\` or never use it.

Since Windows path handling is weird by default, I'd prefer 
always using `\\?\`. It's overhead, but not a huge amount of 
additional overhead compared to filesystem manipulation.
Sep 19 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 01:49 PM, Neia Neutuladh wrote:

 On Wednesday, 19 September 2018 at 08:54:42 UTC, Vladimir Panteleev wrote:

 BTW, something follows from the above:

 write(`C:\` ~ (short path) ~  `con`) will fail

 but:

 write(`C:\` ~ (long path) ~ `con`) will succeed.

 This is just one issue I've noticed... there's probably more lurking.


 
 Also, according to the internet:
 
 write(chainPath(shortDirectory, "A "), "Win32 API strips trailing space");
 readText(chainPath(shortDirectory, "A")); // Win32 API strips trailing 
 space
 
 But:
 write(chainPath(longDirectory, "A "), "Win32 API strips trailing space");
 readText(chainPath(longDirectory, "A")); // File not found
 
 write(chainPath(shortDirectory, "A."));  // fails
 write(chainPath(longDirectory, "A."));  // succeeds
 


Ok, excellent, now we're getting somewhere! :) Based on these actual 
concrete examples, yes, I can now grant that converting paths that are 
past the limit would indeed cause subtly different behaviour.

So clearly this needs to be an all-or-nothing deal...


 This is why we should use the exact same behavior in all cases. Always 
 use `\\?\` or never use it.


Yup.


 Since Windows path handling is weird by default, I'd prefer always using 
 `\\?\`. It's overhead, but not a huge amount of additional overhead 
 compared to filesystem manipulation.


My thoughts, exactly.

I'll also point out that this applies to user-code, too. Ie, with Phobos 
as it currently stands, user code has three options:

1. Never use `\\?\` and any time they compile their code on Windows, 
they must always be acutely aware of, and plan for, all of the fun 
little quirks of the Win33 file APIs.

2. Be *aware* of `\\?\` and always use it any time they compile their 
code on Windows (and always remember use longPath() or some such for all 
file operations when working on a cross-platform codebase).

3. Sometimes use `\\?\` and sometimes don't, and then deal with the fun 
little inconsistencies that creates.


terrible. But gee, wouldn't it be nice if we could just relieve that 

case where "doing the safe/correct thing by default" isn't good enough, 
there'd be nothing stopping anyone from calling the platform API 
directly if they really, really need to.

Opt-out is for safe, reliable, consistent things.
Opt-in is for difficult, tricky, dangerous things.
Not the other way around.
Sep 19 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 08:18:38 UTC, Nick Sabalausky 
(Abscissa) wrote:

 Instead, what it really means is that our APIs should be 
 designed to *REJECT* long paths with an appropriately 
 meaningful error message


On my Windows VM, I get:

C:\(long path here): The filename or extension is too long. 
(error 206)

This seems like a completely reasonable error message to me, so I 
think we're good there already.
Sep 19 2018
parent Kagamin <spam here.lot> writes:
On Wednesday, 19 September 2018 at 09:58:00 UTC, Vladimir 
Panteleev wrote:

 On my Windows VM, I get:

 C:\(long path here): The filename or extension is too long. 
 (error 206)

 This seems like a completely reasonable error message to me, so 
 I think we're good there already.


It can be a long file name. Each individual component in the path 
still should not exceed MAX_PATH even when you use unparsed path.
Sep 19 2018
prev sibling parent reply Kagamin <spam here.lot> writes:
On Wednesday, 19 September 2018 at 06:26:21 UTC, Vladimir 
Panteleev wrote:

 Someone mentioned in this thread that .NET runtime does do the 
 long-path workaround automatically.


AFAIK, CoreFX does, but .net doesn't. .net did its own path 
normalization and length check, which can be turned off since 
4.6.2.
Sep 19 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 09:58:30 UTC, Kagamin wrote:

 On Wednesday, 19 September 2018 at 06:26:21 UTC, Vladimir 
 Panteleev wrote:

 Someone mentioned in this thread that .NET runtime does do the 
 long-path workaround automatically.


 AFAIK, CoreFX does, but .net doesn't. .net did its own path 
 normalization and length check, which can be turned off since 
 4.6.2.


Thanks. I had a quick look. It looks pretty involved. Here are 
some of the relevant parts:

https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathHelper.Windows.cs#L16-L43
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathHelper.Windows.cs#L68-L95
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/Path.Windows.cs#L37-L63
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathInternal.Windows.cs#L82-L99
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/CoreLib/System/IO/PathInternal.Windows.cs#L118-L141

And for example directory deletion:

https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/System.IO.FileSystem/src/System/IO/Directory.cs#L296-L300
https://github.com/dotnet/corefx/blob/0566028e42a8f2fa86b8e0dd856d1f13d0fdce05/src/Common/src/Interop/Windows/kernel32/Interop.RemoveDirectory.cs#L19-L23

Some things stood out to me:

- GetFullPathName is documented as also having the MAX_PATH 
limit, but the framework seems to use it for normalization BEFORE 
prepending the prefix.

   
https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-getfullpathnamea

- GetFullPathName has a big warning on it about how you shouldn't 
use it in multithreaded programs.

- The code seems to compare the length against 260 characters, 
but in my tests, the limit is actually about 12 characters 
shorter. The same file defines MaxShortDirectoryPath = 248, but 
that constant isn't used anywhere in the code.

Maybe we shouldn't use this as a reference after all...
Sep 19 2018
parent Kagamin <spam here.lot> writes:
On Wednesday, 19 September 2018 at 10:29:11 UTC, Vladimir 
Panteleev wrote:

 - GetFullPathName is documented as also having the MAX_PATH 
 limit, but the framework seems to use it for normalization 
 BEFORE prepending the prefix.

   
 https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-getfullpathnamea


That part is erroneous :)
Recent additions to msdn are too often incompetent, e.g. long 
path registry switch in windows 10 doesn't enable the feature, 
it's only the first of two gates, manifest is not optional. Don't 
ask me who does it. They can't even get backslashes right.
Passing unparsed path to GetFullPathName makes no sense because 
such unparsed path must be already full normalized path, and it 
doesn't have MAX_PATH limit, probably because it doesn't talk to 
file system.


 - GetFullPathName has a big warning on it about how you 
 shouldn't use it in multithreaded programs.


I wonder about that too, shouldn't the system do the same to 
resolve the absolute path? Theoretically it can keep an open 
handle to the current directory and get its path from that.
Sep 19 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
(Abscissa) wrote:

 2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 
 bytes[5].


This is not a good criteria: relative paths whose pointing to 
objects whose absolute path exceeds MAX_PATH will fail, too. So, 
it looks like Phobos would need to expand relative paths 
unconditionally.
Sep 19 2018
next sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 11:04:13 UTC, Vladimir 
Panteleev wrote:

 On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick 
 Sabalausky (Abscissa) wrote:

 2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 
 bytes[5].


 This is not a good criteria: relative paths whose pointing to 
 objects whose absolute path exceeds MAX_PATH will fail, too. 
 So, it looks like Phobos would need to expand relative paths 
 unconditionally.


Here is my test program:
https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12a986
Sep 19 2018
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 07:04 AM, Vladimir Panteleev wrote:

 On Wednesday, 19 September 2018 at 05:49:41 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 2. Detect and reject any non-\\?\ path longer than MAX_PATH-12 bytes[5].


 
 This is not a good criteria: relative paths whose pointing to objects 
 whose absolute path exceeds MAX_PATH will fail, too. So, it looks like 
 Phobos would need to expand relative paths unconditionally.
 


I'm not sure I'm quite following you. Is this what you mean?:

string dir = ...; // Such that...
assert( dir.isRelativePath );
assert( dir.length < MAX_LENGTH-12 );
assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );

// ??? This *currently* goes BOOM on Windows
// ??? installations with MAX_LENGTH restriction active?
rmdir(path);

(Not on a Win box at the moment.)
Sep 19 2018
next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 11:23 PM, Nick Sabalausky (Abscissa) wrote:

 
 rmdir(path);


Obviously meant "rmdir(dir);" here. Editing mishap.
Sep 19 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Thursday, 20 September 2018 at 03:25:05 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 09/19/2018 11:23 PM, Nick Sabalausky (Abscissa) wrote:

 
 rmdir(path);


 Obviously meant "rmdir(dir);" here. Editing mishap.


and MAX_PATH instead of MAX_LENGTH, and absolutePath instead of 
toAbsolutePath, and !isAbsolute instead of isRelativePath, but I 
understood what you meant :)
Sep 19 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 11:27 PM, Vladimir Panteleev wrote:

 On Thursday, 20 September 2018 at 03:25:05 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 On 09/19/2018 11:23 PM, Nick Sabalausky (Abscissa) wrote:

 rmdir(path);


 Obviously meant "rmdir(dir);" here. Editing mishap.


 
 and MAX_PATH instead of MAX_LENGTH, and absolutePath instead of 
 toAbsolutePath, and !isAbsolute instead of isRelativePath, but I 
 understood what you meant :)
 


Ugh, yea, the perils of coding from memory ;)
Sep 20 2018
prev sibling next sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Thursday, 20 September 2018 at 03:23:36 UTC, Nick Sabalausky 
(Abscissa) wrote:

 I'm not sure I'm quite following you. Is this what you mean?:

 string dir = ...; // Such that...
 assert( dir.isRelativePath );
 assert( dir.length < MAX_LENGTH-12 );
 assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );

 // ??? This *currently* goes BOOM on Windows
 // ??? installations with MAX_LENGTH restriction active?
 rmdir(path);

 (Not on a Win box at the moment.)


Correct.
Sep 19 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Thursday, 20 September 2018 at 03:23:36 UTC, Nick Sabalausky 
(Abscissa) wrote:

 (Not on a Win box at the moment.)


I added the output of my test program to the gist:
https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12a986#file-output-txt


 assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );


Actually it's crazier than that. The concatenation of the current 
directory plus the relative path must be < MAX_PATH (approx.). 
Meaning, if you are 50 directories deep, a relative path starting 
with 50 `..\` still won't allow you to access C:\file.txt.
Sep 19 2018
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Sep 20, 2018 at 03:45:07AM +0000, Vladimir Panteleev via Digitalmars-d
wrote:
[...]

 Actually it's crazier than that. The concatenation of the current
 directory plus the relative path must be < MAX_PATH (approx.).
 Meaning, if you are 50 directories deep, a relative path starting with
 50 `..\` still won't allow you to access C:\file.txt.


Wow. I didn't know Windows came with its own chroot() functionality!

</sarcasm> :-P


T

-- 
My program has no bugs! Only undocumented features...
Sep 20 2018
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 11:45 PM, Vladimir Panteleev wrote:

 On Thursday, 20 September 2018 at 03:23:36 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 (Not on a Win box at the moment.)


 
 I added the output of my test program to the gist:
 https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12
986#file-output-txt 
 
 

 assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );


 
 Actually it's crazier than that. The concatenation of the current 
 directory plus the relative path must be < MAX_PATH (approx.). Meaning, 
 if you are 50 directories deep, a relative path starting with 50 `..\` 
 still won't allow you to access C:\file.txt.
 


Ouch. Ok, yea, this is pretty solid evidence that ALL usage of 
non-`\\?\` paths on Windows needs to be killed dead, dead, dead.

If it were decided (not that I'm in favor of it) that we should be 
protecting developers from files named " a ", "a." and "COM1", then that 
really needs to be done on our end on top of mandatory `\\?\`-based 
access. Anyone masochistic enough to really WANT to deal with MAX_PATH 
and such is free to access the Win32 APIs directly.
Sep 20 2018
next sibling parent Ecstatic Coder <ecstatic.coder gmail.com> writes:
On Thursday, 20 September 2018 at 19:49:01 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 09/19/2018 11:45 PM, Vladimir Panteleev wrote:

 On Thursday, 20 September 2018 at 03:23:36 UTC, Nick 
 Sabalausky (Abscissa) wrote:

 (Not on a Win box at the moment.)


 
 I added the output of my test program to the gist:
 https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12a986#file-output-txt
 
 

 assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );


 
 Actually it's crazier than that. The concatenation of the 
 current directory plus the relative path must be < MAX_PATH 
 (approx.). Meaning, if you are 50 directories deep, a relative 
 path starting with 50 `..\` still won't allow you to access 
 C:\file.txt.
 


 Ouch. Ok, yea, this is pretty solid evidence that ALL usage of 
 non-`\\?\` paths on Windows needs to be killed dead, dead, dead.

 If it were decided (not that I'm in favor of it) that we should 
 be protecting developers from files named " a ", "a." and 
 "COM1", then that really needs to be done on our end on top of 
 mandatory `\\?\`-based access. Anyone masochistic enough to 
 really WANT to deal with MAX_PATH and such is free to access 
 the Win32 APIs directly.


+1

On Windows, every logical path provided to the std file functions 
should be properly converted to a physical path starting with 
that prefix.

Obviously this won't solve ALL Windows-specific problems, but 
that will AT LEAST remove a whole class of them.
Sep 21 2018
prev sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 20 September 2018 at 19:49:01 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 09/19/2018 11:45 PM, Vladimir Panteleev wrote:

 On Thursday, 20 September 2018 at 03:23:36 UTC, Nick 
 Sabalausky (Abscissa) wrote:

 (Not on a Win box at the moment.)


 
 I added the output of my test program to the gist:
 https://gist.github.com/CyberShadow/049cf06f4ec31b205dde4b0e3c12a986#file-output-txt
 
 

 assert( dir.toAbsolutePath.length > MAX_LENGTH-12 );


 
 Actually it's crazier than that. The concatenation of the 
 current directory plus the relative path must be < MAX_PATH 
 (approx.). Meaning, if you are 50 directories deep, a relative 
 path starting with 50 `..\` still won't allow you to access 
 C:\file.txt.
 


 Ouch. Ok, yea, this is pretty solid evidence that ALL usage of 
 non-`\\?\` paths on Windows needs to be killed dead, dead, dead.

 If it were decided (not that I'm in favor of it) that we should 
 be protecting developers from files named " a ", "a." and 
 "COM1", then that really needs to be done on our end on top of 
 mandatory `\\?\`-based access. Anyone masochistic enough to 
 really WANT to deal with MAX_PATH and such is free to access 
 the Win32 APIs directly.


Decided to play around with this for a bit.  Made a "proof of 
concept" library:

https://github.com/marler8997/longfiles

It's just a prototype/exploration on the topic.  It allows you to 
include "stdx.longfiles" instead of "std.file" which will enable 
the conversion in every call, or you can import "stdx.longfiles : 
toLongPath" and use that on filenames passed to std.file.  
There's also a test you can run

rund test/test_with_longfiles.d (should work)
rund test/test_without_longfiles.d (should fail)

NOTE: use "rund test/cleantests.d" to remove the files...I wasn't 
able to via the windows explorer program.
Sep 22 2018
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 22 September 2018 at 20:46:27 UTC, Jonathan Marler 
wrote:

 Decided to play around with this for a bit.  Made a "proof of 
 concept" library:


I suggest using GetFullPathNameW instead of GetCurrentDirectory + 
manual path appending / normalization. It's also what CoreFX 
seems to be doing.
Sep 22 2018
parent Jonathan Marler <johnnymarler gmail.com> writes:
On Saturday, 22 September 2018 at 21:04:04 UTC, Vladimir 
Panteleev wrote:

 On Saturday, 22 September 2018 at 20:46:27 UTC, Jonathan Marler 
 wrote:

 Decided to play around with this for a bit.  Made a "proof of 
 concept" library:


 I suggest using GetFullPathNameW instead of GetCurrentDirectory 
 + manual path appending / normalization. It's also what CoreFX 
 seems to be doing.


Yes that allows the library to avoid calling buildNormalizedPath. 
  I've implemented and pushed this change.  This change also 
exposed a weakness in the Appender interface and I've created a 
bug for it:

https://issues.dlang.org/show_bug.cgi?id=19259

The problem is there's no way to extend the length of the data in 
an appender if you don't use the `put` functions.  So when I call 
GetFullPathNameW function to populate the data (like the .NET 
CoreFX implementation does) I can't extend the length.
Sep 22 2018
prev sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/22/2018 04:46 PM, Jonathan Marler wrote:

 
 Decided to play around with this for a bit.  Made a "proof of concept" 
 library:
 
 https://github.com/marler8997/longfiles
 
 It's just a prototype/exploration on the topic.  It allows you to 
 include "stdx.longfiles" instead of "std.file" which will enable the 
 conversion in every call, or you can import "stdx.longfiles : 
 toLongPath" and use that on filenames passed to std.file.


Cool! Will have to take a closer look and try it out.

Regarding this: "TODO: what should be done about the MS-DOS FAT 
filesystem?"...

First of all, FAT16 can still be fully-used with the current interfaces 
anyway - it's just that if you attempt anything FAT16 doesn't support, 
the error you get will come from the OS rather than a D lib. But 
*unlike* the non-`\\?\` path issues, there really isn't anything here 
that needs to be worked around, or that even *can* be sensibly worked 
around.

Besides, FAT16 is a rarely-used, long-since-outdated legacy format. Its 
successor, FAT32 has been around for more than 20 years, and I'm not 
aware of anything more recent than the 3.5" floppy that uses it by 
default. I'd say it safely falls into the category of "Too much of an 
esoteric special-case to be worth requiring that special support be 
added in the main 'path of least resistance' interface (as long as 
there's nothing preventing the user from handling it on their own if 
they really need to.)"
Sep 22 2018
prev sibling parent reply tide <tide tide.tide> writes:
On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir Panteleev 
wrote:

 On Sunday, 16 September 2018 at 16:17:21 UTC, tide wrote:

 Nothing is "locked behind management". If you feel that some 
 issue important to you is stalled, you can create a forum 
 thread, or email Walter/Andrei to ask for a resolution.


 Funny the other guy was saying to create a bugzilla issue.


 Do that *first*.


That's already been done.


 The path needs to be normalized, which means that \.\ and 
 \..\ fragments need to be removed away first. Depending on 
 your interpretation of symlinks/junctions/etc., "foo/bar/../" 
 might mean something else than "foo/" if "bar" is a reparse 
 point.


 All these issues yet for some reason this function was 
 included in the lot: 
 https://dlang.org/phobos/std_path.html#absolutePath
 [...]
 This issue exists anyways, you'd only expand the path when it 
 need to be used. If the file changes within milliseconds, I 
 don't see that happening often and if it does there's a flaw 
 in your design that'd happen even if the path didn't have to 
 be constructed first.


 You've missed the point. Complexity breeds bugs and unexpected 
 behavior. The expectation is that D's function to delete a file 
 should do little else than call the OS function.

 If *YOU* are OK with the consequences of complexity, implement 
 this in YOUR code, but do not enforce it upon others.


version(Windows)
{
     if(path.length >= MAX_PATH)
     {
         // throw Exception(...) // effectively what happens now

         // do workaround for
     }
}

The complexity would only exist for those that need it. It'd be 
the difference between their code not working and code working. 
I'm sure people would rather their code work than not work in 
this case.


 So you pass a valid path (selected by a user through a UI) to 
 rmDir, and it doesn't remove the directory. You think this is 
 acceptable behavior?


 It is absolutely not acceptable behavior. Complain to 
 Microsoft. The OS should not allow users to create or select 
 paths that programs cannot operate on without jumping through 
 crazy hoops.


Not that crazy, you can get the actual absolutePath with one of 
the OS functions. It isn't that difficult of a workaround.
Sep 17 2018
next sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 17 September 2018 at 22:58:46 UTC, tide wrote:

 version(Windows)
 {
     if(path.length >= MAX_PATH)
     {
         // throw Exception(...) // effectively what happens now

         // do workaround for
     }
 }

 The complexity would only exist for those that need it. It'd be 
 the difference between their code not working and code working. 
 I'm sure people would rather their code work than not work in 
 this case.


No good:

1. When hitting the situation where the extra logic does make a 
difference, and the program is operating on paths with some being 
under the the limit and some over, this will make it behave 
inconsistently depending on the data it's operating on.

2. When the registry key you mentioned is set, the workaround is 
unnecessary, and the extra logic can introduce unwanted behavior.


 Not that crazy, you can get the actual absolutePath with one of 
 the OS functions. It isn't that difficult of a workaround.


Which OS function is that, for the record?
Sep 17 2018
prev sibling parent Ecstatic Coder <ecstatic.coder gmail.com> writes:
On Monday, 17 September 2018 at 22:58:46 UTC, tide wrote:

 On Sunday, 16 September 2018 at 22:40:45 UTC, Vladimir 
 Panteleev wrote:

 On Sunday, 16 September 2018 at 16:17:21 UTC, tide wrote:

 Nothing is "locked behind management". If you feel that some 
 issue important to you is stalled, you can create a forum 
 thread, or email Walter/Andrei to ask for a resolution.


 Funny the other guy was saying to create a bugzilla issue.


 Do that *first*.


 That's already been done.


 The path needs to be normalized, which means that \.\ and 
 \..\ fragments need to be removed away first. Depending on 
 your interpretation of symlinks/junctions/etc., 
 "foo/bar/../" might mean something else than "foo/" if "bar" 
 is a reparse point.


 All these issues yet for some reason this function was 
 included in the lot: 
 https://dlang.org/phobos/std_path.html#absolutePath
 [...]
 This issue exists anyways, you'd only expand the path when it 
 need to be used. If the file changes within milliseconds, I 
 don't see that happening often and if it does there's a flaw 
 in your design that'd happen even if the path didn't have to 
 be constructed first.


 You've missed the point. Complexity breeds bugs and unexpected 
 behavior. The expectation is that D's function to delete a 
 file should do little else than call the OS function.

 If *YOU* are OK with the consequences of complexity, implement 
 this in YOUR code, but do not enforce it upon others.


 version(Windows)
 {
     if(path.length >= MAX_PATH)
     {
         // throw Exception(...) // effectively what happens now

         // do workaround for
     }
 }

 The complexity would only exist for those that need it. It'd be 
 the difference between their code not working and code working. 
 I'm sure people would rather their code work than not work in 
 this case.


 So you pass a valid path (selected by a user through a UI) to 
 rmDir, and it doesn't remove the directory. You think this is 
 acceptable behavior?


 It is absolutely not acceptable behavior. Complain to 
 Microsoft. The OS should not allow users to create or select 
 paths that programs cannot operate on without jumping through 
 crazy hoops.


 Not that crazy, you can get the actual absolutePath with one of 
 the OS functions. It isn't that difficult of a workaround.


"Workaround" ;)

That's the problem actually.

As suggested previously, the std.file functions should call a 
GetPhysicalPath function which just returns the path unchanged on 
Linux and MacOS, and on Windows simply checks if the file path is 
smaller or not than the 256 character limit, and if needed makes 
it absolute and prefixes it.

This has no performance impact, and brings a consistent behavior 
across platforms.

THAT would be a nice solution for the cross-platform developers 
who erroneously think that the standard library is already Doing 
The Right Thing (TM) so that their code doesn't need 
platform-specific "workarounds"...
Sep 18 2018
prev sibling next sibling parent tide <tide tide.tide> writes:
On Sunday, 16 September 2018 at 03:19:12 UTC, Vladimir Panteleev 
wrote:

 On Sunday, 16 September 2018 at 02:58:30 UTC, tide wrote:

 There are a lot of issues that aren't simple bugs that just 
 anyone can fix. They are issues that are locked behind 
 management. One's that are 4 years old for example, they are 
 probably some bug locked behind management. That's why they 
 get so old. From the comments it is not clear that a pull 
 request wouldn't be accepted to fix the issue. Personally I 
 think phobos should not exception for long file names.


 Nothing is "locked behind management". If you feel that some 
 issue important to you is stalled, you can create a forum 
 thread, or email Walter/Andrei to ask for a resolution.


 Walters concern is that the path will change unexpected for 
 the user. Where does that matter for something like rmDir ? 
 The user passes a long path, and rmDir swallows it, never to 
 be seen again by the user. What does it matter if the path 
 gets corrected if it is too long?


 It's more than that.

 The path needs to be normalized, which means that \.\ and \..\ 
 fragments need to be removed away first. Depending on your 
 interpretation of symlinks/junctions/etc., "foo/bar/../" might 
 mean something else than "foo/" if "bar" is a reparse point.

 The path also needs to be absolute, so it has to be expanded to 
 a full path before it can be prefixed with the UNC prefix. 
 Given how the current directory is state tied to the process 
 (not thread), you get bonus race condition issues. There's also 
 issues like the current directory object being renamed/moved; 
 then a relative path will no longer correspond to what the 
 program thinks the absolute paths is.

 All things considered, this goes well into the territory of 
 "not D's problem". My personal recommendation: if you care 
 about long path names, use an operating system which implements 
 them right.


I'd agree with you that it isn't **Phobos** problem, but since 
most of the functions there aren't  system nor  nogc, I do 
believe it is. And if you want  system and  nogc with no safety 
you can go look into core.stdc for that.
Sep 16 2018
prev sibling parent tide <tide tide.tide> writes:
On Sunday, 16 September 2018 at 03:19:12 UTC, Vladimir Panteleev 
wrote:

 On Sunday, 16 September 2018 at 02:58:30 UTC, tide wrote:

 There are a lot of issues that aren't simple bugs that just 
 anyone can fix. They are issues that are locked behind 
 management. One's that are 4 years old for example, they are 
 probably some bug locked behind management. That's why they 
 get so old. From the comments it is not clear that a pull 
 request wouldn't be accepted to fix the issue. Personally I 
 think phobos should not exception for long file names.


 Nothing is "locked behind management". If you feel that some 
 issue important to you is stalled, you can create a forum 
 thread, or email Walter/Andrei to ask for a resolution.


 Walters concern is that the path will change unexpected for 
 the user. Where does that matter for something like rmDir ? 
 The user passes a long path, and rmDir swallows it, never to 
 be seen again by the user. What does it matter if the path 
 gets corrected if it is too long?


 It's more than that.

 The path needs to be normalized, which means that \.\ and \..\ 
 fragments need to be removed away first. Depending on your 
 interpretation of symlinks/junctions/etc., "foo/bar/../" might 
 mean something else than "foo/" if "bar" is a reparse point.

 The path also needs to be absolute, so it has to be expanded to 
 a full path before it can be prefixed with the UNC prefix. 
 Given how the current directory is state tied to the process 
 (not thread), you get bonus race condition issues. There's also 
 issues like the current directory object being renamed/moved; 
 then a relative path will no longer correspond to what the 
 program thinks the absolute paths is.

 All things considered, this goes well into the territory of 
 "not D's problem". My personal recommendation: if you care 
 about long path names, use an operating system which implements 
 them right.


Well my mistake, seems absolutePath is just named incorrectly and 
should be more accurately named concatenatePath.
Sep 16 2018
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/15/2018 08:14 PM, Jonathan M Davis wrote:

 
 The issue was reported in bugzilla quite some time ago.
 
 https://issues.dlang.org/show_bug.cgi?id=8967
 
 However, while Walter's response on it basically indicates that we should
 just close it as "won't fix," we never actually did


It's worth noting that the discussion made it very clear that Walter's 
viewpoint on the matter was based on his own misunderstanding (ie, 
mistakenly failed to notice that `\\?\` != `\\.\`) He never actually 
made any comment after that was pointed out.
Sep 18 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, September 18, 2018 7:28:43 PM MDT Nick Sabalausky (Abscissa) via 
Digitalmars-d wrote:

 On 09/15/2018 08:14 PM, Jonathan M Davis wrote:

 The issue was reported in bugzilla quite some time ago.

 https://issues.dlang.org/show_bug.cgi?id=8967

 However, while Walter's response on it basically indicates that we
 should
 just close it as "won't fix," we never actually did


 It's worth noting that the discussion made it very clear that Walter's
 viewpoint on the matter was based on his own misunderstanding (ie,
 mistakenly failed to notice that `\\?\` != `\\.\`) He never actually
 made any comment after that was pointed out.


If a clean, simple solution can be found that allows long paths on Windows
to work seemlessly without subtle side effects, then I don't see why it
can't be implemented, but it needs to be something that's not going to cause
problems. Otherwise, it needs to be left up to the caller to do whatever
happens to be correct for their particular circumstances. We want Phobos to
work seemlessly across platforms where reasonably possible, but
unfortunately, it's not always reasonable. Either way, Microsoft has clearly
made a mess of this. So, I don't know how reasonable it is to work around
it. Regardless, we either need to figure out a sane way to work around the
problem (without causing new problems in the process) or document it so that
the situation is clear.

- Jonathan M Davis
Sep 18 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/18/2018 09:46 PM, Jonathan M Davis wrote:

 On Tuesday, September 18, 2018 7:28:43 PM MDT Nick Sabalausky (Abscissa) via
 Digitalmars-d wrote:

 It's worth noting that the discussion made it very clear that Walter's
 viewpoint on the matter was based on his own misunderstanding (ie,
 mistakenly failed to notice that `\\?\` != `\\.\`) He never actually
 made any comment after that was pointed out.


 
 If a clean, simple solution can be found that allows long paths on Windows
 to work seemlessly without subtle side effects, then I don't see why it
 can't be implemented, but it needs to be something that's not going to cause
 problems. Otherwise, it needs to be left up to the caller to do whatever
 happens to be correct for their particular circumstances. We want Phobos to
 work seemlessly across platforms where reasonably possible, but
 unfortunately, it's not always reasonable. Either way, Microsoft has clearly
 made a mess of this. So, I don't know how reasonable it is to work around
 it. Regardless, we either need to figure out a sane way to work around the
 problem (without causing new problems in the process) or document it so that
 the situation is clear.
 


Exactly. And this is precisely why I'm irritated by just how much of 
this entire thread has been meaningless high-level philosophical 
hand-wringing, with barely any attention paid to the technical details 
of the problem itself.

Though I admit, I've allowed myself to get dragged into it, too. So 
allow me to rectify that:

1. For reference, here is the relevant MS documentation:
https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#file-and-directory-names

2. For reference, here is the existing bug report discussion:
https://issues.dlang.org/show_bug.cgi?id=8967

3. Building on what Vladimir and Jay have said in the bug report, I 
propose we do this:

- We add a public function to Phobos which takes a a UTF-8 path and does 
the following:
     - No-op outside of Windows
     - No-op if the path is less than MAX_PATH-12 bytes. (Worrying about 
the case where the real limit is MAX_PATH-1 would be inconsequential and 
thus needless.)
     - No-op if the path begins with "\\" (and is therefore either a 
network share path, a "\\.\" device path, is already a "\\?\" path, or 
is just some undetectable malformed path that happens to look like one 
of the above)
     - Otherwise, returns:
         toUTF16z(buildNormalizedPath("\\?", path.toAbsolute))
     - By "no-op" above, I really mean: toUTF16z(path)
- In all cases where Phobos passes a path to WinAPI, the path is first 
passed through this function (except for any specific cases where it can 
be shown that the path should NOT be passed through this function).

4. What technical problems does this proposal have?

5. For each technical problem with the proposal: How can the proposal be 
adjusted to compensate? Or, why can the technical problem NOT be 
reasonably solved? Or in general: How should this proposal be modified, 
and why?
Sep 18 2018
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 02:20:45 UTC, Nick Sabalausky 
(Abscissa) wrote:

 3. Building on what Vladimir and Jay have said in the bug 
 report, I propose we do this:


This has been proposed before in this thread. I don't think it's 
a good idea:

https://forum.dlang.org/post/bqsjebjxuljlqusaobst forum.dlang.org

I proposed something a little different:

https://forum.dlang.org/post/dnlbtkmdcjucqetkwsey forum.dlang.org
Sep 18 2018
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/15/2018 09:54 AM, tide wrote:

 On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:

 For very long file names it is broke and every command fails. These 
 paths are not all that long but over 256 limit. (For windows)


 Please file a bug report with reproducible examples if you believe 
 it's a bug.


 
 I feel people need to stop saying this. It feels like people are just 
 being told to say this if there is a bug. There is a larger issue, 
 Bugzilla doesn't and isn't working. Someone will probably throw up some 
 stats about how many bugs are filed and how many are resolved. Those 
 exist because someone working on Dlang comes across a bug that affects 
 them, creates a patch for it first, then goes and creates a bugzilla 
 entry and marks it resolved. Issues are rarely resolved by anyone other 
 than the person that created the bug report to begin with. Or issues 
 created by a team member is resolved by another team member.


While that's admittedly all-too-true, filing a proper bug report is 
still an essential step.

Like you, I'm all for bringing attention to important issues on the 
newsgroup. However, it is CRUCIAL for this to be IN ADDITION to filing a 
bug report, and NOT INSTEAD of filing a bug report.
Sep 18 2018
parent bachmeier <no spam.net> writes:
On Tuesday, 18 September 2018 at 19:33:00 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 09/15/2018 09:54 AM, tide wrote:

 On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo 
 wrote:

 For very long file names it is broke and every command 
 fails. These paths are not all that long but over 256 limit. 
 (For windows)


 Please file a bug report with reproducible examples if you 
 believe it's a bug.


 
 I feel people need to stop saying this. It feels like people 
 are just being told to say this if there is a bug. There is a 
 larger issue, Bugzilla doesn't and isn't working. Someone will 
 probably throw up some stats about how many bugs are filed and 
 how many are resolved. Those exist because someone working on 
 Dlang comes across a bug that affects them, creates a patch 
 for it first, then goes and creates a bugzilla entry and marks 
 it resolved. Issues are rarely resolved by anyone other than 
 the person that created the bug report to begin with. Or 
 issues created by a team member is resolved by another team 
 member.


 While that's admittedly all-too-true, filing a proper bug 
 report is still an essential step.

 Like you, I'm all for bringing attention to important issues on 
 the newsgroup. However, it is CRUCIAL for this to be IN 
 ADDITION to filing a bug report, and NOT INSTEAD of filing a 
 bug report.


Correct. There's no point in having a lengthy discussion on the 
topic without a bug report because it'll just waste their time 
and then they'll complain that their bug is being ignored. Unless 
there is a change in the operation of this project, all bugs 
reported in the forum need to be addressed by telling them to 
file a bug report.
Sep 18 2018
prev sibling parent tide <tide tide.tide> writes:
On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo 
 wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)


 Please file a bug report with reproducible examples if you 
 believe it's a bug.


To add to that, a lot of the issues that get posted on the forum 
already have bug reports, and those reports have been there for 
*years*.
Sep 15 2018
prev sibling next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)

 The problem this causes can be disastrous. If some check fails 
 and runs code that isn't mean to run if the file exists, it 
 could destroy data.


 I replaced many of the std.file functions with 
 executeShell(...) and using command line functions and they 
 work. But then my code was still failing and it was because 
 exist was returning false even though the file exists.

 I'm sure this is not a big deal to some...


Without knowing the issue in detail, is this maybe related to 
https://www.google.de/amp/s/www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over
260-characters/amp/ ?

Kind regards
Andre
Sep 14 2018
prev sibling next sibling parent reply WebFreak001 <d.forum webfreak.org> writes:
On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)

 The problem this causes can be disastrous. If some check fails 
 and runs code that isn't mean to run if the file exists, it 
 could destroy data.


 I replaced many of the std.file functions with 
 executeShell(...) and using command line functions and they 
 work. But then my code was still failing and it was because 
 exist was returning false even though the file exists.

 I'm sure this is not a big deal to some...


this is an issue with the Win32 API having that 260 character 
limit. To work around it, you can use the special path syntax 
Microsoft allows you to do, which will pass the string you 
provide directly to the Filesystem instead of parsing it, 
effectively raising your limit to whatever the Filesystem limits 
to.

See also their official site on this: 
https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation

C:/Some/Very/Long/Path.txt

should then become

\\?\C:\Some\Very\Long\Path.txt

converting / to \ is important because according to the docs 
windows doesn't do it with the \\?\ prefix. You also have to 
normalize the path beforehand (i.e. remove ..\ and .\ because 
they would be treated as actual folder names)


To change a server path such as

\\share\public

you change it to

\\?\UNC\share\public



There are also some other useful things in there you might want 
to look at too like the special files such as COM1
Sep 15 2018
parent reply Josphe Brigmo <JospheBrigmo gmail.com> writes:
On Saturday, 15 September 2018 at 09:47:25 UTC, WebFreak001 wrote:

 On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo 
 wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)

 The problem this causes can be disastrous. If some check fails 
 and runs code that isn't mean to run if the file exists, it 
 could destroy data.


 I replaced many of the std.file functions with 
 executeShell(...) and using command line functions and they 
 work. But then my code was still failing and it was because 
 exist was returning false even though the file exists.

 I'm sure this is not a big deal to some...


 this is an issue with the Win32 API having that 260 character 
 limit. To work around it, you can use the special path syntax 
 Microsoft allows you to do, which will pass the string you 
 provide directly to the Filesystem instead of parsing it, 
 effectively raising your limit to whatever the Filesystem 
 limits to.

 See also their official site on this: 
 https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation

 C:/Some/Very/Long/Path.txt

 should then become

 \\?\C:\Some\Very\Long\Path.txt

 converting / to \ is important because according to the docs 
 windows doesn't do it with the \\?\ prefix. You also have to 
 normalize the path beforehand (i.e. remove ..\ and .\ because 
 they would be treated as actual folder names)


 To change a server path such as

 \\share\public

 you change it to

 \\?\UNC\share\public



 There are also some other useful things in there you might want 
 to look at too like the special files such as COM1



Yes, I did that and it worked for some(most) things it seems but 
rmdir, for example, seems to fail.

Also, windows 10 does not have this problem nor does unicode so 
maybe phobos needs to automatically do everything? If it did I 
wouldn't have had this problem and wasted a day of my life trying 
to figure out what is going on(I didn't design my program around 
having to hack such things, I just assumed they would work, 
because, after all, they should, right?). I then used execute 
shell and had to work around that but still had problems because 
I didn't think dirEntries would be failing.

Basically every file function will fail in some odd way(depending 
on it's use) leaving one to deal with a whole complex of "bugs" 
when there is really a common bug.

rmdir, now is failing but this may be some other bug introduced 
by me in trying to fix other bugs.
Sep 15 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 15 September 2018 at 10:05:26 UTC, Josphe Brigmo 
wrote:

 Yes, I did that and it worked for some(most) things it seems 
 but rmdir, for example, seems to fail.


If the file path is passed verbatim to the OS API, and it still 
doesn't work, then the problem is with the OS or the API, not D.


 Also, windows 10 does not have this problem


What do you mean by "windows 10"? Do you mean Explorer, the 
default file manager?

Some reasons why it might work whereas the API doesn't:

- It's using workarounds, like the \\?\ prefix, or 
moving/renaming the object to a shorter name before deleting it.
- It's using a newer API, like WinRT (introduced in Windows 8).
- It's using a secret Microsoft API.


 If it did I wouldn't have had this problem and wasted a day of 
 my life trying to figure out what is going on(I didn't design 
 my program around having to hack such things, I just assumed 
 they would work, because, after all, they should, right?).


I, too, spent a LOT of time fighting the Windows filesystem APIs. 
See e.g. * 
https://dump.thecybershadow.net/d78d9911adc16ec749914b6923759
54/longpathdelete.d (that also sets ownership/ACLs via external processes, as
the C API is unreasonably complicated).

The problem is 100% due to Windows.

It was one of the big reasons why I moved to Linux for software 
development. Such problems do not exist there.
Sep 15 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/15/2018 06:40 AM, Vladimir Panteleev wrote:

 On Saturday, 15 September 2018 at 10:05:26 UTC, Josphe Brigmo wrote:
 

 Also, windows 10 does not have this problem


 
 What do you mean by "windows 10"? Do you mean Explorer, the default file 
 manager?
 


According to MS docs:

"Starting in Windows 10, version 1607, MAX_PATH limitations have been 
removed from common Win32 file and directory functions. However, you 
must opt-in to the new behavior."

-- From: 
https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation

It goes on to explain the registry setting for that.



 If it did I wouldn't have had this problem and wasted a day of my life 
 trying to figure out what is going on(I didn't design my program 
 around having to hack such things, I just assumed they would work, 
 because, after all, they should, right?).


 
 I, too, spent a LOT of time fighting the Windows filesystem APIs. See 
 e.g. * 
 https://dump.thecybershadow.net/d78d9911adc16ec749914b6923759
54/longpathdelete.d 
 (that also sets ownership/ACLs via external processes, as the C API is 
 unreasonably complicated).
 
 The problem is 100% due to Windows.
 
 It was one of the big reasons why I moved to Linux for software 
 development. Such problems do not exist there.
 


Agreed, but this sounds to me like a perfect reason to abstract away as 
much as that garbage as we reasonably can. D's in the business of 
improving the user-experience of programming, not the promotion of one 
OS over another.

Bear in mind, the very point of plenty of things in Phobos, or any std 
lib for that matter, is to abstract away the need to waste everyone's 
time making them fiddle about with pointless little OS differences. 
Sure, there are going to be things that can't be reasonably abstracted 
away, but that's FAR from justifying not doing what we can. Again: 
normal expectations should be the default and just work, exceptional 
needs should still be possible.

And at least for me, moving from Windows to Linux would have been a LOT 
harder if it weren't for the OS abstractions that are already in Phobos.
Sep 18 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 01:50:54 UTC, Nick Sabalausky 
(Abscissa) wrote:

 And at least for me, moving from Windows to Linux would have 
 been a LOT harder if it weren't for the OS abstractions that 
 are already in Phobos.


It's one thing to call unlink on POSIX and RemoveFileW on 
Windows. Another is adding a good deal of extra logic that 
performs additional OS calls and generates additional GC garbage 
to work around API problems even on systems that don't need it.
Sep 18 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 12:04 AM, Vladimir Panteleev wrote:

 On Wednesday, 19 September 2018 at 01:50:54 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 And at least for me, moving from Windows to Linux would have been a 
 LOT harder if it weren't for the OS abstractions that are already in 
 Phobos.


 
 It's one thing to call unlink on POSIX and RemoveFileW on Windows. 


Granted.


 Another is adding a good deal of extra logic that performs additional OS 
 calls and generates additional GC garbage to work around API problems 
 even on systems that don't need it.


- Is it really?

- Does it actually, necessarily perform those additional OS calls?

- If it actually does, are those additional, necessarily OS calls 
prohibitively expensive? (Note that this is being compared to the 
theoretical minimum of successfully performing the same desired 
operation on the same data via the WinAPI, and not compared to the 
software which fails to perform appropriate checks for invalid input.)

- How have you determined that the "additional GC garbage" required to 
"work around API problems" is still significant "even on systems that 
don't need it"?
Sep 18 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 06:34:33 UTC, Nick Sabalausky 
(Abscissa) wrote:

 - Does it actually, necessarily perform those additional OS 
 calls?


We need to expand relative paths to absolute ones, for which we 
need to fetch the current directory.


 - Is it really?


Is what really what? If you mean the memory allocation, we do 
need a buffer to store the current directory. We also need to 
canonicalize away things like \..\, though we may be able to get 
away with it without allocating.


 - If it actually does, are those additional, necessarily OS 
 calls prohibitively expensive?


They are certainly going to be less expensive that actual 
filesystem operations that hit the physical disk, but it will 
still be an unwanted overhead in 99.9% of cases.

In any case, the overhead is only one issue.
Sep 18 2018
next sibling parent Ecstatic Coder <ecstatic.coder gmail.com> writes:
 They are certainly going to be less expensive that actual 
 filesystem operations that hit the physical disk, but it will 
 still be an unwanted overhead in 99.9% of cases.

 In any case, the overhead is only one issue.


Seriously, checking the file path string *length* is above 260 
characters to see if it needs to be fixed is not what I call an 
overhead.

And IF the path is indeed too long, IN THOSE CASES personally I'd 
prefer that the D standard library fixes the path in order to 
make the disk/file operation succeed, than having my application 
crash, because I didn't know I had to put a "version ( Windows )" 
fix somewhere in my code.

But hey, I may be wrong, software robustness and stability is 
often much overrated... ;)
Sep 19 2018
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 02:55 AM, Vladimir Panteleev wrote:

 On Wednesday, 19 September 2018 at 06:34:33 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 - Does it actually, necessarily perform those additional OS calls?


 
 We need to expand relative paths to absolute ones, for which we need to 
 fetch the current directory.
 


So in other words, NO, it does NOT necessarily perform additional OS 
calls. It ONLY performs an additional call if the given path is relative 
*AND* if we've decided to not simply reject too-long relative paths 
outright (which I'd be fine with as a compromise. At least it would be 
well-defined and enforced with a meaningful message.)


 - Is it really?


 
 Is what really what? If you mean the memory allocation, we do need a 
 buffer to store the current directory. We also need to canonicalize away 
 things like \..\, though we may be able to get away with it without 
 allocating.


So, in many cases, it's NOT really "a good deal of extra logic that 
performs additional OS calls and generates additional GC garbage".


 - If it actually does, are those additional, necessarily OS calls 
 prohibitively expensive?


 
 They are certainly going to be less expensive that actual filesystem 
 operations that hit the physical disk,


Sounds like QED to me. Especially if the alternative is silently 
incorrect behaviour on an entirely realistic subset of cases. (Realistic 
enough that both the thread's OP and the bug report's OP each ran into 
purely by accident.)


but it will still be an unwanted 
 overhead in 99.9% of cases.
 


That's an extremely exaggerated figure, and we've already established 
that the overhead is minor and often able to be elided. Weighted against 
the cost of incorrect behaviour on a subset of non-rejected inputs, I'd 
say that's a very clear "Yes, please!"

The extreme minority of currently-hypothetical cases which require 
minimal overhead for individual file I/O operations are free to 
low-level optimize themselves as-needed. Correct behaviour should never 
be sacrificed for minor performance tweaks when the minor performance 
tweak can still be obtained through other means if absolutely necessary.


 In any case, the overhead is only one issue.
 


What's the other issue(s)?
Sep 19 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 08:37:17 UTC, Nick Sabalausky 
(Abscissa) wrote:

 What's the other issue(s)?


Essentially they boil down to "it is impossible to prove the 
algorithm is correct" (for both detecting when the path fix is 
needed, and fixing the path). Forcing the path transformation can 
introduce regressions, or make the situation worse on systems 
where it's not needed.
Sep 19 2018
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/19/2018 04:41 AM, Vladimir Panteleev wrote:

 On Wednesday, 19 September 2018 at 08:37:17 UTC, Nick Sabalausky 
 (Abscissa) wrote:

 What's the other issue(s)?


 
 Essentially they boil down to "it is impossible to prove the algorithm 
 is correct" (for both detecting when the path fix is needed, and fixing 
 the path).


If you're referring to the inability to deterministically reason about 
just what in the h*ll MS's API's actually do, then I agree. But the 
problem is, it's equally true of all Win APIs. Only way to fix that is 
to omit Win support entirely.

Otherwise, I disagree. I think it is not only provable, but also 
unnecessary to prove simply because such proof has never been necessary 
for Phobos, and there is nothing inherent to this problem which is 
inherently more complicated than anything already existing in Phobos 
(you can even omit the questionable modules like std.xml, it all still 
holds). Otherwise, present counterexamples demonstrating the inherent 
ambiguity/non-provability.


 Forcing the path transformation can introduce regressions,


All phobos/compiler changes have the potential for regressions, plus we 
have unittests. Unless you can demonstrate how this necessarily goes 
above and beyond the risk from any other change in a way that cannot be 
sufficiently mitigated by tests, then the concern is irrelevant.


 or 
 make the situation worse on systems where it's not needed.
 


Provide an example where the situation is made worse.
Sep 19 2018
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 09:16:30 UTC, Nick Sabalausky 
(Abscissa) wrote:

 Essentially they boil down to "it is impossible to prove the 
 algorithm is correct" (for both detecting when the path fix is 
 needed, and fixing the path).


 If you're referring to the inability to deterministically 
 reason about just what in the h*ll MS's API's actually do, then 
 I agree. But the problem is, it's equally true of all Win APIs. 
 Only way to fix that is to omit Win support entirely.


It's not our job to fix it. Just provide a D interface to it, 
which already we do well.


 Otherwise, I disagree. I think it is not only provable, but 
 also unnecessary to prove simply because such proof has never 
 been necessary for Phobos, and there is nothing inherent to 
 this problem which is inherently more complicated than anything 
 already existing in Phobos (you can even omit the questionable 
 modules like std.xml, it all still holds).


No, we are mucking with data on the way between the user's 
program and the OS, because we think we can fix it. Not only 
should we not be doing that in the first place, but even if we 
get it right, it might still not be what the user wants.


 Otherwise, present counterexamples demonstrating the inherent 
 ambiguity/non-provability.


I don't understand what you mean here.


 Forcing the path transformation can introduce regressions,


 All phobos/compiler changes have the potential for regressions, 
 plus we have unittests. Unless you can demonstrate how this 
 necessarily goes above and beyond the risk from any other 
 change in a way that cannot be sufficiently mitigated by tests, 
 then the concern is irrelevant.


This might be a change which we won't be able to back out of if 
it turns out to be a bad idea, because then we break other 
classes of programs that depend on this change. See 
https://forum.dlang.org/post/eepblrtjmqzbtopylfib forum.dlang.org 
for an example.


 or make the situation worse on systems where it's not needed.


 Provide an example where the situation is made worse.


1. A user is happily using D on a system where the workaround is 
not needed.
2. A new D version comes out, with the workaround forcibly 
enabled.
3. The user's program is now broken.

If you provide a specific implementation for the workaround 
you're envisioning, I could try to come up with more specific 
situations where it would fail. There's been lots of reasons 
mentioned in this thread where things can go wrong, and surely 
there will be more that we can't think of ahead of time.
Sep 19 2018
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 19 September 2018 at 09:27:29 UTC, Vladimir 
Panteleev wrote:

 This might be a change which we won't be able to back out of if 
 it turns out to be a bad idea, because then we break other 
 classes of programs that depend on this change. See 
 https://forum.dlang.org/post/eepblrtjmqzbtopylfib forum.dlang.org for an
example.


Case in point:

https://msdn.microsoft.com/ru-ru/office/mt762842%28v=vs.90%29?f=255&MSPPError=-2147217396
Sep 19 2018
prev sibling parent tide <tide tide.tide> writes:
On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:

 For very long file names it is broke and every command fails. 
 These paths are not all that long but over 256 limit. (For 
 windows)

 The problem this causes can be disastrous. If some check fails 
 and runs code that isn't mean to run if the file exists, it 
 could destroy data.


 I replaced many of the std.file functions with 
 executeShell(...) and using command line functions and they 
 work. But then my code was still failing and it was because 
 exist was returning false even though the file exists.

 I'm sure this is not a big deal to some...


If you are on Windows 10 version 1607 or later, there's a 
registry key you can set so that the default behavior of the 
Win32 API allows long file path names. But yah, the problem is 
Windows, its horrible file system and structure thereof. You'd 
have faced this problem using any other language like C or C++ 
included.

HKLM\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled 
(Type: REG_DWORD)
Sep 15 2018