www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Bug in -J

reply Mr. Pib <Af343 MS.com> writes:
I have -J added to the command line like

-JC:\Temp

I then use import(r"C:\Temp\a.dat");

and I get an error about the file not existing in the

main.d: Error: file "C:\\Temp\\a.dat" cannot be found or not in a 
path specified with -J

It seems dmd does internally compare the paths to see if they are 
identical.

Of course, removing the C:\Temp\ part of import works fine.

The problem with that approach is it then is not consistent with 
other code. I need to specify the full path because sometimes it 
is used.

essentially

version(X)
import(path)
else
load(path);

while I could do something like

import(baseName(path))

it seems kinda clunky, in any case. It's pretty obvious that one 
excepts the same behavior so it could create bugs in code that 
except the behavior to work correctly.
Aug 11 2017
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 12/08/2017 3:34 AM, Mr. Pib wrote:
 I have -J added to the command line like
 
 -JC:\Temp
 
 I then use import(r"C:\Temp\a.dat");
 
 and I get an error about the file not existing in the
 
 main.d: Error: file "C:\\Temp\\a.dat" cannot be found or not in a path 
 specified with -J
 
 It seems dmd does internally compare the paths to see if they are 
 identical.
 
 Of course, removing the C:\Temp\ part of import works fine.
 
 The problem with that approach is it then is not consistent with other 
 code. I need to specify the full path because sometimes it is used.
 
 essentially
 
 version(X)
 import(path)
 else
 load(path);
 
 while I could do something like
 
 import(baseName(path))
 
 it seems kinda clunky, in any case. It's pretty obvious that one excepts 
 the same behavior so it could create bugs in code that except the 
 behavior to work correctly.
Most likely related or is the issue[0]. [0] https://issues.dlang.org/show_bug.cgi?id=3420
Aug 11 2017
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 12 August 2017 at 02:45:30 UTC, rikki cattermole 
wrote:
 On 12/08/2017 3:34 AM, Mr. Pib wrote:
 I have -J added to the command line like
 
 -JC:\Temp
 
 I then use import(r"C:\Temp\a.dat");
 
 and I get an error about the file not existing in the
 
 main.d: Error: file "C:\\Temp\\a.dat" cannot be found or not 
 in a path specified with -J
 
 It seems dmd does internally compare the paths to see if they 
 are identical.
No, it just prefixes the given import() path will all those given by -J.
 Of course, removing the C:\Temp\ part of import works fine.
 
 The problem with that approach is it then is not consistent 
 with other code. I need to specify the full path because 
 sometimes it is used.
 
 essentially
 
 version(X)
 import(path)
 else
 load(path);
I don't understand why you would want to do that. Using absolute paths makes assumptions about the system your software is being built on. Even if you never plan to build the software on any computer but your own, it still seems like a questionable design decision.
 while I could do something like
 
 import(baseName(path))
 
 it seems kinda clunky, in any case. It's pretty obvious that 
 one excepts the same behavior so it could create bugs in code 
 that except the behavior to work correctly.
You could also use -J/ and import(path[1..$]) (or the Windows equivalent). Still a bad idea.
 Most likely related or is the issue[0].

 [0] https://issues.dlang.org/show_bug.cgi?id=3420
Nope.
Aug 11 2017
parent reply Mr. Pib <Af343 MS.com> writes:
On Saturday, 12 August 2017 at 03:02:31 UTC, Vladimir Panteleev 
wrote:
 On Saturday, 12 August 2017 at 02:45:30 UTC, rikki cattermole 
 wrote:
 On 12/08/2017 3:34 AM, Mr. Pib wrote:
 I have -J added to the command line like
 
 -JC:\Temp
 
 I then use import(r"C:\Temp\a.dat");
 
 and I get an error about the file not existing in the
 
 main.d: Error: file "C:\\Temp\\a.dat" cannot be found or not 
 in a path specified with -J
 
 It seems dmd does internally compare the paths to see if they 
 are identical.
No, it just prefixes the given import() path will all those given by -J.
 Of course, removing the C:\Temp\ part of import works fine.
 
 The problem with that approach is it then is not consistent 
 with other code. I need to specify the full path because 
 sometimes it is used.
 
 essentially
 
 version(X)
 import(path)
 else
 load(path);
I don't understand why you would want to do that. Using absolute paths makes assumptions about the system your software is being built on. Even if you never plan to build the software on any computer but your own, it still seems like a questionable design decision.
You are making assumptions about me making assumptions... please don't make any more assumptions or we will be in an infinite regression ;/
 while I could do something like
 
 import(baseName(path))
 
 it seems kinda clunky, in any case. It's pretty obvious that 
 one excepts the same behavior so it could create bugs in code 
 that except the behavior to work correctly.
You could also use -J/ and import(path[1..$]) (or the Windows equivalent). Still a bad idea.
 Most likely related or is the issue[0].

 [0] https://issues.dlang.org/show_bug.cgi?id=3420
Nope.
If are aware, it has nothing to do with absolute paths as the bug should also be exhibited with relative paths. It is a comparison issue of the strings rather than checking to see if they represent the same physical location. It would be like saying that \x\y is different from \x\..\x\y. Dmd does a blind comparison, I bet, rather than what it should be doing. Obviously \x\y are different strings but they are not different file locations. If that is not the case, then a better error message should be given.
Aug 11 2017
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 12 August 2017 at 03:52:25 UTC, Mr. Pib wrote:
 You are making assumptions about me making assumptions... 
 please don't make any more assumptions or we will be in an 
 infinite regression ;/
Sorry, maybe I don't understand the question. Maybe you could explain in broader terms the higher-level goal or problem you're trying to solve, and maybe we can recommend a better way?
 If are aware, it has nothing to do with absolute paths as the 
 bug should also be exhibited with relative paths. It is a 
 comparison issue of the strings rather than checking to see if 
 they represent the same physical location.

 It would be like saying that \x\y is different from \x\..\x\y. 
 Dmd does a blind comparison, I bet, rather than what it should 
 be doing. Obviously \x\y are different strings but they are not 
 different file locations.
Fairly sure we forbid .. out of security considerations. I don't think this applies to Windows, but on POSIX, depending on how .. is interpreted, \x\y x/y actually can mean a different file from x/../x/y. It has taken a lot of consideration and research until we even allowed path separators in import paths. For a very long time, they were completely forbidden, and a long time after that, they were forbidden on Windows (because on Windows things can be more complicated due to the various kinds of reparse points and things such as short filenames).
 If that is not the case, then a better error message should be 
 given.
Feel free to file a diagnostic enhancement request if you have specific suggestions.
Aug 11 2017
parent reply Mr. Pib <Af343 MS.com> writes:
On Saturday, 12 August 2017 at 03:58:30 UTC, Vladimir Panteleev 
wrote:
 On Saturday, 12 August 2017 at 03:52:25 UTC, Mr. Pib wrote:
 You are making assumptions about me making assumptions... 
 please don't make any more assumptions or we will be in an 
 infinite regression ;/
Sorry, maybe I don't understand the question. Maybe you could explain in broader terms the higher-level goal or problem you're trying to solve, and maybe we can recommend a better way?
 If are aware, it has nothing to do with absolute paths as the 
 bug should also be exhibited with relative paths. It is a 
 comparison issue of the strings rather than checking to see if 
 they represent the same physical location.

 It would be like saying that \x\y is different from \x\..\x\y. 
 Dmd does a blind comparison, I bet, rather than what it should 
 be doing. Obviously \x\y are different strings but they are 
 not different file locations.
Fairly sure we forbid .. out of security considerations. I don't think this applies to Windows, but on POSIX, depending on how .. is interpreted, \x\y x/y actually can mean a different file from x/../x/y. It has taken a lot of consideration and research until we even allowed path separators in import paths. For a very long time, they were completely forbidden, and a long time after that, they were forbidden on Windows (because on Windows things can be more complicated due to the various kinds of reparse points and things such as short filenames).
 If that is not the case, then a better error message should be 
 given.
Feel free to file a diagnostic enhancement request if you have specific suggestions.
I'm pretty sure that on no OS does the same location mean different things? I am not talking about strange stuff but simple stuff. I have code that loads a file at runtime and requires the absolute path. This is only for debugging purposes. When built in release, everything is switched over to use imports and embed the files in the binary. The same path is used for other things like caching/uniqueID but are never actually read from. You see this sort of stuff a lot when you open an executable and see hard coded paths but obviously never used for file system purposes. The files and paths are all the same but import doens't seem to think so. Adding baseName solves the problem immediately but that is a hack. import should know that the path is the same as the one specified by -J. The whole point of -J is to specify the path for security purposes, right? So why does it matter if I use path\filename or baseName(filename)? Both point to the same location and both are consistent with -J, import should understand that. It is an obvious oversight. But there is an obvious programmatic difference between the two versions. Luckily, using baseName does fix the problem so it is not a huge deal but it is still a bug/issue with import for being ignorant of what it is actually doing.
Aug 11 2017
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 12 August 2017 at 05:42:21 UTC, Mr. Pib wrote:
 I'm pretty sure that on no OS does the same location mean 
 different things?
I understand what you mean, but just to clarify on the .. thing: $ mkdir d d/x d/z $ ln -s d/z x $ echo foo > d/z/y $ echo bar > d/x/y $ cat x/y foo + cat x/../x/y bar
 I am not talking about strange stuff but simple stuff.

 I have code that loads a file at runtime and requires the 
 absolute path. This is only for debugging purposes. When built 
 in release, everything is switched over to use imports and 
 embed the files in the binary. The same path is used for other 
 things like caching/uniqueID but are never actually read from. 
 You see this sort of stuff a lot when you open an executable 
 and see hard coded paths but obviously never used for file 
 system purposes.
Sounds like you can also work around it using e.g. enum appRoot = `C:\Temp\`; debug string data = readText(appRoot ~ "a.dat"); else enum data = import("a.dat"); (and build with -JC:\Temp). That should now also work if a.dat is in a subdirectory relative to the -J path.
 The files and paths are all the same but import doens't seem to 
 think so. Adding baseName solves the problem immediately but 
 that is a hack. import should know that the path is the same as 
 the one specified by -J. The whole point of -J is to specify 
 the path for security purposes, right? So why does it matter if 
 I use path\filename or baseName(filename)? Both point to the 
 same location and both are consistent with -J, import should 
 understand that. It is an obvious oversight. But there is an 
 obvious programmatic difference between the two versions. 
 Luckily, using baseName does fix the problem so it is not a 
 huge deal but it is still a bug/issue with import for being 
 ignorant of what it is actually doing.
Sounds reasonable, the compiler could check if paths start with a -J path.
Aug 11 2017
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 12 August 2017 at 06:02:57 UTC, Vladimir Panteleev 
wrote:
 Sounds reasonable, the compiler could check if paths start with 
 a -J path.
There is a potential ambiguity here: dmd -Jsomedir test.d test.d: import("somedir/file.txt"); Does the user mean to import "somedir/file.txt" or "somedir/somedir/file.txt"? Currently the latter is understood. Simply checking for path prefix would break this case. I guess this could be done only with absolute paths, but that introduces an inconsistency with relative paths. I'm not sure it's worth it, considering it's easy to work around.
Aug 11 2017
parent Mr. Pib <Af343 MS.com> writes:
On Saturday, 12 August 2017 at 06:06:33 UTC, Vladimir Panteleev 
wrote:
 On Saturday, 12 August 2017 at 06:02:57 UTC, Vladimir Panteleev 
 wrote:
 Sounds reasonable, the compiler could check if paths start 
 with a -J path.
There is a potential ambiguity here: dmd -Jsomedir test.d test.d: import("somedir/file.txt"); Does the user mean to import "somedir/file.txt" or "somedir/somedir/file.txt"? Currently the latter is understood. Simply checking for path prefix would break this case. I guess this could be done only with absolute paths, but that introduces an inconsistency with relative paths. I'm not sure it's worth it, considering it's easy to work around.
What is the inconsistency? Absolute paths are specific and well defined and so are relatives. Relatives are always relative to J, are they not? so, import("somedir/file.txt"); obviously means somedir/somedir/file.txt. If you wanted somedir/file.txt you would do import(`file.txt`). I see no problem here. With absolute paths, one simply checks if the absolute base path matches the absolute base path specified by J. If it matches, then it passes and can be used directly or removed internally. If not then an error is given. e.g., -JC:/basepath/somedir C:/basepath/somedir/file.txt import('C:/basepath/somedir/file.txt') import('file.txt') import('../somedir/file.txt') import('C:/basepath/../basepath/somedir/file.txt') import('C:/path/../basepath/file.txt') should pass all pass. import('../file.txt') import('C:/basepath/../basepath/file.txt') import('C:/path/file.txt') should all fail. (I'm only using .. for reference, not sure if they should actually be supported) It seems quite simple to me Any relative paths specified by import are prepended with the path specified by -J. Any absolute paths specified by an import are first matched to the path -J simply by example: if (J_path.length < basepath.length) assert("Import path must be located in the same path that is specified by -J"); if (basepath[0..J_path.length] != J_path) ditto; // makes sure absolute path is in the -J path. This is the check that is missing from dmd. (this doesn't resolve ..'s, that could be done before and assumes basepath is an absolute path)
Aug 12 2017