www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Phobos: __FILE__ as template default parameter

reply Johan Engelen <j j.nl> writes:
In std.exception, there are two functions that still take 
__FILE__ as default template parameter;

T enforce
     (T, Dg, string file = __FILE__, size_t line = __LINE__)
     (T value, scope Dg dg)

T errnoEnforce
     (T, string file = __FILE__, size_t line = __LINE__)
     (T value, lazy string msg = null)

The second one has caused me hours of pain :(
The template parameters end up in the symbol's mangled name, and 
besides the resulting template bloat (a new instantiation for 
each "errnoEnforce") it means that the source path is 
hard-encoded into a few function symbols in Phobos (interestingly 
not for DMD, but it is for the way LDC builds the Phobos lib it 
ships with). And so while working on (aggressive) cross-module 
inlining in LDC (it's working now!), I stumbled on linker errors 
because the symbols were missing --> they are there but with a 
different phobos path encoded in them!

All this to say:
   I hope someone can deprecate these two guys from Phobos, and 
replace them with functions that take __FILE__ as runtime 
parameter. (See the `enforce` that does not take a delegate as 
second parameter).

Thanks!
   Johan

(I'll have to work around this in LDC regardless. It'll mean that 
whenever you use __FILE__ as template parameter, that the calling 
function can not be inlined.)
Jun 18 2016
next sibling parent reply Johan Engelen <j j.nl> writes:
An example of how __FILE__ as template parameter will break your 
library:

In library, distributed in binary+source form:
```
alias file_templ_alias = file_templ!bool;
T file_templ(T, string file = __FILE__, size_t line = __LINE__) 
(T value) {
     return value;
}
```
In user code, linking to the library binary:
```
bool foo(bool b){
     return file_templ_alias(b);
}
```
By calling the alias, both DMD and LDC will decide that the 
template does not need reinstantiation in the user object file 
and will try to link with the symbol in the library binary. So 
you have to be lucky to install the library source in the same 
path as where it was when the library was built on someone else's 
machine, otherwise you'll get a linker error and are left 
wondering what went wrong.

-Johan
Jun 18 2016
parent Lass Safin <lasssafin gmail.com> writes:
On Saturday, 18 June 2016 at 18:13:22 UTC, Johan Engelen wrote:

 An example of how __FILE__ as template parameter will break 
 your library:

 In library, distributed in binary+source form:
 ```
 alias file_templ_alias = file_templ!bool;
 T file_templ(T, string file = __FILE__, size_t line = __LINE__) 
 (T value) {
     return value;
 }
 ```
 In user code, linking to the library binary:
 ```
 bool foo(bool b){
     return file_templ_alias(b);
 }
 ```
 By calling the alias, both DMD and LDC will decide that the 
 template does not need reinstantiation in the user object file 
 and will try to link with the symbol in the library binary. So 
 you have to be lucky to install the library source in the same 
 path as where it was when the library was built on someone 
 else's machine, otherwise you'll get a linker error and are 
 left wondering what went wrong.

 -Johan


Can't one just use __MODULE__ instead?

So:
T file_templ(T, string mod = __MODULE__, size_t line = __LINE__)
(T value) {
     return value;
}
Jun 19 2016
prev sibling parent reply Dicebot <public dicebot.lv> writes:
This important feature and can't be simply removed. For example, 
std.experimental.logger also relies on it. It needs to be fixed 
instead.

Two immediate workarounds that come to mmy mind:

1. Make __FILE__ relative to import path
2. Always emit new symbol to object file if __FILE__ is involved
Jun 19 2016
parent reply Johan Engelen <j j.nl> writes:
On Sunday, 19 June 2016 at 08:06:09 UTC, Dicebot wrote:

 This important feature and can't be simply removed. For 
 example, std.experimental.logger also relies on it.


Do you mean it relies on __FILE__ being a template parameter 
(instead of a runtime default parameter, like enforce)?


 It needs to be fixed instead.

 Two immediate workarounds that come to mmy mind:

 1. Make __FILE__ relative to import path
 2. Always emit new symbol to object file if __FILE__ is involved


I also thought about 2. It would fix the second (non-Phobos) 
example.

-Johan
Jun 19 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
On Sunday, 19 June 2016 at 08:33:40 UTC, Johan Engelen wrote:

 On Sunday, 19 June 2016 at 08:06:09 UTC, Dicebot wrote:

 This important feature and can't be simply removed. For 
 example, std.experimental.logger also relies on it.


 Do you mean it relies on __FILE__ being a template parameter 
 (instead of a runtime default parameter, like enforce)?


Yes. It is necessary because runtime parameter list is variadic - 
template bloat in such cases is usually eliminated by forwarding 
to another private method immediately turning file/line into 
first runtime argument instead.


 It needs to be fixed instead.

 Two immediate workarounds that come to mmy mind:

 1. Make __FILE__ relative to import path
 2. Always emit new symbol to object file if __FILE__ is 
 involved


 I also thought about 2. It would fix the second (non-Phobos) 
 example.

 -Johan
Jun 19 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-06-19 12:43, Dicebot wrote:


 Yes. It is necessary because runtime parameter list is variadic -
 template bloat in such cases is usually eliminated by forwarding to
 another private method immediately turning file/line into first runtime
 argument instead.


Would it be a bad idea to allow this in the compiler:

void foo(Args...)(Args args, string file = __FILE__, size_t line = 
__LINE__);

It wouldn't be possible to pass "file" or "line" when calling "foo". But 
it's useful for the special default values, __FILE__ and __LINE__.

-- 
/Jacob Carlborg
Jun 20 2016
next sibling parent Simen Kjaeraas <simen.kjaras gmail.com> writes:
On Monday, 20 June 2016 at 14:28:06 UTC, Jacob Carlborg wrote:

 Would it be a bad idea to allow this in the compiler:

 void foo(Args...)(Args args, string file = __FILE__, size_t 
 line = __LINE__);

 It wouldn't be possible to pass "file" or "line" when calling 
 "foo". But it's useful for the special default values, __FILE__ 
 and __LINE__.


I very much agree with this idea. Like with the current system of 
forwarding to a function with explicit file and line run-time 
arguments, a function can be made accessible that has them as 
explicit parameters for those cases when you want to pretend to 
be somewhere you're not. :p

--
   Simen
Jun 20 2016
prev sibling parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Monday, 20 June 2016 at 14:28:06 UTC, Jacob Carlborg wrote:

 On 2016-06-19 12:43, Dicebot wrote:


 Yes. It is necessary because runtime parameter list is 
 variadic -
 template bloat in such cases is usually eliminated by 
 forwarding to
 another private method immediately turning file/line into 
 first runtime
 argument instead.


 Would it be a bad idea to allow this in the compiler:

 void foo(Args...)(Args args, string file = __FILE__, size_t 
 line = __LINE__);

 It wouldn't be possible to pass "file" or "line" when calling 
 "foo". But it's useful for the special default values, __FILE__ 
 and __LINE__.


I think it would be good idea to take this even further:

T4 foo(T4, T0, T1, Ts..., T2, T3)(T0 t0, T1 t1, Args args, T2 t2, 
T3 t3)

In other words, I think that the limitation that variadic 
template parameter list must be at the end of the function 
parameters is arbitrary and just a deficiency of the current 
implementation.
A fixed number of parameters proceeding or following a variadic 
list should all work equally well, even in combination with IFTI.

BTW, Ruby also allows to define methods with such parameters: 
foo(first_arg, *middle_arguments, last_arg)
Jun 20 2016
next sibling parent ZombineDev <petar.p.kirov gmail.com> writes:
On Tuesday, 21 June 2016 at 02:59:44 UTC, ZombineDev wrote:

 On Monday, 20 June 2016 at 14:28:06 UTC, Jacob Carlborg wrote:

 On 2016-06-19 12:43, Dicebot wrote:


 Yes. It is necessary because runtime parameter list is 
 variadic -
 template bloat in such cases is usually eliminated by 
 forwarding to
 another private method immediately turning file/line into 
 first runtime
 argument instead.


 Would it be a bad idea to allow this in the compiler:

 void foo(Args...)(Args args, string file = __FILE__, size_t 
 line = __LINE__);

 It wouldn't be possible to pass "file" or "line" when calling 
 "foo". But it's useful for the special default values, 
 __FILE__ and __LINE__.


 I think it would be good idea to take this even further:

 T4 foo(T4, T0, T1, Ts..., T2, T3)(T0 t0, T1 t1, Args args, T2 
 t2, T3 t3)

 In other words, I think that the limitation that variadic 
 template parameter list must be at the end of the function 
 parameters is arbitrary and just a deficiency of the current 
 implementation.
 A fixed number of parameters proceeding or following a variadic 
 list should all work equally well, even in combination with 
 IFTI.

 BTW, Ruby also allows to define methods with such parameters: 
 foo(first_arg, *middle_arguments, last_arg)


I meant:

T4 foo(T4, T0, T1, Ts..., T2, T3)(T0 t0, T1 t1, Ts ts, T2 t2, T3 
t3)
Jun 20 2016
prev sibling parent reply pineapple <meapineapple gmail.com> writes:
On Tuesday, 21 June 2016 at 02:59:44 UTC, ZombineDev wrote:

 I think it would be good idea to take this even further:

 T4 foo(T4, T0, T1, Ts..., T2, T3)(T0 t0, T1 t1, Args args, T2 
 t2, T3 t3)

 In other words, I think that the limitation that variadic 
 template parameter list must be at the end of the function 
 parameters is arbitrary and just a deficiency of the current 
 implementation.


I don't disagree with you, but this is a separate issue. If the 
arguments trailing Args have default values (as they would in the 
case of __FILE__, __LINE__) it will create ambiguity.

void foo(Args...)(Args args, int n = 1);

What happens if I do this? foo(1, 2, 3, 4); Is n 1 or 4?
Jun 21 2016
parent reply pineapple <meapineapple gmail.com> writes:
On Tuesday, 21 June 2016 at 10:28:03 UTC, pineapple wrote:

 On Tuesday, 21 June 2016 at 02:59:44 UTC, ZombineDev wrote:

 I think it would be good idea to take this even further:

 T4 foo(T4, T0, T1, Ts..., T2, T3)(T0 t0, T1 t1, Args args, T2 
 t2, T3 t3)

 In other words, I think that the limitation that variadic 
 template parameter list must be at the end of the function 
 parameters is arbitrary and just a deficiency of the current 
 implementation.


 I don't disagree with you, but this is a separate issue. If the 
 arguments trailing Args have default values (as they would in 
 the case of __FILE__, __LINE__) it will create ambiguity.

 void foo(Args...)(Args args, int n = 1);

 What happens if I do this? foo(1, 2, 3, 4); Is n 1 or 4?


Another thought: What if in this case the trailing arguments are 
always the default value? In this case n is /always/ 1, file and 
line would always be __FILE__ and __LINE__. This would be a more 
general solution and maybe not too intolerably ugly.
Jun 21 2016
parent ZombineDev <petar.p.kirov gmail.com> writes:
On Tuesday, 21 June 2016 at 10:34:01 UTC, pineapple wrote:

 On Tuesday, 21 June 2016 at 10:28:03 UTC, pineapple wrote:

 On Tuesday, 21 June 2016 at 02:59:44 UTC, ZombineDev wrote:

 I think it would be good idea to take this even further:

 T4 foo(T4, T0, T1, Ts..., T2, T3)(T0 t0, T1 t1, Args args, T2 
 t2, T3 t3)

 In other words, I think that the limitation that variadic 
 template parameter list must be at the end of the function 
 parameters is arbitrary and just a deficiency of the current 
 implementation.


 I don't disagree with you, but this is a separate issue. If 
 the arguments trailing Args have default values (as they would 
 in the case of __FILE__, __LINE__) it will create ambiguity.

 void foo(Args...)(Args args, int n = 1);

 What happens if I do this? foo(1, 2, 3, 4); Is n 1 or 4?


 Another thought: What if in this case the trailing arguments 
 are always the default value? In this case n is /always/ 1, 
 file and line would always be __FILE__ and __LINE__. This would 
 be a more general solution and maybe not too intolerably ugly.


It should work just like the regular expression .*.? - if all 
parameters after the variadic list have default values, then the 
list should capture all arguments and the last parameters use 
their default values. E.g. args is (1, 2, 3, 4) and n is 1 in 
your example.
In general the strategy should be: the variadic list captures as 
much arguments as its constraints allow it to capture.

// regex: (\d*)(.)(.?)
// (pretend that \d means scalar type)
void foo(T1, T2, Args...)(Args args, T1 t1, T2 t2 = "default"w)
     if (allSatisfy!(isScalarType, Args))
{
     pragma (msg, Args, ", ", T1, ", ", T2);
}

foo(0, 0L, "asd");       // (int, long), string, wstring
foo("asd", "sdf");       // (), string, string
foo('c', new Object, 3); // (char), Object, int
Jun 21 2016
prev sibling parent reply Dicebot <public dicebot.lv> writes:
On 06/19/2016 11:33 AM, Johan Engelen wrote:

 On Sunday, 19 June 2016 at 08:06:09 UTC, Dicebot wrote:

 This important feature and can't be simply removed. For example,
 std.experimental.logger also relies on it.


 
 Do you mean it relies on __FILE__ being a template parameter (instead of
 a runtime default parameter, like enforce)?
 

 It needs to be fixed instead.

 Two immediate workarounds that come to mmy mind:

 1. Make __FILE__ relative to import path
 2. Always emit new symbol to object file if __FILE__ is involved


 
 I also thought about 2. It would fix the second (non-Phobos) example.
 
 -Johan


Another option would be to treat functions relying on __FILE__ as
template argument as force inline ones.
Jun 19 2016
parent reply Johan Engelen <j j.nl> writes:
On Sunday, 19 June 2016 at 20:37:02 UTC, Dicebot wrote:

 On 06/19/2016 11:33 AM, Johan Engelen wrote:

 On Sunday, 19 June 2016 at 08:06:09 UTC, Dicebot wrote:

 This important feature and can't be simply removed. For 
 example, std.experimental.logger also relies on it.


 
 Do you mean it relies on __FILE__ being a template parameter 
 (instead of a runtime default parameter, like enforce)?
 

 It needs to be fixed instead.

 Two immediate workarounds that come to mmy mind:

 1. Make __FILE__ relative to import path
 2. Always emit new symbol to object file if __FILE__ is 
 involved


 
 I also thought about 2. It would fix the second (non-Phobos) 
 example.
 
 -Johan


 Another option would be to treat functions relying on __FILE__ 
 as template argument as force inline ones.


Yes, but it's a flaky solution I think.
For LDC that would work (I think we can pretty much inline 
anything the user throws at us). I don't know about DMD.
Jun 19 2016
parent reply Johan Engelen <j j.nl> writes:
On Sunday, 19 June 2016 at 21:11:59 UTC, Johan Engelen wrote:

 (I think we can pretty much inline anything the user throws at 
 us)


(as long as it is not a naked asm function)
Jun 19 2016
parent reply David Nadlinger <code klickverbot.at> writes:
On Sunday, 19 June 2016 at 21:13:00 UTC, Johan Engelen wrote:

 On Sunday, 19 June 2016 at 21:11:59 UTC, Johan Engelen wrote:

 (I think we can pretty much inline anything the user throws at 
 us)


 (as long as it is not a naked asm function)


Another example is `alloca`, which you might not want to inline.

  — David
Jun 19 2016
parent reply Johan Engelen <j j.nl> writes:
On Sunday, 19 June 2016 at 21:40:20 UTC, David Nadlinger wrote:

 On Sunday, 19 June 2016 at 21:13:00 UTC, Johan Engelen wrote:

 On Sunday, 19 June 2016 at 21:11:59 UTC, Johan Engelen wrote:

 (I think we can pretty much inline anything the user throws 
 at us)


 (as long as it is not a naked asm function)


 Another example is `alloca`, which you might not want to inline.


And also variadic functions, for which Dicebot needs __FILE__ to 
be a template param..............
Jun 19 2016
parent reply Dicebot <public dicebot.lv> writes:
On 06/20/2016 01:13 AM, Johan Engelen wrote:

 On Sunday, 19 June 2016 at 21:40:20 UTC, David Nadlinger wrote:

 On Sunday, 19 June 2016 at 21:13:00 UTC, Johan Engelen wrote:

 On Sunday, 19 June 2016 at 21:11:59 UTC, Johan Engelen wrote:

 (I think we can pretty much inline anything the user throws at us)


 (as long as it is not a naked asm function)


 Another example is `alloca`, which you might not want to inline.


 
 And also variadic functions, for which Dicebot needs __FILE__ to be a
 template param..............


It is template-based variadic list though in all Phobos cases. This is
one of logger examples I was referring to :
https://github.com/dlang/phobos/blob/master/std/experimental/logger/core.d#L202-L216

How about defining semantics like "try inlining if possible, fallback to
always emitting symbol to object file otherwise"? That would also allow
compatible implementation in dmd.

My reasoning for proposing that is that for all legitimate cases I'd
have to use __FILE__ as a template argument I'd also want the symbol
inlined to prevent the bloat. After all CT computation such functions
tend to contain 1-2 lines of runtime code only.
Jun 20 2016
next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 20 June 2016 at 08:10:19 UTC, Dicebot wrote:

 How about defining semantics like "try inlining if possible, 
 fallback to always emitting symbol to object file otherwise"? 
 That would also allow compatible implementation in dmd.


This would get rid of the undefined symbols, but there is a much 
more subtle problem here that nobody has brought up yet: If the 
template ends up being emitted twice with different mangled 
names, then, well, it ends up existing twice in the final 
executable.

This is not really an issue for functions, but very much so for 
data symbols (e.g. a static variable inside a function), where 
you could end up with the same alias referring to two different 
pieces of data depending on where it is used from.

The root of this issue is that __FILE__ introduces incidental 
environmental state into the otherwise pure module system.

  — David
Jun 21 2016
parent Johan Engelen <j j.nl> writes:
On Tuesday, 21 June 2016 at 10:39:52 UTC, David Nadlinger wrote:

 On Monday, 20 June 2016 at 08:10:19 UTC, Dicebot wrote:

 How about defining semantics like "try inlining if possible, 
 fallback to always emitting symbol to object file otherwise"? 
 That would also allow compatible implementation in dmd.


 This would get rid of the undefined symbols, but there is a 
 much more subtle problem here that nobody has brought up yet: 
 If the template ends up being emitted twice with different 
 mangled names, then, well, it ends up existing twice in the 
 final executable.

 This is not really an issue for functions, but very much so for 
 data symbols (e.g. a static variable inside a function), where 
 you could end up with the same alias referring to two different 
 pieces of data depending on where it is used from.

 The root of this issue is that __FILE__ introduces incidental 
 environmental state into the otherwise pure module system.


+1
Jun 22 2016
prev sibling parent Johan Engelen <j j.nl> writes:
On Monday, 20 June 2016 at 08:10:19 UTC, Dicebot wrote:

 How about defining semantics like "try inlining if possible, 
 fallback to always emitting symbol to object file otherwise"? 
 That would also allow compatible implementation in dmd.

 My reasoning for proposing that is that for all legitimate 
 cases I'd have to use __FILE__ as a template argument I'd also 
 want the symbol inlined to prevent the bloat. After all CT 
 computation such functions tend to contain 1-2 lines of runtime 
 code only.


In LDC, Phobos functions with __FILE__ as default template 
parameter value are now marked with `pragma(inline, true)`, as a 
workaround for the larger issue of what to do with __FILE__ as 
template parameter.
https://github.com/ldc-developers/phobos/commit/8b2052cffe159be3db7debbe28ee967630cb4588

(the issue was preventing cross-module inlining, so a workaround 
was needed)
Jul 05 2016