digitalmars.D - nothrow function callbacks in extern(C) code - solution
- Walter Bright (28/28) Jun 19 2014 With nothrow and @nogc annotations, we've been motivated to add these
- H. S. Teoh via Digitalmars-d (51/92) Jun 19 2014 This is a clever hack to work around the type system, but it introduces
- Dicebot (3/3) Jun 19 2014 +1
- w0rp (7/11) Jun 19 2014 I think the most common function this kind of thing could be
- H. S. Teoh via Digitalmars-d (10/21) Jun 19 2014 Yes I mention opApply in my post. ;-)
- anonymous (27/30) Jun 19 2014 I don't know how practical this is, but since attributes are
- Timon Gehr (21/24) Jun 19 2014 I have furthermore always wondered why there can always only be one
- H. S. Teoh via Digitalmars-d (25/50) Jun 19 2014 [...]
- Mason McGill (33/39) Jun 22 2014 Here's one idea:
- bearophile (9/11) Jun 22 2014 I think problems come from refusing to have formalized features
- Dmitry Olshansky (7/16) Jun 22 2014 +1
- Artur Skawina via Digitalmars-d (16/37) Jun 20 2014 Simple propagation, from just one source, would be enough for almost all
- Rainer Schuetze (13/43) Jun 20 2014 This only works for those functions that call the callback function
- Andrei Alexandrescu (4/7) Jun 21 2014 Callbacks passed into OS/clib functions are never supposed to throw so
- w0rp (4/14) Jun 22 2014 Yeah. I think the signature should communicate, "Woah, hold on
- Rainer Schuetze (18/25) Jun 22 2014 For nothrow that restriction makes sense, though there are functions in
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (3/16) Jun 23 2014 The SEGV handles throws Errors, not Exceptions, it's ok to be
-
Paolo Invernizzi
(13/18)
Jun 20 2014
- w0rp (9/27) Jun 20 2014 This is actually a really good point. How can a callback in C
- Marco Leise (16/27) Jul 08 2014 I just stumbled upon this thread now...
- "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= (3/5) Jul 08 2014 Yes, longjmp() could happen in a callback and may be expected to
With nothrow and nogc annotations, we've been motivated to add these annotations to C system API functions, because obviously such functions aren't going to throw D exceptions or call the D garbage collector. But this exposed a problem - functions like C's qsort() take a pointer to a callback function. The callback function, being supplied by the D programmer, may throw and may call the garbage collector. By requiring the callback function to be also nothrow nogc, this is an unreasonable requirement besides breaking most existing D code that uses qsort(). This problem applies as well to the Windows APIs and the Posix APIs with callbacks. The solution is to use overloading so that if your callback is nothrow, it will call the nothrow version of qsort, if it is throwable, it calls the throwable version of qsort. Never mind that those two versions of qsort are actually the same function (!), even though D's type system regards them as different. Although this looks like an usafe hack, it actually is quite safe, presuming that the rest of the qsort code itself does not throw. This technique relies on the fact that extern(C) functions do not get their types mangled into the names. Some example code: extern (C) { alias int function() fp_t; } extern (C) nothrow { alias int function() fpnothrow_t; } extern (C) int foo(int a, fp_t fp); extern (C) nothrow int foo(int a, fpnothrow_t fp); extern (C) int bar(); extern (C) nothrow int barnothrow(); void test() { foo(1, &bar); // calls the 'throwing' foo() foo(1, &barnothrow); // calls the 'nothrow' foo() }
Jun 19 2014
On Thu, Jun 19, 2014 at 12:59:00PM -0700, Walter Bright via Digitalmars-d wrote:With nothrow and nogc annotations, we've been motivated to add these annotations to C system API functions, because obviously such functions aren't going to throw D exceptions or call the D garbage collector. But this exposed a problem - functions like C's qsort() take a pointer to a callback function. The callback function, being supplied by the D programmer, may throw and may call the garbage collector. By requiring the callback function to be also nothrow nogc, this is an unreasonable requirement besides breaking most existing D code that uses qsort(). This problem applies as well to the Windows APIs and the Posix APIs with callbacks. The solution is to use overloading so that if your callback is nothrow, it will call the nothrow version of qsort, if it is throwable, it calls the throwable version of qsort. Never mind that those two versions of qsort are actually the same function (!), even though D's type system regards them as different. Although this looks like an usafe hack, it actually is quite safe, presuming that the rest of the qsort code itself does not throw. This technique relies on the fact that extern(C) functions do not get their types mangled into the names. Some example code: extern (C) { alias int function() fp_t; } extern (C) nothrow { alias int function() fpnothrow_t; } extern (C) int foo(int a, fp_t fp); extern (C) nothrow int foo(int a, fpnothrow_t fp); extern (C) int bar(); extern (C) nothrow int barnothrow(); void test() { foo(1, &bar); // calls the 'throwing' foo() foo(1, &barnothrow); // calls the 'nothrow' foo() }This is a clever hack to work around the type system, but it introduces boilerplate, and doesn't ultimately fix the underlying problem. I agree that it's "good enough" for now -- to get those system APIs working without massive breakage of existing code -- but I think for the long run, we should face the root issue: functions that call callbacks have attributes that *depend* on an input argument. So we really should have the equivalent of "inout" for other attributes than const. For example, a function can be "dependently nothrow", meaning that the body of the function is nothrow, except for that call to user-supplied delegate: // This is hypothetical syntax. int dgCaller(scope int delegate(int) dg inout(nothrow)) inout(nothrow) { //throw new Exception(...); // illegal: body of function must not throw int result = dg(1); // OK: we inherit nothrow-ness from dg(). return result; } void f1() nothrow { // OK: the delegate is nothrow, so dgCaller is nothrow, // so it's permitted to call it from a nothrow function. auto x = dgCaller((x) => x+1); } void f2() { // OK: the delegate is throwing, which makes dgCaller // nothrowing w.r.t. this call. So this function becomes // throwing. auto x = dgCaller((int) { throw new Exception(...); }); } void f3() nothrow { // ILLEGAL: the delegate throws, so dgCaller may throw, // so we can't call it from a nothrow function. auto x = dgCaller((int) { throw new Exception(...); }); } This capability will solve a lot of attribute-related issues especially in generic code. For example, a container that implements opApply() currently cannot be marked pure, because it would impose purity on the delegate passed to it, which greatly limits its applicability. But accepting a non-pure delegate makes it uncallable from pure code, even if opApply() itself doesn't do anything impure. So we end up needing to write two identical versions of opApply: one pure, and one non-pure. And this has to be multiplied for each attribute we wish to support (pure, nothrow, safe, nogc -- that's 2^4 = 16 copies of the same function), which is clearly untenable. With inout(nothrow), inout(pure), etc., we can collapse all of these variants into a single function, and at the same time have the compiler statically verify that the result does not violate any attribute. T -- "I'm not childish; I'm just in touch with the child within!" - RL
Jun 19 2014
+1 I have always wondered why `inout` is limited to const when problem is almost identical with all other restrictive attributes.
Jun 19 2014
On Thursday, 19 June 2014 at 20:29:42 UTC, Dicebot wrote:+1 I have always wondered why `inout` is limited to const when problem is almost identical with all other restrictive attributes.I think the most common function this kind of thing could be useful for would be opApply functions. I haven't yet figured out a good way to make opApply implementations get all of the nice qualifiers without writing a bunch of overloads. Of course, in my own code I often just enforce the qualifiers to the exclusion of code without them, but that's no good for a standard library.
Jun 19 2014
On Thu, Jun 19, 2014 at 10:22:32PM +0000, w0rp via Digitalmars-d wrote:On Thursday, 19 June 2014 at 20:29:42 UTC, Dicebot wrote:Yes I mention opApply in my post. ;-) This isn't the only issue with opApply, though. The other issue is that you can't (easily) control the ref-ness of the loop index. And another the (lack of) inlining of loop bodies when opApply is involved. This makes them slightly less attractive in performance-sensitive situations, which is unfortunate. T -- It is impossible to make anything foolproof because fools are so ingenious. -- Sammy+1 I have always wondered why `inout` is limited to const when problem is almost identical with all other restrictive attributes.I think the most common function this kind of thing could be useful for would be opApply functions. I haven't yet figured out a good way to make opApply implementations get all of the nice qualifiers without writing a bunch of overloads. Of course, in my own code I often just enforce the qualifiers to the exclusion of code without them, but that's no good for a standard library.
Jun 19 2014
On Thursday, 19 June 2014 at 22:22:33 UTC, w0rp wrote:I haven't yet figured out a good way to make opApply implementations get all of the nice qualifiers without writing a bunch of overloads.I don't know how practical this is, but since attributes are inferred for templated methods ... struct S { int opApply(Dg : int delegate(ref int))(Dg dg) { int e; return dg(e); } } void f1() pure nothrow nogc safe { S s; foreach(int x; s) {} } void f2() { S s; import std.stdio; foreach(int x; s) writeln(x); } void main() { f1(); f2(); }
Jun 19 2014
On 06/19/2014 10:29 PM, Dicebot wrote:+1 I have always wondered why `inout` is limited to const when problem is almost identical with all other restrictive attributes.I have furthermore always wondered why there can always only be one `inout' wildcard in scope. This is not the best existing way to solve this kind of problem: Parametric (i.e. not query-able at either runtime or compile time inside the function) compile-time arguments do it better. I.e. instead of: inout(int)[] foo(inout(int)[] arg){ return arg; } do: T foo![T <: const(int)[]](T arg){ return arg; } this can be extended to other attributes, for example in the following way (this is just an example): void evaluate![transitive_attributes a](void delegate() a dg) a{ dg(); } {void foo() safe{} evaluate(&foo);} // evaluate is safe {void foo()pure{} evaluate(&foo);} // evaluate is pure {void foo()pure safe{} evaluate(&foo); // evaluate is pure safe evaluate![pure](&foo);} // argument explicitly specified, system pure
Jun 19 2014
On Fri, Jun 20, 2014 at 12:36:53AM +0200, Timon Gehr via Digitalmars-d wrote:On 06/19/2014 10:29 PM, Dicebot wrote:[...] What if there are multiple delegate arguments? void evalTwo(void delegate() a dg1, void delegate() b dg2) // how to express union of a and b? { dg1(); dg2(); } What if the delegate arguments themselves take delegate arguments? void eval(void delegate(void delegate() a) b dg1, void delegate() c dg2) d // how to express that b depends on a, and // d depends on b and c? { dg1(dg2); } Pretty soon, we need an attribute algebra to express these complicated relationships. It would be nice to have a solution that can handle all of these cases without exploding complexity in the syntax. T -- "The number you have dialed is imaginary. Please rotate your phone 90 degrees and try again."+1 I have always wondered why `inout` is limited to const when problem is almost identical with all other restrictive attributes.I have furthermore always wondered why there can always only be one `inout' wildcard in scope. This is not the best existing way to solve this kind of problem: Parametric (i.e. not query-able at either runtime or compile time inside the function) compile-time arguments do it better. I.e. instead of: inout(int)[] foo(inout(int)[] arg){ return arg; } do: T foo![T <: const(int)[]](T arg){ return arg; } this can be extended to other attributes, for example in the following way (this is just an example): void evaluate![transitive_attributes a](void delegate() a dg) a{ dg(); }
Jun 19 2014
On Thursday, 19 June 2014 at 23:41:25 UTC, H. S. Teoh via Digitalmars-d wrote:Pretty soon, we need an attribute algebra to express these complicated relationships. It would be nice to have a solution that can handle all of these cases without exploding complexity in the syntax.Here's one idea: Attributes can be thought of as "declarative" compile-time parameters--they don't directly define a computation, but they feel like compile-time parameters in that - They are variable in number. - They may take on many possible values. - They may be parameterized (like `extern`). - They play a role in overload resolution. - Their combinations may lead to combinatorial explosions that can hopefully be mitigated with metaprogramming. My point is, many of the problems we're encountering with attributes have already been solved (via templates, tuples, CTFE, `static if`, etc.) Attribute propagation would be simple (not trivial, but appropriately simple) for user-defined attributes, because they can be arbitrary expressions. If built-in attributes were just treated as pre-defined symbols, they could be manipulated with templates and CTFE: enum mayThrow(alias func) = staticIndexOf!(nothrow, functionAttributes!func) == -1; template throwsLike(alias func) { static if (mayThrow!func) enum throwsLike = tuple().expand; else enum throwsLike = nothrow; } void call(alias func)() throwsLike!func { func(); } (This thread may be relevant: http://forum.dlang.org/thread/hfmulninvghjntqkpguk forum.dlang.org)
Jun 22 2014
H. S. Teoh:Pretty soon, we need an attribute algebra to express these complicated relationships.I think problems come from refusing to have formalized features in D. Having an attribute algebra is the lesser problem. Having formalized features causes problems, but no having them formalized is worse, as visible with D safety, D pureness, D uniqueness, that are a growing mess. D design needs more mathematicians and less piling of patches on patches. Bye, bearophile
Jun 22 2014
22-Jun-2014 14:39, bearophile пишет:H. S. Teoh:+1 I often find myself having second thoughts about all of the 'implicit' rules esp. with regard to value-range propagation, and half-formalized such as inout. -- Dmitry OlshanskyPretty soon, we need an attribute algebra to express these complicated relationships.I think problems come from refusing to have formalized features in D. Having an attribute algebra is the lesser problem. Having formalized features causes problems, but no having them formalized is worse, as visible with D safety, D pureness, D uniqueness, that are a growing mess. D design needs more mathematicians and less piling of patches on patches.
Jun 22 2014
On 06/20/14 01:39, H. S. Teoh via Digitalmars-d wrote:On Fri, Jun 20, 2014 at 12:36:53AM +0200, Timon Gehr via Digitalmars-d wrote:On 06/19/2014 10:29 PM, Dicebot wrote:I have always wondered why `inout` is limited to const when problem is almost identical with all other restrictive attributes.I have furthermore always wondered why there can always only be one `inout' wildcard in scope. This is not the best existing way to solveT foo![T <: const(int)[]](T arg){ return arg; } this can be extended to other attributes, for example in the following way (this is just an example): void evaluate![transitive_attributes a](void delegate() a dg) a{ dg(); }What if there are multiple delegate arguments?What if the delegate arguments themselves take delegate arguments?Pretty soon, we need an attribute algebra to express these complicated relationships.Simple propagation, from just one source, would be enough for almost all cases - there's no need to over-complicate this. The remaining cases could be handled via introspection and ctfe; this way allows for more options, not just using some pre-defined algebra subset, which happens to be supported by a particular compiler (-version).It would be nice to have a solution that can handle all of these cases without exploding complexity in the syntax.Actually supporting parametrized attributes, is something that I think everybody agrees on in principle (hence the lack of discussions when this topic is mentioned, every few weeks or so). The required semantics are pretty clear; what I still haven't seen is a good enough syntax proposal. One syntax that might have worked for built-in attributes could have been "const!A" etc, but I'm not sure if the parameter inference would be intuitive enough, and it would appear, at least superficially, to potentially clash with user defined attributes, especially once those become more powerful. artur
Jun 20 2014
On 19.06.2014 21:59, Walter Bright wrote:With nothrow and nogc annotations, we've been motivated to add these annotations to C system API functions, because obviously such functions aren't going to throw D exceptions or call the D garbage collector. But this exposed a problem - functions like C's qsort() take a pointer to a callback function. The callback function, being supplied by the D programmer, may throw and may call the garbage collector. By requiring the callback function to be also nothrow nogc, this is an unreasonable requirement besides breaking most existing D code that uses qsort(). This problem applies as well to the Windows APIs and the Posix APIs with callbacks. The solution is to use overloading so that if your callback is nothrow, it will call the nothrow version of qsort, if it is throwable, it calls the throwable version of qsort. Never mind that those two versions of qsort are actually the same function (!), even though D's type system regards them as different. Although this looks like an usafe hack, it actually is quite safe, presuming that the rest of the qsort code itself does not throw. This technique relies on the fact that extern(C) functions do not get their types mangled into the names. Some example code: extern (C) { alias int function() fp_t; } extern (C) nothrow { alias int function() fpnothrow_t; } extern (C) int foo(int a, fp_t fp); extern (C) nothrow int foo(int a, fpnothrow_t fp); extern (C) int bar(); extern (C) nothrow int barnothrow(); void test() { foo(1, &bar); // calls the 'throwing' foo() foo(1, &barnothrow); // calls the 'nothrow' foo() }This only works for those functions that call the callback function directly. OS function do not always work this way. They register callbacks for later use like a windows procedure or a signal handler. This causes innocent looking functions to not behave as annotated because they internally use the callback functions. E.g. a lot of the Windows API functions might use message sending/dispatching internally, which might execute both throwing or GC allocating callbacks. These are currently not meeting the promise of their annotations. We either have to be more conservative with annotating OS functions or relax the guarantees of nothrow or nogc. Both alternatives are not very compelling.
Jun 20 2014
On 6/20/14, 3:14 AM, Rainer Schuetze wrote:We either have to be more conservative with annotating OS functions or relax the guarantees of nothrow or nogc. Both alternatives are not very compelling.Callbacks passed into OS/clib functions are never supposed to throw so we must annotate them all with nothrow. C functions are never designed under the assumption callbacks may throw. -- Andrei
Jun 21 2014
On Sunday, 22 June 2014 at 00:23:06 UTC, Andrei Alexandrescu wrote:On 6/20/14, 3:14 AM, Rainer Schuetze wrote:Yeah. I think the signature should communicate, "Woah, hold on there. That isn't going to work."We either have to be more conservative with annotating OS functions or relax the guarantees of nothrow or nogc. Both alternatives are not very compelling.Callbacks passed into OS/clib functions are never supposed to throw so we must annotate them all with nothrow. C functions are never designed under the assumption callbacks may throw. -- Andrei
Jun 22 2014
On 22.06.2014 02:23, Andrei Alexandrescu wrote:On 6/20/14, 3:14 AM, Rainer Schuetze wrote:For nothrow that restriction makes sense, though there are functions in the Windows "C API" that actually may throw, e.g. HeapAlloc [1] and MmProbeAndLockPages [2]. No callbacks are involved in these examples, though. Under unix, signal handlers that throw exceptions seem to be used, though this might be non-standard. We even have one in druntime [3]. I'm ok with the current treatment of nothrow, but what can we do about nogc? [1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa366597%28v=vs.85%29.aspx [2] http://msdn.microsoft.com/en-us/library/windows/hardware/ff554664%28v=vs.85%29.aspx [3] https://github.com/D-Programming-Language/druntime/blob/master/src/etc/linux/memoryerror.d BTW: The D exception handling is compatible with SEH for Win32, but not for Win64. You cannot handle exceptions by type across languages, though, only unwinding and "catch(Throwable)" works here.We either have to be more conservative with annotating OS functions or relax the guarantees of nothrow or nogc. Both alternatives are not very compelling.Callbacks passed into OS/clib functions are never supposed to throw so we must annotate them all with nothrow. C functions are never designed under the assumption callbacks may throw. -- Andrei
Jun 22 2014
On Sunday, 22 June 2014 at 08:43:20 UTC, Rainer Schuetze wrote:On 22.06.2014 02:23, Andrei Alexandrescu wrote:The SEGV handles throws Errors, not Exceptions, it's ok to be nothrow.Callbacks passed into OS/clib functions are never supposed to throw so we must annotate them all with nothrow. C functions are never designed under the assumption callbacks may throw. -- AndreiFor nothrow that restriction makes sense, though there are functions in the Windows "C API" that actually may throw, e.g. HeapAlloc [1] and MmProbeAndLockPages [2]. No callbacks are involved in these examples, though. Under unix, signal handlers that throw exceptions seem to be used, though this might be non-standard. We even have one in druntime [3].
Jun 23 2014
On Thursday, 19 June 2014 at 19:58:58 UTC, Walter Bright wrote: <snip>The callback function, being supplied by the D programmer, may throw and may call the garbage collector. By requiring the callback function to be also nothrow nogc, this is an unreasonable requirement besides breaking most existing D code that uses qsort().<d.learn> I'm missing something, as I'm annotating all my C/API/etc callback function with nothrow: when the callback throws, what happens? I was thinking that this will mess-up the stack once the unwind will proceed... What's the use-case for having such a callback 'throwable'? Thanks! </d.learn> --- Paolo
Jun 20 2014
On Friday, 20 June 2014 at 11:07:48 UTC, Paolo Invernizzi wrote:On Thursday, 19 June 2014 at 19:58:58 UTC, Walter Bright wrote: <snip>This is actually a really good point. How can a callback in C code expect to throw exceptions? Surely it should be nothrow anyway, because it's just not going to work otherwise. Maybe we should just strengthen the constraints for that, and make people update their code which isn't likely to work anyway. You can make any throwing function nothrow by catching Exceptions and throwing Errors instead at least. Ideally you wouldn't even throw Errors in C callbacks.The callback function, being supplied by the D programmer, may throw and may call the garbage collector. By requiring the callback function to be also nothrow nogc, this is an unreasonable requirement besides breaking most existing D code that uses qsort().<d.learn> I'm missing something, as I'm annotating all my C/API/etc callback function with nothrow: when the callback throws, what happens? I was thinking that this will mess-up the stack once the unwind will proceed... What's the use-case for having such a callback 'throwable'? Thanks! </d.learn> --- Paolo
Jun 20 2014
Am Thu, 19 Jun 2014 12:59:00 -0700 schrieb Walter Bright <newshound2 digitalmars.com>:With nothrow and nogc annotations, we've been motivated to add these annotations to C system API functions, because obviously such functions aren't going to throw D exceptions or call the D garbage collector. But this exposed a problem - functions like C's qsort() take a pointer to a callback function. The callback function, being supplied by the D programmer, may throw and may call the garbage collector. By requiring the callback function to be also nothrow nogc, this is an unreasonable requirement besides breaking most existing D code that uses qsort(). This problem applies as well to the Windows APIs and the Posix APIs with callbacks.I just stumbled upon this thread now... In general you cannot throw exceptions unless you know how the call stack outside of your language's barrier is constructed. What I mean is that while DMD on 64-bit Linux uses frame pointers that druntime uses to unwind, GCC omits them. So druntime bails out once it reaches the C part of the call stack. That makes for two options: 1) D callbacks passed to other languages must not throw, like Andrei proposes if I understood that right. 2) Druntime must adapt to the system's C compiler's ABI. (by the use of libunwind) -- Marco
Jul 08 2014
On Tuesday, 8 July 2014 at 17:16:27 UTC, Marco Leise wrote:2) Druntime must adapt to the system's C compiler's ABI. (by the use of libunwind)Yes, longjmp() could happen in a callback and may be expected to unwind the stack (depending on the dev environment).
Jul 08 2014