digitalmars.dip.ideas - Detached unit tests
- Steven Schveighoffer (53/53) Apr 30 I want to propose that we provide a mechanism to semantically
- Jonathan M Davis (26/34) Apr 30 It would be valuable for ddoc-ed unittest blocks, because it would allow...
- Richard (Rikki) Andrew Cattermole (10/48) Apr 30 Static in the context of a type like a class implies no this pointer, a
- Timon Gehr (3/16) May 01 Looks useful, as usually you want to test the API. Maybe this should
- Dom DiSc (5/10) May 01 No, white-box test is correct as default, as black-box tests
- Timon Gehr (13/25) May 01 Ddoc unittests have to be in a very specific place though and they are
- Steven Schveighoffer (12/15) May 01 Hm... `public unittest` syntax?
- Jonathan M Davis (30/44) May 01 I would expect it to be problematic for public or private attributes whi...
- Quirin Schroll (6/12) May 16 I suggested something like this before and used the syntax
I want to propose that we provide a mechanism to semantically
detach unit tests from the scope they are written in. Such
unittests would be treated as if they were not in the same scope
or module. This includes detached unit tests inside templates.
The only stipulation is that the current module is assumed
imported.
For a strawman syntax, let's do what everyone does ... overload
`static`!
```d
import std.stdio;
// static unittests help validate API does not include private
functions
struct S
{
private int x;
unittest {
auto s = S(42);
s.foo(); // ok, this is not a detached unittest
}
static unittest {
auto s = S(23);
//s.x == 23; // error, x not visible
}
}
private void foo(S s) {
assert(s.x == 42);
}
static unittest {
auto s = S(42);
//s.foo(); // error, foo not visible.
}
// static unittests help with documentation on templated types
struct Arr(T) {
private T[] arr;
void append(T val) {
arr ~= val;
}
///
static unittest {
// Arr arr; // Error, you are not in this scope, so Arr is
an uninstantiated template
Arr!int arr; // ok
arr.append(1);
import std.stdio; // must re-import things as if you weren't
in this module
// writeln(arr.arr); // nope, this is a private member
writeln(arr.getArray()); // must use public API
}
T[] getArray() => arr;
}
```
thoughts?
-Steve
Apr 30
On Tuesday, April 30, 2024 11:02:51 AM MDT Steven Schveighoffer via dip.ideas wrote:I want to propose that we provide a mechanism to semantically detach unit tests from the scope they are written in. Such unittests would be treated as if they were not in the same scope or module. This includes detached unit tests inside templates. The only stipulation is that the current module is assumed imported. For a strawman syntax, let's do what everyone does ... overload `static`!It would be valuable for ddoc-ed unittest blocks, because it would allow us to guarantee that they don't use any part of the API which isn't public. Phobos currently has to do that by running a separate program to pull out all of the tests and make sure that they're runnable outside of the module, which is a lot more complicated and doesn't help anyone else. As for the syntax, static is probably fine so long as static: is ignored. I don't recall at the moment how much that affects, but it's terrible practice to use static that way anyway given how little sense it makes to try to apply static to multiple symbols at once. When I'd thought about this the other day, I'd thought that maybe we could use a pragma for it, but I don't really care much about the syntax, and static seems fine. It fits in with the basic meaning of how static is normally used. Part of me thinks that we should just make ddoc-ed unit tests do this without additional syntax, but that would undoubtedly break existing code, since _someone_ will have a documented test somewhere that accidentally uses a private member, and it might also help with templates to be able to do this with undocumented unittest blocks (though I vaguely recall seeing something about how the behavior on those was changed at some point so that they're already not part of the template any longer, though I'm not sure on that, since I haven't written any templated types with tests inside them recently). - Jonathan M Davis
Apr 30
"Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> On 01/05/2024 5:29 AM, Jonathan M Davis wrote:On Tuesday, April 30, 2024 11:02:51 AM MDT Steven Schveighoffer via dip.ideas wrote:Static in the context of a type like a class implies no this pointer, a unittest should never have one to begin with, therefore it doesn't fit. For this reason I do not like it.I want to propose that we provide a mechanism to semantically detach unit tests from the scope they are written in. Such unittests would be treated as if they were not in the same scope or module. This includes detached unit tests inside templates. The only stipulation is that the current module is assumed imported. For a strawman syntax, let's do what everyone does ... overload `static`!It would be valuable for ddoc-ed unittest blocks, because it would allow us to guarantee that they don't use any part of the API which isn't public. Phobos currently has to do that by running a separate program to pull out all of the tests and make sure that they're runnable outside of the module, which is a lot more complicated and doesn't help anyone else. As for the syntax, static is probably fine so long as static: is ignored. I don't recall at the moment how much that affects, but it's terrible practice to use static that way anyway given how little sense it makes to try to apply static to multiple symbols at once.When I'd thought about this the other day, I'd thought that maybe we could use a pragma for it, but I don't really care much about the syntax, and static seems fine. It fits in with the basic meaning of how static is normally used.I'd go for a UDA in core.attribute. For finer grained control that is a great way to do it, since most people won't need it.Part of me thinks that we should just make ddoc-ed unit tests do this without additional syntax, but that would undoubtedly break existing code, since _someone_ will have a documented test somewhere that accidentally uses a private member, and it might also help with templates to be able to do this with undocumented unittest blocks (though I vaguely recall seeing something about how the behavior on those was changed at some point so that they're already not part of the template any longer, though I'm not sure on that, since I haven't written any templated types with tests inside them recently).This is what I proposed on Discord. We can change this behavior for newer editions. No reason to break existing code.
Apr 30
On 4/30/24 19:02, Steven Schveighoffer wrote:I want to propose that we provide a mechanism to semantically detach unit tests from the scope they are written in. Such unittests would be treated as if they were not in the same scope or module. This includes detached unit tests inside templates. The only stipulation is that the current module is assumed imported. For a strawman syntax, let's do what everyone does ... overload `static`! ... thoughts? -SteveLooks useful, as usually you want to test the API. Maybe this should have been the default behavior for unit tests with `public` visibility.
May 01
On Wednesday, 1 May 2024 at 12:55:01 UTC, Timon Gehr wrote:No, white-box test is correct as default, as black-box tests shouldn't be in the same file anyway. So, if you want your "logically black-box" tests in the same file, it is good that you have to explicitly opt in.For a strawman syntax, let's do what everyone does ... overload `static`!Looks useful, as usually you want to test the API. Maybe this should have been the default behavior for unit tests with `public` visibility.
May 01
On 5/1/24 15:44, Dom DiSc wrote:On Wednesday, 1 May 2024 at 12:55:01 UTC, Timon Gehr wrote:Ddoc unittests have to be in a very specific place though and they are generating user-facing documentation, that as a result of being white-box may accidentally rely on internal implementation details and not actually work for a user, even though all unit tests passed. Essentially, this makes documented unit tests less suitable for their intended purpose:No, white-box test is correct as default, as black-box tests shouldn't be in the same file anyway.For a strawman syntax, let's do what everyone does ... overload `static`!Looks useful, as usually you want to test the API. Maybe this should have been the default behavior for unit tests with `public` visibility.Documented unittests allow the developer to deliver code examples to the user, while at the same time automatically verifying that the examples are valid. This avoids the frequent problem of having outdated documentation for some piece of code.https://dlang.org/spec/unittest.html#documented-unittestsSo, if you want your "logically black-box" tests in the same file, it is good that you have to explicitly opt in.Well, I guess we'll disagree on this, I think it is not good that you have to opt into doing the more restrictive thing because the compiler will not complain about it if you get it wrong. My idea was to make a difference between `public unittest` and `private unittest`, but I do not feel strongly about it.
May 01
On Wednesday, 1 May 2024 at 12:55:01 UTC, Timon Gehr wrote:Looks useful, as usually you want to test the API. Maybe this should have been the default behavior for unit tests with `public` visibility.Hm... `public unittest` syntax? The problem I see with that syntax is that `public:` is way way more likely than `static:`. Perhaps you mean ddoc'd unittests? Maybe... I dislike comments affecting code generation (one of the reasons I think we should never have `__traits(getDocumentation, symbol)`), and also you may want to ddoc private functions for internal docs. I'm not attached to the `static unittest` syntax, it's just what's handy. A `core.attribute` UDA would work fine for me. -Steve
May 01
On Wednesday, May 1, 2024 8:26:21 AM MDT Steven Schveighoffer via dip.ideas wrote:On Wednesday, 1 May 2024 at 12:55:01 UTC, Timon Gehr wrote:I would expect it to be problematic for public or private attributes which weren't directly on the unittest blocks to affect unittest blocks (and not having public or private elsewhere affect unittest blocks would arguably be bad for language consistency). unittest blocks which aren't intended to be for documentation normally use private symbols or imports from the module, and we'd be forced to slap private on them all over the place once this feature was enabled. We'd also have to mark more unittest blocks with attributes that way, because typically, there are more unittest blocks which are not intended to be part of the documentation than those that are. Having it be automatic for ddoc-ed unittest blocks makes far more sense, because those are clearly intended to be part of the documentation and thus arguably shouldn't be using any private symbols or using any imports from the module at large (though for better or worse, it's certainly being more pedantic about it if we force that for all ddoc-ed unittest blocks rather than giving programmers the choice). However, that doesn't help us put undocumented unittest blocks next to what they're testing within templates without having those tests duplicated across every instantiation, and it would be very nice if we could fix that as part of this (though that would require that the compiler deal with templates differently than it does now, whereas for non-templated unittest blocks, it's simply a question of adding additional checks). So, I think that we're probably best off if we require that detached unit tests be explicitly marked as such. But I'm also not all that concerned about what that explicit marking looks like. static makes a lot of sense to me given what that means in general, but I'd also be totally fine with with some other attribute, including something like detached. - Jonathan M DavisLooks useful, as usually you want to test the API. Maybe this should have been the default behavior for unit tests with `public` visibility.Hm... `public unittest` syntax? The problem I see with that syntax is that `public:` is way way more likely than `static:`. Perhaps you mean ddoc'd unittests? Maybe... I dislike comments affecting code generation (one of the reasons I think we should never have `__traits(getDocumentation, symbol)`), and also you may want to ddoc private functions for internal docs. I'm not attached to the `static unittest` syntax, it's just what's handy. A `core.attribute` UDA would work fine for me.
May 01
On Tuesday, 30 April 2024 at 17:02:51 UTC, Steven Schveighoffer wrote:I want to propose that we provide a mechanism to semantically detach unit tests from the scope they are written in. Such unittests would be treated as if they were not in the same scope or module. This includes detached unit tests inside templates. The only stipulation is that the current module is assumed imported.I suggested something like this before and used the syntax `unittest public`. With a `static unittest`, I’d think it’s meant to be in a template, but independent of the template instance.
May 16