www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Let's bikeshed std.experimental.testing

reply "Atila Neves" <atila.neves gmail.com> writes:
In case you don't know what I'm talking about: 
https://github.com/D-Programming-Language/phobos/pull/3207

Since this is an API issue it's import to get it right the first 
time. Personally I'm not sure what I prefer (well, I am, but what 
I actually want isn't syntactically valid D). I think the options 
so far are:

1) What's there already, namely `shouldEquals`, `shouldBeIn`, etc.
2a) Compile-time strings for operators: `should!"=="`, 
`should!"in"`
2b) Dicebot's `test!"=="`. `assert` is so much better, I wish we 
could use that.
3) Composable ones: should.equals, should.not.equals, or another 
word that isn't "should"
4) Anything else?

I'm not convinced composability brings anything to the table 
except for editor dot-completion. I don't like the verbosity of 
what's there now, but my prefererred syntax doesn't work except 
for the ubiquitous  check for equality (`should ==`). Well, the 
dream would be that `assert(foo == bar)` did what part of this PR 
does, but that's another story and something that can't be done 
by a library unless we had AST macros, which we won't. Or Lisp's 
reader macros, but we won't get those either.

Thoughts? Votes?

Atila
Jun 30 2015
next sibling parent reply "Adrian Matoga" <epi atari8.info> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:


 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity of 
 what's there now, but my prefererred syntax doesn't work except 
 for the ubiquitous  check for equality (`should ==`).


Could you give some examples of your preferred syntax and why it 
doesn't work?
Jun 30 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 08:38:44 UTC, Adrian Matoga wrote:

 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:


 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity 
 of what's there now, but my prefererred syntax doesn't work 
 except for the ubiquitous  check for equality (`should ==`).


 Could you give some examples of your preferred syntax and why 
 it doesn't work?


`foo.should == "bar";` works. Nothing else does (and in fact in 
retrospect it's surprising that == does) because they do nothing 
by themselves. `foo.should != bar` is the same as `!(foo == 
bar)`, which on a statement by itself is nonsensical and rejected 
by the compiler with "has no effect in expression" error. You 
could write something like `if(foo.should != "bar") {}` and that 
compiles fine but it's super hacky and ugly.

Atila
Jun 30 2015
parent "Adrian Matoga" <epi atari8.info> writes:
On Tuesday, 30 June 2015 at 11:14:55 UTC, Atila Neves wrote:

 On Tuesday, 30 June 2015 at 08:38:44 UTC, Adrian Matoga wrote:

 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:


 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity 
 of what's there now, but my prefererred syntax doesn't work 
 except for the ubiquitous  check for equality (`should ==`).


 Could you give some examples of your preferred syntax and why 
 it doesn't work?


 `foo.should == "bar";` works. Nothing else does (and in fact in 
 retrospect it's surprising that == does) because they do 
 nothing by themselves. `foo.should != bar` is the same as 
 `!(foo == bar)`, which on a statement by itself is nonsensical 
 and rejected by the compiler with "has no effect in expression" 
 error. You could write something like `if(foo.should != "bar") 
 {}` and that compiles fine but it's super hacky and ugly.

 Atila


Thanks. I took a second look and now it looks obvious.

I'd vote for Dicebot's bikeshed design (2b). It's short and looks 
equally readable for every possible test.

Another hackish possibility could be to change the behavior of 
assert so that it also passes the stringified expression to 
_d_assert* and _d_unittest* in druntime.
Jun 30 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
To do it _really_ nice, we would need some sort of ".codeof" 
feature. Possibly, with implicit returns in lambdas. For example:

void test(alias expr)
{
     if (!expr())
         throw new TestException(
             expr.codeof ~ " has failed,\n" ~
             /* investigate expr context pointer and grab a/b */
             "\ta = x\n" ~
             "\tb = y"
}

unittest
{
     int a, b;
     test!({ a == b; });
}

Of course, this is one of cases where AST macros would really 
shine. But I think it should be possible to provide necessary 
functionality subset in a much more simple and limited feature.

In absence of language changes, I don't see anything as clear and 
simple as operator mixins. Less magic in unittests -> better. 
Common misconception IMHO is that tests should look nice for 
library/app author, while, from the ecosystem PoV, they should 
look simple and predictable for contributors - and that is most 
important property.
Jun 30 2015
parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 30 June 2015 at 11:43:36 UTC, Dicebot wrote:

 In absence of language changes, I don't see anything as clear 
 and simple as operator mixins. Less magic in unittests -> 
 better. Common misconception IMHO is that tests should look 
 nice for library/app author, while, from the ecosystem PoV, 
 they should look simple and predictable for contributors - and 
 that is most important property.


I tend to agree with your position on testing frameworks. It 
seems really cool to utilize English to spell out an expectation, 
but it ends up more complicated. For one, it can never actually 
be English (I wouldn't want it to be) and for another, I've 
already got expressions from the language being used that mean 
the same thing.

     unittest {
         struct A {
             int m;
         }

         A a;
         A b;
         a.m = 5;
         b.m = 5;

         auto testA = a.test;
         with(testA) {
             verify(testA.lhs == b);
             b.m = 6;
             verify(testA.lhs != b);
             verify(testA.lhs == b);
         }
     }

It is kind of like what you're saying, but the condition isn't 
reportable. I'm actually less concerned about the condition as I 
am being able to report all the values utilized by the condition. 
I'd like to see assert be more descriptive or a framework that 
looks like assert and is more descriptive. Here is the crap code 
to run the above:

https://gist.github.com/JesseKPhillips/df79479cbf6a0e3c6b0d
Jun 30 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 1 July 2015 at 00:26:21 UTC, Jesse Phillips wrote:

 On Tuesday, 30 June 2015 at 11:43:36 UTC, Dicebot wrote:

 [...]


 I tend to agree with your position on testing frameworks. It 
 seems really cool to utilize English to spell out an 
 expectation, but it ends up more complicated. For one, it can 
 never actually be English (I wouldn't want it to be) and for 
 another, I've already got expressions from the language being 
 used that mean the same thing.

 [...]


opEquals is easy. It's everything else that's hard.

Atila
Jul 01 2015
parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Wednesday, 1 July 2015 at 08:22:50 UTC, Atila Neves wrote:

 opEquals is easy. It's everything else that's hard.

 Atila


I thought that opNotEquals was also a challenge. The way I 
approached it should be usable by all operator overloads, but I'm 
not making the claim that the way I approached it was any good. 
Only saying its the direction I prefer.
Jul 01 2015
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 1) What's there already, namely `shouldEquals`, `shouldBeIn`, 
 etc.
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
 3) Composable ones: should.equals, should.not.equals, or 
 another word that isn't "should"
 4) Anything else?

 - snip -

 Thoughts? Votes?


I really hate the 3rd persons "s" in "shouldEquals" / 
"should.equals". Not only is it grammatically wrong, it's also 
inconsistent with "shouldBeIn", which - following the same scheme 
- would have to be "shouldIsIn".
Jun 30 2015
prev sibling next sibling parent "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 In case you don't know what I'm talking about: 
 https://github.com/D-Programming-Language/phobos/pull/3207

 Since this is an API issue it's import to get it right the 
 first time. Personally I'm not sure what I prefer (well, I am, 
 but what I actually want isn't syntactically valid D). I think 
 the options so far are:

 1) What's there already, namely `shouldEquals`, `shouldBeIn`, 
 etc.
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
 3) Composable ones: should.equals, should.not.equals, or 
 another word that isn't "should"
 4) Anything else?

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity of 
 what's there now, but my prefererred syntax doesn't work except 
 for the ubiquitous  check for equality (`should ==`). Well, the 
 dream would be that `assert(foo == bar)` did what part of this 
 PR does, but that's another story and something that can't be 
 done by a library unless we had AST macros, which we won't. Or 
 Lisp's reader macros, but we won't get those either.

 Thoughts? Votes?

 Atila


Google uses gMock. On top of being a mock framework it provides 
composable matchers API which are a great way of expressing what 
you want to test.
https://www.youtube.com/watch?v=sYpCyLI47rM?t=19m15s (excerpt 
from outdated gMock presentation from 2008)

The matcher api is based on Hamcrest and I think it's pretty 
convenient.

Crash or log :
assertThat(...);
expectThat(...);

Simple tests are easy to express.
assertThat(4, 5);

More complex things:
assertThat(theBiscuit, is(equalTo(myBiscuit)));
assertThat(theBiscuit, not(instanceOf(Liquid.class)));

Same thing with container:
assertThat(map, not(hasKey(lessThan(5))));

The tests are pretty readable and predicates compose nicely.

Also because you express the condition as a tree of predicates 
the error reporting can be really nice. Like "map contains 
element which key is not less than 5".

Also one can write custom predicates for more specific tests.
The C++ implementation relies on ugly template/macros. I'm sure a 
D implementation can be cleaner and pretty sweet to use.

I suspect throwing ufcs in here will makes it even better.
Jun 30 2015
prev sibling next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.


I think these are both bad, for the reason that ! also means 
logical not. I read `should!"=="` as "should not equal" before 
catching myself.
Jun 30 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 12:00:36 UTC, Meta wrote:

 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.


 I think these are both bad, for the reason that ! also means 
 logical not. I read `should!"=="` as "should not equal" before 
 catching myself.


I'd forgotten about it, but I knew there was a reason I didn't 
like `should!"=="`, and it was the same reason as you.

Atila
Jun 30 2015
prev sibling next sibling parent reply "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 In case you don't know what I'm talking about: 
 https://github.com/D-Programming-Language/phobos/pull/3207

 Since this is an API issue it's import to get it right the 
 first time. Personally I'm not sure what I prefer (well, I am, 
 but what I actually want isn't syntactically valid D). I think 
 the options so far are:

 1) What's there already, namely `shouldEquals`, `shouldBeIn`, 
 etc.
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
 3) Composable ones: should.equals, should.not.equals, or 
 another word that isn't "should"
 4) Anything else?

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity of 
 what's there now, but my prefererred syntax doesn't work except 
 for the ubiquitous  check for equality (`should ==`). Well, the 
 dream would be that `assert(foo == bar)` did what part of this 
 PR does, but that's another story and something that can't be 
 done by a library unless we had AST macros, which we won't. Or 
 Lisp's reader macros, but we won't get those either.

 Thoughts? Votes?

 Atila



Much rather prefer the composable ones over the `shouldEquals`, 
simply for readability and easy extending.

These days I am leaning towards BDD, but everybody has his 
favorite. Maybe just providing the low-level details in 
std.testing would enough; e.g. a test runner, UDA's and 
assertions.

Then everyone can write his on version of given().when().then() 
on top of it. Or simply make a pull-request for std.testing.bdd
Jun 30 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 12:42:40 UTC, Sebastiaan Koppe wrote:

 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 [...]



 Much rather prefer the composable ones over the `shouldEquals`, 
 simply for readability and easy extending.

 These days I am leaning towards BDD, but everybody has his 
 favorite. Maybe just providing the low-level details in 
 std.testing would enough; e.g. a test runner, UDA's and 
 assertions.

 Then everyone can write his on version of given().when().then() 
 on top of it. Or simply make a pull-request for std.testing.bdd


Yeah, I'm starting to think it might be better to delete 
`should.d` from my current PR, try to get the rest approved then 
work on where the community wants the fancy assertions to go. 
It's a shame though because I think it's a massively important 
piece of the whole thing. It's a night and day difference when a 
test fails.

Atila
Jun 30 2015
next sibling parent "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Tuesday, 30 June 2015 at 14:58:45 UTC, Atila Neves wrote:

 On Tuesday, 30 June 2015 at 12:42:40 UTC, Sebastiaan Koppe 
 wrote:

 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 [...]


 These days I am leaning towards BDD, but everybody has his 
 favorite. Maybe just providing the low-level details in 
 std.testing would enough; e.g. a test runner, UDA's and 
 assertions.


 Yeah, I'm starting to think it might be better to delete 
 `should.d` from my current PR, try to get the rest approved 
 then work on where the community wants the fancy assertions to 
 go. It's a shame though because I think it's a massively 
 important piece of the whole thing. It's a night and day 
 difference when a test fails.

 Atila


Makes sense. You could still keep the should's, just rename the 
whole lot to isEmpty / isNotEmpty / isGreaterThan and have it 
return a bool instead of calling fail internally. Then you would 
simply expect the callee to do that. As in: 
`assert(5.isEqual(6));`.
Jun 30 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 30/06/15 16:58, Atila Neves wrote:


 Yeah, I'm starting to think it might be better to delete `should.d` from
 my current PR, try to get the rest approved then work on where the
 community wants the fancy assertions to go. It's a shame though because
 I think it's a massively important piece of the whole thing. It's a
 night and day difference when a test fails.


Yeah, I agree, it would be a shame to not have these assertions.

-- 
/Jacob Carlborg
Jun 30 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 30/06/15 10:06, Atila Neves wrote:


 Well, the dream would be that `assert(foo ==
 bar)` did what part of this PR does, but that's another story and
 something that can't be done by a library unless we had AST macros,
 which we won't. Or Lisp's reader macros, but we won't get those either.


I was thinking the same. Both the test!"==" and shouldEqual are 
workarounds to get a nice message on an assertion failure. I'm wondering 
how hard it would be to have the compiler generate a string representing 
the failing expression.

-- 
/Jacob Carlborg
Jun 30 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 19:48:46 UTC, Jacob Carlborg wrote:

 On 30/06/15 10:06, Atila Neves wrote:


 Well, the dream would be that `assert(foo ==
 bar)` did what part of this PR does, but that's another story 
 and
 something that can't be done by a library unless we had AST 
 macros,
 which we won't. Or Lisp's reader macros, but we won't get 
 those either.


 I was thinking the same. Both the test!"==" and shouldEqual are 
 workarounds to get a nice message on an assertion failure. I'm 
 wondering how hard it would be to have the compiler generate a 
 string representing the failing expression.


Well, there's a PR for improving assertions here: 
https://github.com/D-Programming-Language/dmd/pull/1426

Since I have 0 experience in compilers in general and dmd in 
particular, I thought that'd be an easy way for me to get an in 
on the assert situation. It seems... more complicated than I can 
handle at the moment.

The 100% ideal situation is for assert to do what I'm doing with 
the functions in the `should` module. That module really 
shouldn't even exist.

Atila
Jul 01 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 1 July 2015 at 08:21:28 UTC, Atila Neves wrote:

 Well, there's a PR for improving assertions here: 
 https://github.com/D-Programming-Language/dmd/pull/1426


Sadly, this does not really fix the issue in the long term. Using 
assertions in tests prevents building non-fatal test runners 
because by spec AssertError is non-recoverable. That is quite a 
serious limitation.

To really fix it, this information needs to be available in 
library code.
Jul 01 2015
parent Jacob Carlborg <doob me.com> writes:
On 01/07/15 15:45, Dicebot wrote:


 Sadly, this does not really fix the issue in the long term. Using
 assertions in tests prevents building non-fatal test runners because by
 spec AssertError is non-recoverable. That is quite a serious limitation.


There's an assertion handler that can be set in druntime [1]. 
Unfortunately the callback is declared as "nothrow". If this "nothrow" 
it could be used to throw an exception instead of an error.

[1] 
https://github.com/D-Programming-Language/druntime/blob/master/src/core/exception.d#L378

-- 
/Jacob Carlborg
Jul 01 2015
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
https://www.youtube.com/watch?v=pWS8Mg-JWSg&feature=player_detailpage#t=81
Jun 30 2015
prev sibling next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 In case you don't know what I'm talking about: 
 https://github.com/D-Programming-Language/phobos/pull/3207

 Since this is an API issue it's import to get it right the 
 first time. Personally I'm not sure what I prefer (well, I am, 
 but what I actually want isn't syntactically valid D). I think 
 the options so far are:

 1) What's there already, namely `shouldEquals`, `shouldBeIn`, 
 etc.
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
 3) Composable ones: should.equals, should.not.equals, or 
 another word that isn't "should"
 4) Anything else?

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity of 
 what's there now, but my prefererred syntax doesn't work except 
 for the ubiquitous  check for equality (`should ==`). Well, the 
 dream would be that `assert(foo == bar)` did what part of this 
 PR does, but that's another story and something that can't be 
 done by a library unless we had AST macros, which we won't. Or 
 Lisp's reader macros, but we won't get those either.

 Thoughts? Votes?

 Atila


After much thinking and some _incredibly_ hacky attempts to come 
up with a library solution that can use operators (it was really 
hacky, I was using `assert()` and making the expression always 
return true), I came to the conclusion that what I have there 
already is probably as good as it's going to get bar language 
changes.

The reason is this: D has very smartly side-stepped C++'s 
problems with operator overloading and boilerplate by making 
things like `a >=b` be translated to `a.opCmp(b) >=0`. This is 
one of them Good Things. The same with `!=`, it's 
`!(a.opEquals(b)`. Again, good.

But for test asserts, that means that by the time any possible 
library code gets to handle it there's been a loss of 
information. There's so way to know whether opEquals was called 
with or without negation, or if opCmp was called because the user 
typed `>`, `>=`, `<` or `<=`.

I can make this work:

3.timesTwo.should == 6;
3.timesTwo.should.not == 5;


But anything with opCmp is a no-go. The only other operator that 
would make sense (to me) to overload is opBinary!("in"), but the 
naming there is a problem. `should.in`? `should.beIn`? Oh wait, 
that's not the operator anymore. Which brings me to...

Naming. Any one word that isn't `assert` is... not as good. It 
always breaks down for one use case or another, and I think 
what's in the PR is the best I've heard so far on this forum or 
in the comments. A compile-time string is... ugly and error-prone 
especially since the most common assertion is for equality and 
`!"=="` looks too much like `!=`. So, despite the fact that I 
wrote `shouldBeGreaterThan`, I hate its verbosity and all I 
really want to write is `assert(a > b)` and get a decent error 
message.

I don't buy that `should.` is more extensible. For two reasons, 
first because in all the test frameworks I've seen there aren't 
that many more types of assertions, and secondly because adding a 
member function to the struct returned by `should` and adding a 
new `shouldCamelCase` function is the same amount of work. As for 
auto-completion, I also see no real difference between the two.

Other than assert being semi-magical, I don't see how it can be 
any better.

Atila
Jul 01 2015
next sibling parent reply "Mike" <none none.com> writes:
On Wednesday, 1 July 2015 at 08:40:24 UTC, Atila Neves wrote:

 I hate its verbosity and all I really want to write is 
 `assert(a > b)` and get a decent error message.


I haven't been following this too closely as there is only so 
much I can pay attention to at any time, so I apologize in 
advance if the following comments are uninformed.  If so, just 
ignore them.

I suspect from your comments that you want to use `assert`, but 
it's currently reserved by the language and 
https://github.com/D-Programming-Language/dmd/pull/1426 was not 
followed through on, and is currently owned by a ghost 
(literally).

I therefore suggest using `assertThat(whatever)`.  If/when 
`assert` is improved upon to allow integration into this library, 
`assertThat` can be deprecated.

Sorry if I'm making noise,
Mike
Jul 01 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 1 July 2015 at 09:08:22 UTC, Mike wrote:

 On Wednesday, 1 July 2015 at 08:40:24 UTC, Atila Neves wrote:

 I hate its verbosity and all I really want to write is 
 `assert(a > b)` and get a decent error message.


 I haven't been following this too closely as there is only so 
 much I can pay attention to at any time, so I apologize in 
 advance if the following comments are uninformed.  If so, just 
 ignore them.

 I suspect from your comments that you want to use `assert`, but 
 it's currently reserved by the language and 
 https://github.com/D-Programming-Language/dmd/pull/1426 was not 
 followed through on, and is currently owned by a ghost 
 (literally).

 I therefore suggest using `assertThat(whatever)`.  If/when 
 `assert` is improved upon to allow integration into this 
 library, `assertThat` can be deprecated.


The problem there is `assertThat` gives no more information than 
`assert` currently does. This thread is all about how do we want 
to deal with custom assertions that give you useful information. 
My conclusion so far has been that what's there is probably the 
best we're going to get.

I could always change the shouldXXX functions to be called 
assertXXX instead, but it doesn't look as good with UFCS. That 
might get me Dicebot's vote though :)

Atila

Atila


 Sorry if I'm making noise,
 Mike
Jul 01 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 01/07/15 10:40, Atila Neves wrote:


 So, despite the fact that I wrote `shouldBeGreaterThan`, I hate its
 verbosity


You could write "shouldBe.gt(value)".


 I don't buy that `should.` is more extensible. For two reasons, first
 because in all the test frameworks I've seen there aren't that many more
 types of assertions


In every project I have used RSpec I have added custom 
matchers/assertions. Just a couple of days ago I added a custom matcher 
to one of my projects:

code = code_to_file('void foo() {}')
code.should be_parsed_as('meta.definition.method.d')

The point of these custom matchers/assertions are that they should make 
the tests (or specs in the BDD case) more readable and provide better 
assertion failure messages. Of course, none of these assertions are 
necessary, we could all just use "assert" or "shouldEqual", in the end 
every assertion is just a boolean condition.

My test could instead look like this, without using a custom matcher:

file = code_to_file('void foo() {}')

line = result.split("\n").first
scope = 'meta.definition.method.d'


Which one of the two versions are more readable?


 and secondly because adding a member function to
 the struct returned by `should` and adding a new `shouldCamelCase`
 function is the same amount of work.


The idea would be that the part after "should" would be a free function 
and use UFCS to call it.

Should should()
{
     return Should();
}

void beCamelCase(Should s);

should.beCamelCase

That's what allows to create custom assertions. A user of the library 
can't extend the struct returned by "should".

-- 
/Jacob Carlborg
Jul 01 2015
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 1 July 2015 at 19:38:20 UTC, Jacob Carlborg wrote:

 On 01/07/15 10:40, Atila Neves wrote:


 [...]


 You could write "shouldBe.gt(value)".


 [...]


 In every project I have used RSpec I have added custom 
 matchers/assertions. Just a couple of days ago I added a custom 
 matcher to one of my projects:

 [...]


Ah, makes sense. I think I'm convinced now. The UFCS as an 
extension mechanism could indeed be handy.

Atila
Jul 02 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Thursday, 2 July 2015 at 12:22:31 UTC, Atila Neves wrote:

 On Wednesday, 1 July 2015 at 19:38:20 UTC, Jacob Carlborg wrote:

 On 01/07/15 10:40, Atila Neves wrote:


 [...]


 You could write "shouldBe.gt(value)".


 [...]


 In every project I have used RSpec I have added custom 
 matchers/assertions. Just a couple of days ago I added a 
 custom matcher to one of my projects:

 [...]


 Ah, makes sense. I think I'm convinced now. The UFCS as an 
 extension mechanism could indeed be handy.

 Atila


So... unconvinced again. I tried implementing it and it started 
getting to be a right royal pain, and then I realised that 
there's nothing to prevent a user from writing their own matchers 
right now. From your example:

     void shouldBeParsedAs(Code code, ASTNode node) {  //I don't 
really know what the types should be
         if(...) {
             throw new UnitTestException(...);
         }
     }

And... done. No need for a `Should` struct, no need for the 
complications I faced trying to use one. I think the design is as 
good as it can be.

Atila
Jul 10 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-07-10 14:37, Atila Neves wrote:


 So... unconvinced again. I tried implementing it and it started getting
 to be a right royal pain, and then I realised that there's nothing to
 prevent a user from writing their own matchers right now. From your
 example:

      void shouldBeParsedAs(Code code, ASTNode node) {  //I don't really
 know what the types should be
          if(...) {
              throw new UnitTestException(...);
          }
      }

 And... done. No need for a `Should` struct, no need for the
 complications I faced trying to use one. I think the design is as good
 as it can be.


If I compare this with RSpec. A custom matcher in RSpec would implement 
one method, returning a bool, to indicate if the assertion passed or 
not. In your example the user need to worry about throwing exceptions 
and which kind to throw. I consider that an implementation detail.

Also, RSpec is able to automatically infer a failure message based on 
the name of the matcher. Optionally the matcher can customize the 
failing message.

It seems like with your version you need to implement both a positive 
and negative version.

-- 
/Jacob Carlborg
Jul 12 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Sunday, 12 July 2015 at 15:15:07 UTC, Jacob Carlborg wrote:

 On 2015-07-10 14:37, Atila Neves wrote:


 So... unconvinced again. I tried implementing it and it 
 started getting
 to be a right royal pain, and then I realised that there's 
 nothing to
 prevent a user from writing their own matchers right now. From 
 your
 example:

      void shouldBeParsedAs(Code code, ASTNode node) {  //I 
 don't really
 know what the types should be
          if(...) {
              throw new UnitTestException(...);
          }
      }

 And... done. No need for a `Should` struct, no need for the
 complications I faced trying to use one. I think the design is 
 as good
 as it can be.


 If I compare this with RSpec. A custom matcher in RSpec would 
 implement one method, returning a bool, to indicate if the 
 assertion passed or not. In your example the user need to worry 
 about throwing exceptions and which kind to throw. I consider 
 that an implementation detail.

 Also, RSpec is able to automatically infer a failure message 
 based on the name of the matcher. Optionally the matcher can 
 customize the failing message.

 It seems like with your version you need to implement both a 
 positive and negative version.


There's a reason for that and I'm curious as to how RSpec handles 
it. If it were just a matter of pairing up assertions for the two 
boolean values of a condition that'd be easy. But the formatting 
of the error message is anything but. `shouldEqual` and 
`shouldNotEqual` have very different outputs even if what causes 
an exception to be thrown is just a check on the opposite 
condition.

For the same reason, a user wanting to extend the framework has 
to format their own error messages, which means just defining a 
predicate isn't enough.

If I'd seen any more duplication I'd've removed it, but the 
different positive and negative versions of the `should` 
assertions are due to the very asymmetrical error messages. If 
there's a better way of doing it I'd love to find out.

Atila
Jul 13 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 1 July 2015 at 19:38:20 UTC, Jacob Carlborg wrote:

 In every project I have used RSpec I have added custom 
 matchers/assertions. Just a couple of days ago I added a custom 
 matcher to one of my projects:

 code = code_to_file('void foo() {}')
 code.should be_parsed_as('meta.definition.method.d')

 The point of these custom matchers/assertions are that they 
 should make the tests (or specs in the BDD case) more readable 
 and provide better assertion failure messages. Of course, none 
 of these assertions are necessary, we could all just use 
 "assert" or "shouldEqual", in the end every assertion is just a 
 boolean condition.

 My test could instead look like this, without using a custom 
 matcher:

 file = code_to_file('void foo() {}')

 2>&1`
 line = result.split("\n").first
 scope = 'meta.definition.method.d'


 Which one of the two versions are more readable?


Neither. But with the second one I at least have a chance to 
figure out what it actually tests. To understand the first one 
I'd need to read the code of that custom matcher.

This is exactly the problem with all that fancy - library/app 
author (who by definition knows the domain well) implicitly 
introduces custom DSL and expects everyone (who, by definition, 
have very vague understanding of domain) to be able to read it as 
comfortably as himself.
Jul 02 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 02/07/15 14:28, Dicebot wrote:


 Neither. But with the second one I at least have a chance to figure out
 what it actually tests. To understand the first one I'd need to read the
 code of that custom matcher.


Once you know what a matcher does or how it's used it's a lot more 
readable and easier to create new tests.


 This is exactly the problem with all that fancy - library/app author
 (who by definition knows the domain well) implicitly introduces custom
 DSL and expects everyone (who, by definition, have very vague
 understanding of domain) to be able to read it as comfortably as himself.


By that definition we should all write object oriented code in C. Heck, 
why don't we just use assembly and be done with it.

-- 
/Jacob Carlborg
Jul 02 2015
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 2 July 2015 at 19:06:22 UTC, Jacob Carlborg wrote:

 On 02/07/15 14:28, Dicebot wrote:


 Neither. But with the second one I at least have a chance to 
 figure out
 what it actually tests. To understand the first one I'd need 
 to read the
 code of that custom matcher.


 Once you know what a matcher does or how it's used it's a lot 
 more readable and easier to create new tests.


I am not going to investigate all custom matchers and stuff to 
submit one simple pull request to a project I don't care much 
about. Simple helper function may be slightly less "pretty" but 
much easier to grep for and reason about.


 By that definition we should all write object oriented code in 
 C. Heck, why don't we just use assembly and be done with it.


That is exactly why language features are better than free form 
AST macros. Any abstraction has inherent learning costs. Any 
non-standard abstraction - even more so. For in-house project it 
tends to be still worth it because maintenance costs tend to be 
higher than learning costs and development team is relatively 
stable. For open-source ecosystem - quite the contrary. And here 
we speak about something that lays foundation to any open-source 
contributions - standard testing system.
Jul 02 2015
prev sibling parent "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Thursday, 2 July 2015 at 12:28:41 UTC, Dicebot wrote:

 On Wednesday, 1 July 2015 at 19:38:20 UTC, Jacob Carlborg wrote:

 In every project I have used RSpec I have added custom 
 matchers/assertions. Just a couple of days ago I added a 
 custom matcher to one of my projects:

 code = code_to_file('void foo() {}')
 code.should be_parsed_as('meta.definition.method.d')

 The point of these custom matchers/assertions are that they 
 should make the tests (or specs in the BDD case) more readable 
 and provide better assertion failure messages. Of course, none 
 of these assertions are necessary, we could all just use 
 "assert" or "shouldEqual", in the end every assertion is just 
 a boolean condition.

 My test could instead look like this, without using a custom 
 matcher:

 file = code_to_file('void foo() {}')

 2>&1`
 line = result.split("\n").first
 scope = 'meta.definition.method.d'


 Which one of the two versions are more readable?


 Neither. But with the second one I at least have a chance to 
 figure out what it actually tests. To understand the first one 
 I'd need to read the code of that custom matcher.


The good thing about the first one is that you give it a name. 
The names tells you something about what it does, without having 
to understand the low-level details - but also, it gives you 
something to refer to in the documentation.

Moreover, it is the inevitable outcome after one has 
copy-and-pasted the second version 10 times.


 This is exactly the problem with all that fancy - library/app 
 author (who by definition knows the domain well) implicitly 
 introduces custom DSL and expects everyone (who, by definition, 
 have very vague understanding of domain) to be able to read it 
 as comfortably as himself.


A natural consequence of having a domain in the first place.
Jul 03 2015
prev sibling parent reply "linkrope" <linkrope github.com> writes:
1. 'assert' is the wrong thing

I do not need a stack trace for a failed expectation in a unit 
test.
But, usually, I need a stack trace for a contract violation from 
deep down in the unit under test:

https://github.com/linkrope/dunit#failures-vs-errors

Consequently, the failed expectation should throw some "failed 
expectation" 'Exception' instead of an 'AssertError'. So, 
std.exception.assertThrown cannot be used for expectations.

2. something like an 'assert' with better diagnostics could be 
nice

Catch for C++ introduced 'REQUIRE' with helpful failure messages:

     Example.cpp:9: FAILED:
       REQUIRE( Factorial(0) == 1 )
     with expansion:
       0 == 1

https://github.com/philsquared/Catch

3. 'should' seems to be obsolete

Now, 'expect' should be used for expectations:

http://rspec.info/blog/2012/06/rspecs-new-expectation-syntax/
Jul 03 2015
parent Jacob Carlborg <doob me.com> writes:
On 03/07/15 16:07, linkrope wrote:


 3. 'should' seems to be obsolete

 Now, 'expect' should be used for expectations:

 http://rspec.info/blog/2012/06/rspecs-new-expectation-syntax/


As far as I know, the only reason for that is that "should" is monkey 
patched on every class. This can cause problems if "method_missing" is 
implemented. D doesn't have this problem since "should" is not monkey 
patched on anything. In this code "should" is implemented like a free 
function and UFCS is used to get the same syntax as in Ruby. If there 
are problems with opDispatch/alias this (the D version of 
method_missing) then the regular function call syntax can be used.

-- 
/Jacob Carlborg
Jul 03 2015