www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Integration tests

reply Russel Winder <russel winder.org.uk> writes:
Hi,

Thinking of trying to do the next project in D rather than Rust, but=E2=80=
=A6

Rust has built in unit testing on a module basis. D has this so no
problem.

Rust allows for integration tests in the tests directory of a project.
These are automatically build and run along with all unit tests as part
of "cargo test".

Does D have any integrated support for integration tests in the way
Rust does?
=20
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Apr 17 2020
next sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Friday, 17 April 2020 at 16:56:57 UTC, Russel Winder wrote:

 Hi,

 Thinking of trying to do the next project in D rather than 
 Rust, but…

 Rust has built in unit testing on a module basis. D has this so 
 no problem.

 Rust allows for integration tests in the tests directory of a 
 project. These are automatically build and run along with all 
 unit tests as part of "cargo test".

 Does D have any integrated support for integration tests in the 
 way
 Rust does?


Automated testing is important, perhaps you describe further 
what's needed? I haven't worked with Rust test frameworks, but I 
took a look at the description of the integration tests and unit 
tests. It wasn't immediately obvious what can be done with the 
Rust integration test framework that cannot be done with D's 
unittest framework.

An important concept described was testing a module as an 
external caller. That would seem very be doable using D's 
unittest framework. For example, one could create a set of tests 
against Phobos, put them in a separate location (e.g. a separate 
file), and arrange to have the unittests run as part of a CI 
process run along with a build.

My look was very superficial, perhaps you could explain more.
Apr 17 2020
parent Russel Winder <russel winder.org.uk> writes:
On Fri, 2020-04-17 at 17:51 +0000, Jon Degenhardt via Digitalmars-d-
learn wrote:

 On Friday, 17 April 2020 at 16:56:57 UTC, Russel Winder wrote:

 Hi,
=20
 Thinking of trying to do the next project in D rather than=20
 Rust, but=E2=80=A6
=20
 Rust has built in unit testing on a module basis. D has this so=20
 no problem.
=20
 Rust allows for integration tests in the tests directory of a=20
 project. These are automatically build and run along with all=20
 unit tests as part of "cargo test".
=20
 Does D have any integrated support for integration tests in the=20
 way
 Rust does?


=20
 Automated testing is important, perhaps you describe further=20
 what's needed? I haven't worked with Rust test frameworks, but I=20
 took a look at the description of the integration tests and unit=20
 tests. It wasn't immediately obvious what can be done with the=20
 Rust integration test framework that cannot be done with D's=20
 unittest framework.


I should point out that Rust is far behind Python's PyTest in terms of
testing sophistication, and I fear D lags behind Rust.=20

I think the following is just saying what everyone knows, but it seems
making it explicit for this thread.

Let us distinguish unit testing, integration testing, and system
testing using the definitions: unit testing is self standing testing
the code with no extra dependencies; integration testing is testing
some functionality of the system with all external resources mocked out
(possibly by using a process to simulate the external resource, which
is what I want for this project); system testing is testing some
functionality with real external resources.

D, like Rust, supports unit testing very nicely, well unit_threading
seems essential, on a per module basis. Python unit testing with PyTest
is very straightforward.=20

Rust supports integration testing explicitly, D does not. Python PyTest
does but only because with Python you need no extra infrastructure.
Integration testing must not have access to the internals of a module
or group of modules, but should appear as a using application.
Rust/Cargo integrates this using the tests directory as a peer to the
src directory: each file in the tests directory represents a crate that
uses the code under test as a crate.

Integration and automated system testing in Rust is really a question
of whether external resources are mocked or used for real. Python
PyTest integration and system testing is very much the same.

As far as I know, D/Dub, D/Meson, D/SCons, D/CMake, and D/Make do not
provide an out of the box way of building integration tests.

Hidden agenda item: system testing really needs to be in a sandbox of
some sort.=20


 An important concept described was testing a module as an=20
 external caller. That would seem very be doable using D's=20
 unittest framework. For example, one could create a set of tests=20
 against Phobos, put them in a separate location (e.g. a separate=20
 file), and arrange to have the unittests run as part of a CI=20
 process run along with a build.
=20
 My look was very superficial, perhaps you could explain more.


The important difference between using a unit test infrastructure and
an integration/system test infrastructure is that:

=E2=80=93 unit tests are inside the system testing functions and other code
features;
=E2=80=93 integration tests are outside the system testing functionality wi=
th
mocked external resources.

So the build system needs to build all the codes that realise the=20
mocks of the external resources. So for a vestigial Rust project:

.
=E2=94=9C=E2=94=80=E2=94=80 Cargo.lock
=E2=94=9C=E2=94=80=E2=94=80 Cargo.toml
=E2=94=9C=E2=94=80=E2=94=80 README.md
=E2=94=9C=E2=94=80=E2=94=80 src
=E2=94=82   =E2=94=9C=E2=94=80=E2=94=80 arcam_protocol.rs
=E2=94=82   =E2=94=9C=E2=94=80=E2=94=80 lib.rs
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 main.rs
=E2=94=94=E2=94=80=E2=94=80 tests
    =E2=94=9C=E2=94=80=E2=94=80 integration_tests.rs
    =E2=94=94=E2=94=80=E2=94=80 mock_avr850.rs

src/arcam_protocol/rs has all the unit tests in it. D can do the
equivalent.

tests/mock_avr850.rs is an application that mocks a real amplifier;
tests/integration_tests is an application that uses the project as an
external crate and starts the mock amplifier for each of the tests. As
far as I can tell there is no Dub, Meson, Scons, CMake, Make
infrastructure for building D code that supports this sort of thing.

Python using PyTest can do all the above really very trivially, D/Cargo
has yet to catch up!


--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Apr 20 2020
prev sibling parent reply aliak <something something.com> writes:
On Friday, 17 April 2020 at 16:56:57 UTC, Russel Winder wrote:

 Hi,

 Thinking of trying to do the next project in D rather than 
 Rust, but…

 Rust has built in unit testing on a module basis. D has this so 
 no problem.

 Rust allows for integration tests in the tests directory of a 
 project. These are automatically build and run along with all 
 unit tests as part of "cargo test".

 Does D have any integrated support for integration tests in the 
 way
 Rust does?


D does not recognise any special testing directories afaik. But 
you'll get the same affect as in rust if you have your 
integration tests in a file that is not the modules you are 
testing and run it after building the main library - or during 
even.

With Dub it's quite easy to set up, you can see the optional 
package as an example [0]. There's a tests directory that does 
not have access to the optional package's internals. There's also 
a unittest-compat configuration in dub.json that tests the 
package alongside vibed (another d package)

For auto mocking, the only thing I've see that provides some 
functionality is the unit-threaded package. But I've never tried 
it.

[0]: https://github.com/aliak00/optional
Apr 20 2020
next sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Mon, 2020-04-20 at 20:19 +0000, aliak via Digitalmars-d-learn wrote:

=20
 [=E2=80=A6]



 [0]: https://github.com/aliak00/optional


Rust has Option and Result, and most languages are rapidly introducing
at least Option if not Result =E2=80=93 and yes it is almost certain all th=
is
comes from Haskell.

Is Option intended for adding to Phobos?

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Apr 21 2020
next sibling parent Alex <sascha.orlov gmail.com> writes:
On Tuesday, 21 April 2020 at 16:30:15 UTC, Russel Winder wrote:

 On Mon, 2020-04-20 at 20:19 +0000, aliak via 
 Digitalmars-d-learn wrote:

 
 […]



 [0]: https://github.com/aliak00/optional


 Rust has Option and Result, and most languages are rapidly 
 introducing at least Option if not Result – and yes it is 
 almost certain all this comes from Haskell.

 Is Option intended for adding to Phobos?


There is Nullable in Phobos. This is not really an Option, I 
know. But just in case.

But back to integration tests, there are indeed only few 
possibilities to mock something. However, the basics are there.
I found a section of code generation a while ago in std.typecons.
https://dlang.org/library/std/typecons.html

There are Black and White holes there, for example, which could 
help to implement a basic mocking framework.
Apr 21 2020
prev sibling parent reply aliak <something something.com> writes:
On Tuesday, 21 April 2020 at 16:30:15 UTC, Russel Winder wrote:

 On Mon, 2020-04-20 at 20:19 +0000, aliak via 
 Digitalmars-d-learn wrote:

 
 […]



 [0]: https://github.com/aliak00/optional


 Rust has Option and Result, and most languages are rapidly 
 introducing at least Option if not Result – and yes it is 
 almost certain all this comes from Haskell.


Yeah it's a great abstraction that's part of modern languages 
now. Some static languages are even building syntax for it right 
in (e.g. swift, kotlin, v, zig - to name a few).

There've been a few attempts at building a Result type:

https://code.dlang.org/search?q=expect
And here: 
https://github.com/aliak00/ddash/blob/master/utils/source/ddash/utils/expect.d


 Is Option intended for adding to Phobos?


Not that I am aware of. There was an attempt to PR an Option type 
way back when which never made it: 
https://github.com/dlang/phobos/pull/3915

There was a post here: 
https://forum.dlang.org/thread/hqtdekjtdgbhhbjgyvvo forum.dlang.org
Apr 21 2020
parent Russel Winder <russel winder.org.uk> writes:
On Tue, 2020-04-21 at 21:29 +0000, aliak via Digitalmars-d-learn wrote:

 On Tuesday, 21 April 2020 at 16:30:15 UTC, Russel Winder wrote:


[=E2=80=A6]

=20
 There've been a few attempts at building a Result type:
=20
 https://code.dlang.org/search?q=3Dexpect
 And here:=20
 https://github.com/aliak00/ddash/blob/master/utils/source/ddash/utils/exp=


ect.d

=20

 Is Option intended for adding to Phobos?


=20
 Not that I am aware of. There was an attempt to PR an Option type=20
 way back when which never made it:=20
 https://github.com/dlang/phobos/pull/3915
=20
 There was a post here:=20
 https://forum.dlang.org/thread/hqtdekjtdgbhhbjgyvvo forum.dlang.org


Perhaps there can be one implementation package of Nullable, Option,
and Result that is in the Dub repository that everyone can use even though =
they really ought to be part of Phobos.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Apr 23 2020
prev sibling parent reply Russel Winder <russel winder.org.uk> writes:
I ended up creating the following project structure:

.
=E2=94=9C=E2=94=80=E2=94=80 dub.sdl
=E2=94=9C=E2=94=80=E2=94=80 dub.selections.json
=E2=94=9C=E2=94=80=E2=94=80 source
=E2=94=82   =E2=94=9C=E2=94=80=E2=94=80 arcam_protocol.d
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 main.d
=E2=94=9C=E2=94=80=E2=94=80 tests
=E2=94=82   =E2=94=94=E2=94=80=E2=94=80 integration_tests.d
=E2=94=94=E2=94=80=E2=94=80 test_support
    =E2=94=94=E2=94=80=E2=94=80 mock_avr850
        =E2=94=94=E2=94=80=E2=94=80 main.d

with the following Dub control file:

name "arcamclient"
description "arcamclient is a Rust/gtk-rs/GTK+ desktop application to contr=
ol an Arcam amplifier over the Ethernet connection."
authors "Russel Winder"
copyright "Copyright =C2=A9 2020 Russel Winder."
license "GPL-3.0"
targetType "executable"
targetPath "bin"

configuration "application" {
}

configuration "unittest" {
    targetName "arcamclient_test"
    dependency "unit-threaded" version=3D"*"
    mainSourceFile "bin/ut.d"
    excludedSourceFiles "source/main.d"
    preBuildCommands "$DUB run --compiler=3D$$DC unit-threaded -c gen_ut_ma=
in -- -f bin/ut.d -d $DUB"
    preBuildCommands "$DUB build arcamclient:mock_avr850"
    importPaths "tests"
    sourcePaths "tests"
}

subPackage {
    name "mock_avr850"
    targetName "mock_avr850"
    targetType "executable"
    targetPath "bin"
    sourcePaths "source" "test_support/mock_avr850"
    importPaths "source" "test_support/mock_avr850"
    excludedSourceFiles "source/main.d"
}

This seems a bit more sensible that what I have been able to achieve
with Rust, but is still second rate compared to how easy things are
using Python.=20

Now I discover Python, Rust, and Go have far nicer abstractions for
writing Internet code than D does. Does D really not have a TcpListener
abstraction?

To date all I can get is:

std.socket.SocketOSException std/socket.d(2792): Unable to bind socket: Bad=
 file descriptor

when trying to open a TCP server on 127.0.0.1:50000, with Python, Rust,
or Go it all worked first time. This is really sad given D has so many
advantages over Rust. :-(

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
Apr 22 2020
next sibling parent reply aliak <something something.com> writes:
On Wednesday, 22 April 2020 at 10:32:48 UTC, Russel Winder wrote:

 Now I discover Python, Rust, and Go have far nicer abstractions 
 for writing Internet code than D does. Does D really not have a 
 TcpListener abstraction?


It really doesn't :(

And D has so much potential as server tech with the potential 
combination of fibers + TLS + shared + static introspection.

The package Vibe-d is quite nice though. I don't know if you've 
tried it but it's very simple to get a listener up with it.


 To date all I can get is:

 std.socket.SocketOSException std/socket.d(2792): Unable to bind 
 socket: Bad file descriptor

 when trying to open a TCP server on 127.0.0.1:50000, with 
 Python, Rust, or Go it all worked first time. This is really 
 sad given D has so many advantages over Rust. :-(
Apr 22 2020
parent Russel Winder <russel winder.org.uk> writes:
On Wed, 2020-04-22 at 11:19 +0000, aliak via Digitalmars-d-learn wrote:

 On Wednesday, 22 April 2020 at 10:32:48 UTC, Russel Winder wrote:

 Now I discover Python, Rust, and Go have far nicer abstractions=20
 for writing Internet code than D does. Does D really not have a=20
 TcpListener abstraction?


=20
 It really doesn't :(


:-(

Even GTK+ has it's own wrappers around the base socket API to make it sensi=
ble
for programmers. GtkD offers these for GtkD-based applications, but D has n=
o
language support for asynchronous (via Futures/Promises/event loops) which
leaves Rust (and Python) far ahead in this race to support asynchronous
programming.


 And D has so much potential as server tech with the potential=20
 combination of fibers + TLS + shared + static introspection.


Potential is necessary but not sufficient. Vibe.d seems to be one solution =
(in
the Rust Async_std and Tokio sense) but Rust has language level support for
Futures that make everything a lot easier in Rust than seemingly in D.


 The package Vibe-d is quite nice though. I don't know if you've=20
 tried it but it's very simple to get a listener up with it.


And now it seems we have Hunt. I am now dithering whether to use Vibe.d or
Hunt for my async TCP (but not HTTP(S)) server.

[=E2=80=A6]
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
May 11 2020
prev sibling parent Luis <luis.panadero gmail.com> writes:
On Wednesday, 22 April 2020 at 10:32:48 UTC, Russel Winder wrote:

 I ended up creating the following project structure:

 .
 ├── dub.sdl
 ├── dub.selections.json
 ├── source
 │   ├── arcam_protocol.d
 │   └── main.d
 ├── tests
 │   └── integration_tests.d
 └── test_support
     └── mock_avr850
         └── main.d

 with the following Dub control file:

 name "arcamclient"
 description "arcamclient is a Rust/gtk-rs/GTK+ desktop 
 application to control an Arcam amplifier over the Ethernet 
 connection."
 authors "Russel Winder"
 copyright "Copyright © 2020 Russel Winder."
 license "GPL-3.0"
 targetType "executable"
 targetPath "bin"

 configuration "application" {
 }

 configuration "unittest" {
     targetName "arcamclient_test"
     dependency "unit-threaded" version="*"
     mainSourceFile "bin/ut.d"
     excludedSourceFiles "source/main.d"
     preBuildCommands "$DUB run --compiler=$$DC unit-threaded -c 
 gen_ut_main -- -f bin/ut.d -d $DUB"
     preBuildCommands "$DUB build arcamclient:mock_avr850"
     importPaths "tests"
     sourcePaths "tests"
 }

 subPackage {
     name "mock_avr850"
     targetName "mock_avr850"
     targetType "executable"
     targetPath "bin"
     sourcePaths "source" "test_support/mock_avr850"
     importPaths "source" "test_support/mock_avr850"
     excludedSourceFiles "source/main.d"
 }

 This seems a bit more sensible that what I have been able to 
 achieve with Rust, but is still second rate compared to how 
 easy things are using Python.


Have you try Silly? I found far more straightforward to use, that 
unit-threaded.

On my pet game engine, I got it working with this :

"configurations": [
...
     {
         "dependencies": {
             "beep": "~>0.0.2",
             "silly": "~>1.0.2"
         },
         "name": "unittest",
         "targetType": "library"
     }
],

Sadly, Silly only is the test runner (and one far prettier that 
unit-threaded). I need to add separated dub module to make 
assertion more easy and fluent. Beep go on the correct path with 
a API like this :

 ("My test")
unittest {
     myFunc(1, 2, 3).expect!equals(42);
     auto a = [1, 2 ,3];
     a.expect!contains(2);
     a.lenght.expect!great(0);
     true.expect!true();
}

And the output on case of fail, shows a human friendly message 
with the expected value and the real value.

Another assertion module that not depends on unit-threaded and 
have better API, is pyjamas . But actually is abandoned. I did a 
try to get it working again, but depends on another dead dub 
module... another test runner called "bed" (that I did a PR to 
fix it to get it working against the actual DMD version).

Finally... What I really miss are two things :

- A test/integration framework like Spock (Yeah, I professionally 
  work on Java world :( )
- Test runners had a common output format that make IDE/tools 
work far more easy. See 
https://github.com/nomad-software/dunit/issues/19#issuecomment-435209223

Offtopic: Should be a way to mark a dub module dead on dub 
register. This last weeks I don't stop of finding dead/not 
working stuff and given a really bad image of the state of D.
May 11 2020