www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Approach to Integration Testing in D

reply Vijay Nayar <madric gmail.com> writes:
Greetings everyone,



What is your approach to integration testing in D? Do you use 
`unittest` blocks? Do you write stand-alone programs that 
interact with a running version of your program? Is there a 
library that makes certain kinds of testing easier?

For example, if I have a D project based on vibe.d, and I have 
custom converters to receive REST API request bodies in different 
formats based on the "Content-Type" HTTP header and other 
converter for the response based on the "Accepts" header, what is 
the best approach to test the entire program end-to-end?



Having done considerable work in Java using Spring and Spring 
Boot, it's very common to have [integration 
tests](https://softwaretestingfundamentals.com/integration-testing/), which the
Spring Boot framework makes quite easy to set up with annotations like
[ SpringBootTest](https://www.baeldung.com/spring-boot-testing#integration-testing-with-springboottest).

In a nutshell, Spring Boot is built on top of dependency 
injection, with a "Context" acting as a container for all the 
objects (called Beans) that are created and injected into that 
container. ` SpringBootTest` lets the user pick and choose which 
objects will be created for the test, and then when the program 
is run, only those objects are loaded. This allows one to do 
things like, start a complete web server environment, test by 
sending a request to some endpoint, and then validate that the 
response is what you would expect. This is especially helpful to 
validate that you are using the Spring Framework correctly, e.g. 
that custom converters you created that allow messages to be 
received in binary 
[protobuf](https://developers.google.com/protocol-buffers/) 
format instead of JSON work correctly.
Feb 04 2022
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/4/22 7:38 AM, Vijay Nayar wrote:

 Greetings everyone,
 

 
 What is your approach to integration testing in D? Do you use `unittest` 
 blocks? Do you write stand-alone programs that interact with a running 
 version of your program? Is there a library that makes certain kinds of 
 testing easier?
 
 For example, if I have a D project based on vibe.d, and I have custom 
 converters to receive REST API request bodies in different formats based 
 on the "Content-Type" HTTP header and other converter for the response 
 based on the "Accepts" header, what is the best approach to test the 
 entire program end-to-end?


I recently moved mysql-native integration tests to a subproject. I don't 
think unittesting should do integration tests *in the library itself*. 
But doing them via a unittesting framework in an external program seems 
to work well.

Links:
- common integration test project: 
https://github.com/mysql-d/mysql-native/tree/master/integration-tests
- vibe tests: 
https://github.com/mysql-d/mysql-native/tree/master/integration-tests-vibe
- phobos tests: 
https://github.com/mysql-d/mysql-native/tree/master/integration-tests-phobos

-Steve
Feb 04 2022
prev sibling next sibling parent Mathias LANG <geod24 gmail.com> writes:
On Friday, 4 February 2022 at 12:38:08 UTC, Vijay Nayar wrote:

 Greetings everyone,



 What is your approach to integration testing in D? Do you use 
 `unittest` blocks? Do you write stand-alone programs that 
 interact with a running version of your program? Is there a 
 library that makes certain kinds of testing easier?


For server nodes, I've built a library just for this:
https://github.com/Geod24/localrest

It required careful planning on the application side (e.g. we're 
not calling `runTask`, we have some classes for dependency 
injection), but allowed us to write "integration tests" in 
unittests block: 
https://github.com/bosagora/agora/tree/75967aac4f41272c6e5e8487c4066174825290e0/source/agora/test#agora-test-folder
Feb 04 2022
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Feb 04, 2022 at 12:38:08PM +0000, Vijay Nayar via Digitalmars-d-learn
wrote:
[...]

 What is your approach to integration testing in D? Do you use
 `unittest` blocks? Do you write stand-alone programs that interact
 with a running version of your program? Is there a library that makes
 certain kinds of testing easier?


Unittests are, by definition, *unit* tests, :-) meaning they are more
suitable for testing individual functions or modules, not really for
integration testing of the entire program.

Though in practice, the line is somewhat blurry, and I have written
unittests that do test functionality across modules at times. The key is
to write your code in a way that's amenable to testing, one principle of
which is to avoid dependency on global state (cf. dependency injection).
For example, a function that uses std.stdio.File could be made
unittest-able by parametrizing `File` as a template parameter, so that a
unittest block can inject a proxy type that performs the test without
actually touching the filesystem (which could cause unwanted
side-effects, esp. if the same file(s) are touched by multiple tests).



 For example, if I have a D project based on vibe.d, and I have custom
 converters to receive REST API request bodies in different formats
 based on the "Content-Type" HTTP header and other converter for the
 response based on the "Accepts" header, what is the best approach to
 test the entire program end-to-end?


Depending on how you structured your program, templatizing types used by
your program could make it unittestable on a large scale, e.g., if there
was a way to inject your own request/response types into the code. Then
unittests could just pass in mock request/response types containing test
data and test program logic that way.

In some cases, however, it may be difficult to do this on a program-wide
scale, so in many cases an external program tester may be necessary.  In
some of my projects I've written helper programs that read a directory
of test cases (which specifies program options, inputs, expected
outputs, etc.) and invokes the program and compares its output to the
expected output. This is hooked up to my build script so that it gets
run automatically upon each rebuild. Then adding a test case is just a
matter of adding some files to the directory and re-running the build.


T

-- 
They say that "guns don't kill people, people kill people." Well I think the
gun helps. If you just stood there and yelled BANG, I don't think you'd kill
too many people. -- Eddie Izzard, Dressed to Kill
Feb 04 2022
parent reply Vijay Nayar <madric gmail.com> writes:
On Friday, 4 February 2022 at 17:39:00 UTC, H. S. Teoh wrote:

 On Fri, Feb 04, 2022 at 12:38:08PM +0000, Vijay Nayar via 
 Digitalmars-d-learn wrote: [...]


I am still in the process of experimenting, but the advice on 
this thread has all been very helpful. Right now I'm 
experimenting by creating a separate "integration" `dub` 
configuration which uses `std.process : spawnProcess` to run my 
vibe.d-based web server and try out requests on it.

However, in any kind of large organization using a build server, 
even when there are multiple services/builds running in the same 
environment, it can be very inconvenient use a fixed port on 
which to run this server. Another running service, another 
person's build, or a test that ran and died without closing its 
sockets can lead to conflicts during development.

In order to solve that problem of port conflicts during 
integration testing, I converted a utility class from the Java 
Spring Framework into D and wanted to share it here: 
https://gist.github.com/vnayar/04c6172d9f9991062974585bb3ccc8a4

The usage is very simple, and can be used by integration tests to 
pick random free ports on which to run their tests. Here is one 
of the available methods:

```d
/**
  * Find an available TCP port randomly selected from the range
  * \[ [PORT_RANGE_MIN], [PORT_RANGE_MAX] \].
  * Returns: an available TCP port number
  * Throws: Exception if no available port could be found
  */
ushort findAvailableTcpPort() {
   return findAvailableTcpPort(PORT_RANGE_MIN);
}

unittest {
   foreach (ushort i; 0..10) {
     ushort port = findAvailableTcpPort();
     assert(port >= PORT_RANGE_MIN && port <= PORT_RANGE_MAX);
   }
}
```
Feb 06 2022
parent Vijay Nayar <madric gmail.com> writes:
On Sunday, 6 February 2022 at 17:36:05 UTC, Vijay Nayar wrote:

 On Friday, 4 February 2022 at 17:39:00 UTC, H. S. Teoh wrote:

 On Fri, Feb 04, 2022 at 12:38:08PM +0000, Vijay Nayar via




When working on a dub configuration needed to separately run 
integration tests that operate on the fully built program from 
the outside, treating it like a black-box, I had run into a 
number of problems because `dub` implicitly created 
configurations called `application` or `library` only if you have 
no configurations of your own defined. But as soon as I added my 
own configuration, those default configurations were no longer 
being created and I suddenly found myself seeing strange errors 
when trying to run unit-tests or builds.

See [GitHub Dub Issue 
https://github.com/dlang/dub/issues/1270).

However, I did finally navigate the tricky waters and make a 
configuration that does actually work for `dub build`, `dub test` 
and `dub test --config=integration`.

I thought I would share what I found here:

```sdl
authors "Vijay Nayar"
copyright "Copyright © 2019, Vijay Nayar"
description "Receives data from sources, converts it to 
protobufs, and delivers it in batches to stem."
license "proprietary"
name "mouth"
targetPath "target"


config if no


forces us



type, and



configuration "application" {
   targetName "mouth"
   targetType "executable"
   dependency "funnel:common" path="../"
   dependency "funnel:proto" path="../"
   dependency "nanomsg-wrapper" version="~>0.5.3"
   dependency "poodinis" version="~>8.0.3"
   dependency "vibe-d" version="~>0.9.4"

   mainSourceFile "source/app.d"
}


configuration "integration" {
   targetName "mouth_integration"

   targetType "executable"

explicitly

   excludedSourceFiles "source/*"

   sourcePaths "integration"
   importPaths "integration"

   preBuildCommands "echo IMPORT_PATHS=$$IMPORT_PATHS"
   preRunCommands "cd $PACKAGE_DIR ; dub build"
}
```
Feb 06 2022