www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D Unittest shortcomings with DLLs

reply Benjamin Thaut <code benjamin-thaut.de> writes:
I'm currently in the process of annotating all of phobos with "export" 
and its quite cumbersome. To verify that I annoted all relevant 
functions and types with export I would like to run the unitests against 
the shared version of phobos. There is a problem with this though. The 
unittests are always compiled into the same binary the modules are in so 
the unittests would end up in the phobos.dll and would not test phobos 
across a shared library boundary. It would also be very usefull to have 
this verifycation for the future when DLL support is officially in and 
new features need to be tested if they are propperly annotated with 
"export". Once "export" is used for shared library symbol visibility on 
non-windows as well this will be an issue on all pattforms.

The best solution would be if you could simply compile the unittests and 
just the unittests into a executable. This would require some compiler 
modifications, but even when done would cause some problems:
- Unittests may use private functions / types of a module. Private 
functions and types are however not exported from an shared library so 
this would require making a lot of private functions / types public just 
to be able to run the unittests.

Instead of building this into the compiler you could use a tool which 
extracts all "unittest { }" blocks out of all source files and puts them 
into new .d files. Then compile these into an executable. Now the code 
would not even compile because private symbols would not be accessible.

Any suggestions how to fix this issue? I'm also open for implementation 
hints.

Kind Regards
Benjamin Thaut
Mar 03 2015
next sibling parent reply "TheFlyingFiddle" <kurtyan student.chalmers.se> writes:
On Tuesday, 3 March 2015 at 17:49:07 UTC, Benjamin Thaut wrote:

 Any suggestions how to fix this issue? I'm also open for 
 implementation hints.

 Kind Regards
 Benjamin Thaut


Running unittests that access private symbols from the other side 
of the dll boundary sounds like a very hard problem to solve. On 
one hand you want to be able to access the symbols in order to 
run the unittests and on the other hand they should be private to 
clients of the api. The question is do we even want to run 
unittests that test private implementation on the other side of 
the dll boundery at all? These unittests test functionality the 
users of the dll are unable to access. As such they are not 
actually testing if symbols that should be exported are. However 
unittests that test the public interface of a module do. I think 
that any tool that is created should only care about the 
unittests which are testing the public interface of the module. 
Since this is the interface users of the dll will interact with.

Given the above, a tool will have to be able to identify if a 
unittest tests a public interface of a module and only extract 
the unittests that do. Since these unittests only test the public 
interface they should be able to be compiled if the programmer 
has remembered to export all the public interfaces.

You still have a problem if the unittest is actually testing the 
public interfaces but makes use of some private helper function 
in the process. Since this would not be identified by the tool as 
a public interface-testing unittest.
This problem would not matter if unittest that test the public 
interface are written in another testing module. In my opinion 
these kinds of unittests that test the public interface should 
always be written in another module to ensure that they don't 
unintentionally make use of private symbols.

Finding out if a unittest only accesses public symbols could be 
done by analyzing the ast of the method. Either inside the 
compiler of via one of the third party D parsers currently in use.

I hope that this reply was somewhat helpful to you.
Mar 04 2015
parent "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Thursday, 5 March 2015 at 03:24:05 UTC, TheFlyingFiddle wrote:


 Finding out if a unittest only accesses public symbols could be 
 done by analyzing the ast of the method. Either inside the 
 compiler of via one of the third party D parsers currently in 
 use.


I thought about writing a tool which extracts all "unittest" 
blocks and puts each one into a template. Then I could use 
__pragma(compiles, ) on the template to check if it uses any 
private symbol and if not instanciate that template. That way 
unittests that use private functions would automatically be 
filtered out.

Kind Regards
Benjamin Thaut
Mar 04 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-03-03 18:49, Benjamin Thaut wrote:

 I'm currently in the process of annotating all of phobos with "export"
 and its quite cumbersome. To verify that I annoted all relevant
 functions and types with export I would like to run the unitests against
 the shared version of phobos. There is a problem with this though. The
 unittests are always compiled into the same binary the modules are in so
 the unittests would end up in the phobos.dll and would not test phobos
 across a shared library boundary. It would also be very usefull to have
 this verifycation for the future when DLL support is officially in and
 new features need to be tested if they are propperly annotated with
 "export". Once "export" is used for shared library symbol visibility on
 non-windows as well this will be an issue on all pattforms.

 The best solution would be if you could simply compile the unittests and
 just the unittests into a executable. This would require some compiler
 modifications, but even when done would cause some problems:
 - Unittests may use private functions / types of a module. Private
 functions and types are however not exported from an shared library so
 this would require making a lot of private functions / types public just
 to be able to run the unittests.

 Instead of building this into the compiler you could use a tool which
 extracts all "unittest { }" blocks out of all source files and puts them
 into new .d files. Then compile these into an executable. Now the code
 would not even compile because private symbols would not be accessible.

 Any suggestions how to fix this issue? I'm also open for implementation
 hints.


Sounds like you want to test an application that uses Phobos as a DLL. 
To me that sounds like a more higher level test than a unit test, i.e. 
integration tests. I would put those tests in completely separate files, 
in a separate testing directory. Then it should be no problem to compile 
the parts separately.

As for private functions. That is more a question of if you really 
should write specific test for those or just test the public API. If you 
really want to, you can bypass the protection using pointers.

BTW, I think we need to have integration tests in general. Testing how 
different parts/modules interact with each other.

-- 
/Jacob Carlborg
Mar 04 2015
parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Thursday, 5 March 2015 at 07:28:12 UTC, Jacob Carlborg wrote:

 On 2015-03-03 18:49, Benjamin Thaut wrote:

 Sounds like you want to test an application that uses Phobos as 
 a DLL. To me that sounds like a more higher level test than a 
 unit test, i.e. integration tests. I would put those tests in 
 completely separate files, in a separate testing directory. 
 Then it should be no problem to compile the parts separately.

 As for private functions. That is more a question of if you 
 really should write specific test for those or just test the 
 public API. If you really want to, you can bypass the 
 protection using pointers.

 BTW, I think we need to have integration tests in general. 
 Testing how different parts/modules interact with each other.


I don't want to write new tests. I want to use the tests for 
phobos which are already there. And I want to use them in a way, 
that if new tests are added to phobos they are also tested 
against the dll version of phobos. Yes integration tests would be 
nice, but the PR for Dll support is already going to be huge 
without me adding integration tests.
Mar 04 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-03-05 08:38, Benjamin Thaut wrote:


 I don't want to write new tests. I want to use the tests for phobos
 which are already there. And I want to use them in a way, that if new
 tests are added to phobos they are also tested against the dll version
 of phobos. Yes integration tests would be nice, but the PR for Dll
 support is already going to be huge without me adding integration tests.


Wouldn't it require more code to write a tool that extracts the unittest 
blocks?

-- 
/Jacob Carlborg
Mar 04 2015
parent "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Thursday, 5 March 2015 at 07:42:10 UTC, Jacob Carlborg wrote:

 On 2015-03-05 08:38, Benjamin Thaut wrote:


 I don't want to write new tests. I want to use the tests for 
 phobos
 which are already there. And I want to use them in a way, that 
 if new
 tests are added to phobos they are also tested against the dll 
 version
 of phobos. Yes integration tests would be nice, but the PR for 
 Dll
 support is already going to be huge without me adding 
 integration tests.


 Wouldn't it require more code to write a tool that extracts the 
 unittest blocks?


I've written a simple D Tokenizer in a few lines of D code 
already. Building on that it would be pretty easy to extract 
unittest blocks.
Mar 04 2015