This is the first post in a 3 part series. Part 2 is here.
Earlier today I completed the implementation of an adapter that provides a common interface to four C++ unit testing frameworks:
I decided to begin work on Cloud Hydra in spite of Kickstarter not accepting the campaign and the Indiegogo campaign fizzling. I have choosen C++ as the implementation language as it is the language, which I already know, that will provide portability across my target platforms, provide sufficiently high performance, provide sufficiently low resource utilization, doesn’t require productivity sacrifices that working in C requires, and is more desired for use in technical interviews.
As any production software needs a logging framework and I favor unit testing, a unit testing framework and mock object frameworks are needed first thing. We were using boost::test at Mindspark, but that isn’t thread safe even in the success path and has no support for multi-processing. Cloud Hydra will be multi-threaded and multi-processed. After creating a list of requirements, I surveyed open source unit testing and logging frameworks. While there aren’t numerous logging or mock object frameworks for C++, there are dozens of unit testing frameworks.
On a side note, I developed the unit testing and mock object framework that is part of the Visual Studio Library. When development of VSL began Microsoft legal wouldn’t sign-off on including an open source unit testing framework in it, so it was necessary to create a new one. The license terms for the Visual Studio SDK, of which VSL is part, are ambiguous so it isn’t an option for me now and that is also the reason we didn’t use it at Mindspark. Even if licensing weren’t an issue the unit testing framework in VSL doesn’t met all of my current requirements.
Unfortunately none of the OSS C++ unit testing frameworks met all of my requirements. Two obvious options are modifying an existing framework or creating a new one. Either way, I would like to be able unit test the new or existing unit testing framework, and I would prefer my unit tests for the required unit test framework have the same style as those for production code that utilize the required unit test framework. Thus the idea of C++ unit test framework adapter was born.
Follows is a simple test of the verbose macros. Several sets of macro aliases providing much short names are also included, but only the underlying verbose macros are tested. In reviewing the code, you will note the lack of an identifying parameter on the *BEGIN_TESTS and *DEFINE_TEST macros, even though all four of the frameworks require identifiers be provided. The adapter macros make use of a combination of the C processor, anonymous namespaces, and/or C++ template metaprogramming to automatically create unique identifiers for a test and groups of tests.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
G42_TEST_BEGIN_TESTS()
G42_TEST_DEFINE_TEST()
{
}
G42_TEST_DEFINE_TEST()
{
G42_TEST_IF_NOT_DEFAULT(true);
#if G42_TEST_IF_NOT_REPORT_ONLY
G42_TEST_IF_NOT_REPORT_ONLY(true);
#endif
#if G42_TEST_IF_NOT_REPORT_TEST_ABORT
G42_TEST_IF_NOT_REPORT_TEST_ABORT(true);
#endif
}
G42_TEST_END_TESTS()
G42_TEST_BEGIN_TESTS()
G42_TEST_DEFINE_TEST()
{
G42_TEST_IF_NOT_DEFAULT(true);
}
G42_TEST_END_TESTS()
|
The technique of generating the id varies with each test, but the most difficult proved to be providing the compile-time constant integer values needed by TUT for individual tests since they are specified as specializations of a TUT defined template function. The task was made more difficult when constraining the solution to utilize only standard language features and no implementation specific compiler extensions (e.g. __if_exists and __counter__). This test code is repeated in two separate C++ files for each of the four frameworks. This ensures there are not any multiply defined symbol errors between compilation units as well as with-in a compilation unit.
TUT is a header only library. boost::test has a header that can be included in one and only one object file to avoid linking in a library. Both gtest and UnitTest++ require linking in a library. All four can be utilized in the same executable, although gtest and UnitTest++ do both define the TEST macro, but this isn’t terminal as the only object file both needed to be included in is the one with main (and even that is technically avoidable) and the macro can simple be undefined there if no tests are defined there.
All of the unit test frameworks except TUT produced errors formatted similar to compiler errors that allowed Visual Studio to navigate directly to source code line when the error line was double clicked in the output. TUT doesn’t provide macros, so in turn it doesn’t generate errors with source filename and line number included in the output. Even when a macro is created, the error message is appended to some TUT supplied text so it doesn’t conform to VS’s output parsing requirements. Thus I did make one change to TUT code to put the error message on it’s own line.
Google Test provides verbose output about test execution even when there are no failures, where as both boost::test and UnitTest++ provide minimal output when there are no failures.
My next task is to see if the output of the 4 C++ unit testing frameworks can be normalized without modifying the frameworks. I expect to utilize some of this error reporting code as the basis for a new logging framework that will in turn be used by the new unit testing framework that will be used to test the less foundational parts of the new logging framework.
Josh Heitzman on Codecrafting 