This is the third post in a 3 part series. Part 2 is here.
I have completed reviewing all five C++ unit testing frameworks to determine if any could be made thread safe during the the execution of a test just by providing a thread safe custom formatter / reporter. Unsurprisingly, none could and the details follow.
Google Test does have thread safety when pthreads are available on the target platform, which of course does not include Windows platforms. Google Test does allow turning on pthreads support explicitly through a preprocessor define, so it might be possible to provide a compatible implementation of pthreads on Windows (or perhaps one exists already; Update 11/02/2012 – a Win32 pthreads library exists but gtest still needs some changes; details here).
If not, it will be necessary to modify one of the frameworks or create a new one in order to be able create multi-threaded unit tests in C++. Either way, it should be possible to utilize one framework per thread to create multi-threaded tests that are constrained to one test per executable and 5 threads since all five framework can be utilized in the same module. These limited multi-threaded unit tests can in turn used to test the custom multi-threaded formatter / reporter, the pthread implementation, and/or the modified or new thread safe unit testing framework.
I am also considering creating a new framework so that it can be designed to with threading as a first order feature, rather than merely being thread safe. For example tests could be declared in parts with the logical id of the thread it should be executed on, rather than the tests needed to create threads directly themselves.
- Not thread safe:
- The assertion macros utilize a single instance retrieved from a function that includes the instance as a static. The reporter isn’t utilized in the function, so it has no opportunity to ensure the thread safety of the initialization of the static instance.
- On failure, the assertion macros utilize member methods of the instance that modify a number of data members on the instance and reporter isn’t call in such a manner that locking in the reporter would serialize modification to these data members.
TUT / Template Unit Test Framework
- Not multithreading compatible:
- Assertion functions throw C++ exceptions that are caught by the framework method executing the test.
- The test execution code of the framework does not provide a hook that could be utilized to propagate an exception from a child thread back to the main thread.
boost::unit_test / Boost Test Library
- Not thread safe:
- The assertion macros utilize a single instance of a class static data member to store file and line information on every usage (not just on assertion failure).
- Member methods of the the above instance utilize a single instance retrieved from a function that includes the instance as a static. The formatters aren’t utilized in the function, so it has no opportunity to ensure the thread safety of the initialization of the static instance.
- The member methods of the instance retrieved from the function modify data members on the instance, but the formatters aren’t utilized, so it has no opportunity to ensure the thread safety of the modifications.
gtest / Google Test
- Not thread safe on Windows platforms:
- Has locks when pthreads are present and assumes pthreads are present on Linux, Mac, and HP-UX.
- The threading related classes can not be overridden / supplied by the consumer of the framework.
- There is function returning a static instance, so if the function isn’t called before any tests are run there is a possible threading issue there with respect to initialize the static.
- Not thread safe:
- The assertion macros utilize a instance that is either a data member of a instance that is a static in a function or a static class member. The output instance is not utilized, so it has no opportunity to ensure the thread safety of the initialization of the static instances. The assertion macros utilize member methods on that instance that modify data members, and the output instance is not utilized in a manner that allows it to provide thread safety for the modifications to the members.