Grail42: single threaded test framework complete

Grail42 now provides its own independent implementation of the normalized API and output format that the TUT, gtest, and CppUTest test frameworks were adapted to provide by Grail42 already.  The unit_test template project also now defaults to utilizing Grail42’s own framework instead of TUT.

Since TUT and CppUTest are not thread safe and gtest is not thread safe on Windows platforms, the normalized API doesn’t account for threads, so the next step is to extend the API to provide the ability to create multi-threaded tests when using Grail42’s own framework rather than one of the three adapted frameworks.

A decent chunk of the required  implementation is already in place with only a few more C++ classes remaining to be implemented, and two of those aren’t required to be thread safe so they can be unit tested using Grail42’s own framework as a dogfooding exercise.

Posted in Coding | Tagged , , , | Leave a comment

Grail42: CppUTest adapter added

In addition to the Template Unit Test Framework (TUT) and Google Test (gtest), Grail42 Core now includes an adapter to normalize the use and output of CppUTest.  Since the point of the three adapters is to normalize the API and output of these C++ unit testing frameworks, the examples of usage and output provided on this post are relevant to CppUTest as well as TUT and gtest.

My next task is to add a template project to Grail42 Templates that starts each of the unit test frameworks on a separate thread while capturing their output and then serializes the output to stdout.  That template will enable creating a number of 3 threaded unit tests to used in the process of creating a new C++ unit testing framework that is thread safe.

Posted in Coding | Tagged , , , , | Leave a comment

Grail42: C++ Template Unit Test Framework Adapter

The Grail42 github project now contains adapters to normalize and simplify the use of the C++ unit testing frameworks Template Unit Test Framework (TUT) and Google Test (gtest) in Visual Studio.  The code has been tested to build for and execute on Windows 7, Android 2.2, and Chrome Native Client 23.  For the later two platforms the VS add-ins for those platforms were used, so the code is known to build with gcc as well as msvc.  An adapter for cpputest will be added shortly as well.

As TUT does not utilize any C preprocessor macros its test failure messages do not include filename and line number information.  On the other hand, the adapter macros and common reporter adapter that Grail42 supplies for TUT do include filenames and line numbers in failures messages.  The adapter macros are also result in more concise unit test code as demonstrated by the example below.

The benefits the adapter for gtest provide are less significant than for TUT.  Its primary purpose is actually just to facilitate the unit testing of a to be developed thread safe unit testing framework as mentioned here.

This is an example of the output provided by the Grail42 adapter for TUT or gtest for a single failing test:

*** Tests started ***
d:\code\prototypes\unit_test000\unit_test000.cpp(19): false
*** Tests complete: 0 passed, 1 failed, 0 skipped ***

versus the stock output of TUT for the same:

basic test: [1=F]

---> group: basic test, test: test
     problem: assertion failed

tests summary: failures:1 ok:0

This is the code using the Grail42 adapter for TUT or gtest to produce the above output:

#include <g42core/test/test.hpp>
#include <g42core/test/short_defines.hpp>
#include <g42core/test/main.hpp>

BEGIN_TESTS()

DEFINE_TEST()
{
    VERIFY(false);
}

END_TESTS()

int wmain(int, wchar_t* [])
{
    return G42CORE_TEST_RUN_TESTS();
}

versus code that directly uses TUT to accomplish the same:

#include <tut/tut.hpp>
#include <tut/tut_reporter.hpp>

namespace tut
{
    struct basic{};
    typedef test_group<basic> factory;
    typedef factory::object object;
}

namespace
{
    tut::factory tf("basic test");
}

namespace tut
{
    template<>
    template<>
    void object::test<1>()
    {
        ensure(false);
    }
}

int wmain(int argc, wchar_t* argv[])
{
    tut::reporter reporter;
    tut::test_runner_singleton::get().set_callback(&reporter);
    tut::test_runner_singleton::get().run_tests();
    return !reporter.all_ok();
}
Posted in Coding | Tagged , , , , , , , , , , , , | 1 Comment

Portability of 5 C++ Unit Test Frameworks to Android NDK and Chrome NaCl

After finishing up the Python script for cloning example projects that was mentioned in my last post, I was able to get back to cross-platform, multi-threaded C++ unit tests.  The next step was to 1) attempt building the 5 C++ unit testing frameworks in Visual Studio 2010 using the add-ins for the Android and Chrome Native Client (NaCl) platforms, and 2) get a simple test executing for each framework on each platform.

The detailed results follow.  Considering boost’s efforts toward wide portability, I was surprised to see that boost::unit_test was the only unit testing framework that failed to compile, link, and execute on Android.  I was also surprised to see that the last official source release of for Google Test didn’t run successfully on Chrome NaCl without a minor change.

To bootstrap the unit testing of the multi-threaded, multi-platform C++ unit test framework I am developing, I have goal of using a minimum of three frameworks on three platforms, since one framework is needed per thread.  When I added CppUTest to the set, it didn’t appear to be worthwhile.  However, CppUTest proved to be necessary in that it was the third of only three frameworks to work for the Chrome NaCl platform.

TUT / Template Unit Test Framework v2009-08-30

  • Android:  Successfully compiles, links and executes without changes.
  • Chrome Native Client:  Successfully compiles, links and executes without changes.

CppUTest v3.3

  • Android:  Successfully compiles, links and executes without changes.
  • Chrome Native Client:  Successfully compiles, links and executes without changes.

gtest / Google Test v1.6.0

  • Android:  Successfully compiles, links and executes without changes.
  • Chrome Native Client:  Official sources successfully compile and link without changes when targeting the newlib runtime, but fails to compile when targeting the glibc runtime.  There is a version the source on naclports, but I don’t know if that one builds successfully with the glibc runtime.  After the first test case is included, the resulting binary to execute due to the inability to get the current working directory (it executes fine when there are no test cases).  This can be resolved by excluding one block of code from a header file that appears specific to death tests, which is a feature of gtest I don’t need here.

UnitTest++ v1.4

  • Android:  Successfully compiles, links and executes without changes.
  • Chrome Native Client:  SignalTranslator.cpp doesn’t compile without pulling in some things from setjmp.h and signal.h as possibly because there aren’t enough instances of __native_client__ rather than __CYGWIN__.  However, once it compiled successfully it failed to link due to the missing symbol ‘sigaction’, which isn’t present in any of the newlib libraries, so the failure to compile looks to be accurate with respect to what the platform supports.

boost::unit_test / Boost Test Library v1.52.0

  • Android:  One minor change to a boost header was required to successfully compile, link, and execute.
  • Chrome Native Client:  Officially released sources do not compile successfully due to missing APIs.  There is a boost port in the naclports project but it doesn’t seem to be well supported, especially if the host OS isn’t isn’t Linux and apparently even if it is a version of Linux errors may still be encountered due to differences in the Linux variants.

General Notes

  • Only the project needed to build the framework’s library was updated to build the Android and NaCl platforms.
  • Only debug builds have been attempted thus far.  Typically debug builds use more APIs than release builds, so I’m currently assuming release will build if debug does.
  • For Android only the toolset “ARM 4.4.3 Toolchain (arm-linux-androideabi-4.4.3)” was used with a target architecture of “ARM Architecture v5te (armv5te)” with an API level of 9 (Android 2.3).
  • For Chrome Native Client only the newlib variant of the NaCl64 platform was attempted.  The one exception was gtest as noted above.  The tests were only executed using sel_ldr.
Posted in Coding | Tagged , , , , , , , , , , , , , | 2 Comments

Faux Console Application for Android NDK and Chrome Native Client SDK

While Android and the Chrome Native Client (NaCl) don’t actually provide support for console style applications, they are still useful for C++ unit testing.  As a second step towards developing cross-platform multi-threaded C++ unit tests, I developed a Visual Studio 2010 project that includes the platforms for Android and NaCl as well as the 32 and 64-bit versions of a Windows console application.  The VS project and source code can be directly browsed here.

To actually build and run the project, it is necessary to get the full Grail42 project and its subprojects Grail42 Core and Grail42 Templates, and install the Android NDK plug-in and the NaCl SDK plug-in for VS.  Note that the NaCl plug-in for VS requires a 64-bit version of Windows.

For NaCl platforms the project is configured so that it can run as a faux console application loaded into Chrome or be run standalone via the sel_ldr*.exe tool included in the NaCl SDK.

I completed the first step towards cross-platform multi-theaded unit tests, as explained here, after getting 5 threads running in a Windows console app where each thread was using a separate unit testing framework and all of the frameworks were configured to use a common thread safe results reporter.

Upon completing this first step, I thought it would be a good idea to ensure the code was portable and took a look the state of various cross platform build systems.  As I did not turn up any build systems that were generally better than the others*, and in my investigation I ran across the VS add-ins for the Android NDK and the Chrome Native Client SDK, I decided to stick with the build system I was most familiar (VS / msbuild.exe) to minimize the length of the detour I was taking away from developing the C++ unit testing framework.  It didn’t hurt that I had been considering investing in Android development anyway, and NaCl support OpenGL ES 2.0 so it provides the opportunity to write graphical code for Android, web pages, and possibly Windows as well (I suspect there may be an OpenGL ES 2.0 port for Windows, but I haven’t investigated yet).

The next step is to develop a Python script that can clone example projects so that they can be used as templates for new projects, and then I can get back to developing the framework for cross-platform, multi-threaded C++ unit tests.

* – certainly some build systems are better than others in specific situations or when measured against specific requirements.

Posted in Coding | Tagged , , , , , | 1 Comment

Google Test, pthreads, and Windows

As a quick follow-up to my last post, there is an existing library providing a pthreads implementation for Windows in the form of an OSS project named pthread-win32.  Unfortunately it appears a patch to Google Test is needed to successfully make use of pthread-win32 with Google Test on Windows and the issue tracking this hasn’t been resolved yet, so it would be necessary to modify Google Test to make it thread safe on Windows.

Posted in Coding | Tagged , , , | 1 Comment

C++ Unit Test Framework Adapter: Part 3

This is the third post in a 3 part series.  Part 2 is here.

I have completed reviewing all five C++ unit testing frameworks to determine if any could be made thread safe during the the execution of a test just by providing a thread safe custom formatter / reporter.  Unsurprisingly, none could and the details follow.

Google Test does have thread safety when pthreads are available on the target platform, which of course does not include Windows platforms.  Google Test does allow turning on pthreads support explicitly through a preprocessor define, so it might be possible to provide a compatible implementation of pthreads on Windows (or perhaps one exists already; Update 11/02/2012 – a Win32 pthreads library exists but gtest still needs some changes; details here).

If not, it will be necessary to modify one of the frameworks or create a new one in order to be able create multi-threaded unit tests in C++.  Either way, it should be possible to utilize one framework per thread to create multi-threaded tests that are constrained to one test per executable and 5 threads since all five framework can be utilized in the same module.  These limited multi-threaded unit tests can in turn used to test the custom multi-threaded formatter / reporter, the pthread implementation, and/or the modified or new thread safe unit testing framework.

I am also considering creating a new framework so that it can be designed to with threading as a first order feature, rather than merely being thread safe.  For example tests could be declared in parts with the logical id of the thread it should be executed on, rather than the tests needed to create threads directly themselves.

UnitTest++

  • Not thread safe:
    • The assertion macros utilize a single instance retrieved from a function that includes the instance as a static.  The reporter isn’t utilized in the function, so it has no opportunity to ensure the thread safety of the initialization of the static instance.
    • On failure, the assertion macros utilize member methods of the instance that modify a number of data members on the instance and reporter isn’t call in such a manner that locking in the reporter would serialize modification to these data members.

TUT / Template Unit Test Framework

  • Not multithreading compatible:
    • Assertion functions throw C++ exceptions that are caught by the framework method executing the test.
    • The test execution code of the framework does not provide a hook that could be utilized to propagate an exception from a child thread back to the main thread.

boost::unit_test / Boost Test Library

  • Not thread safe:
    • The assertion macros utilize a single instance of a class static data member to store file and line information on every usage (not just on assertion failure).
    • Member methods of the the above instance utilize a single instance retrieved from a function that includes the instance as a static.  The formatters aren’t utilized in the function, so it has no opportunity to ensure the thread safety of the initialization of the static instance.
    • The member methods of the instance retrieved from the function modify data members on the instance, but the formatters aren’t utilized, so it has no opportunity to ensure the thread safety of the modifications.

gtest / Google Test

  • Not thread safe on Windows platforms:
    • Has locks when pthreads are present and assumes pthreads are present on Linux, Mac, and HP-UX.
    • The threading related classes can not be overridden / supplied by the consumer of the framework.
  • There is function returning a static instance, so if the function isn’t called before any tests are run there is a possible threading issue there with respect to initialize the static.

CppUTest

 

  • Not thread safe:
    • The assertion macros utilize a instance that is either a data member of a instance that is a static in a function or a static class member.  The output instance is not utilized, so it has no opportunity to ensure the thread safety of the initialization of the static instances. The assertion macros utilize member methods on that instance that modify data members, and the output instance is not utilized in a manner that allows it to provide thread safety for the modifications to the members.
Posted in Coding | Tagged , , , , , , , , , | 4 Comments

C++ Unit Test Framework Adapter: Part 2

This is the second post in a 3 part series.  Part 1 is here and part 3 is here.

After reviewing the source for the four C++ unit testing frameworks with respect to if their output can be normalized without modifying the frameworks, I found that creating the normalized  output solution for Google Test would require parsing the error strings formed at a higher level of the framework’s stack.  In turn I decided to include CppUTest as a fifth framework, but it turned out that CppUTest actually had the same issue.

The code necessary to normalize the output for the 5 frameworks has been written and the points of interest regarding each framework follow.  Next up is to evaluate the implementation of assertions in four of the frameworks to determine if any of the framework’s can be made thread safe during the the execution of a test just by providing a thread safe custom formatter.  boost::unit_test has been evaluated already and it is necessary to modify that framework’s code for it to be the thread safe.

UnitTest++

  • Of the five frameworks, setting up the custom reporter for UnitTest++ was the quickest.
  • It has a simple reporter and the design of the reporter aligned well with my desired output format and content.

TUT / Template Unit Test Framework

  • Of the five frameworks, setting up the custom reporter for TUT was the second quickest.
  • Like UnitTest++ its reporter was simple, but its design was not aligned as closely with my desired output format and content.

boost::unit_test / Boost Test Library

  • Of the five frameworks, setting up the custom reporter for boost::unit_test was the most time consuming (by several factors).
  • It has two different formatters involved in output, and it allows overriding the stream that each formatter uses.
  • The current docs do not detail the use of custom formatters.  Since the code is also more complex, it was necessary to spend more time reviewing framework code.
  • Setting the formatter has to be done after other initialization or it will be overwritten by the framework.
  • It was not necessary to parse the assertion failure string reducing the fragility of the custom output solution.

gtest / Google Test

  • The custom printer solution has some fragility as it was necessary to parse the assertion failure strings to pull out the information of interest and disregard additional formatting added at a higher level in the framework’s stack.

CppUTest

  • The custom output solution has some fragility as it was necessary to parse the assertion failure strings to pull out the information of interest and disregard additional formatting added at a higher level in the framework’s stack.
Posted in Coding | Tagged , , , , , , , , , | 3 Comments

C++ Unit Test Framework Adapter: Part 1

This is the first post in a 3 part series.  Part 2 is here.

Earlier today I completed the implementation of an adapter that provides a common interface to four C++ unit testing frameworks:

I decided to begin work on Cloud Hydra in spite of Kickstarter not accepting the campaign and the Indiegogo campaign fizzling.  I have choosen C++ as the implementation language as it is the language, which I already know, that will provide portability across my target platforms, provide sufficiently high performance, provide sufficiently low resource utilization, doesn’t require productivity sacrifices that working in C requires, and is more desired for use in technical interviews.

As any production software needs a logging framework and I favor unit testing, a unit testing framework and mock object frameworks are needed first thing.  We were using boost::test at Mindspark, but that isn’t thread safe even in the success path and has no support for multi-processing.  Cloud Hydra will be multi-threaded and multi-processed.  After creating a list of requirements, I surveyed open source unit testing and logging frameworks.  While there aren’t numerous logging or mock object frameworks for C++, there are dozens of unit testing frameworks.

On a side note, I developed the unit testing and mock object framework that is part of the Visual Studio Library.  When development of VSL began Microsoft legal wouldn’t sign-off on including an open source unit testing framework in it, so it was necessary to create a new one.  The license terms for the Visual Studio SDK, of which VSL is  part, are ambiguous so it isn’t an option for me now and that is also the reason we didn’t use it at Mindspark.   Even if licensing weren’t an issue the unit testing framework in VSL doesn’t met all of my current requirements.

Unfortunately none of the OSS C++ unit testing frameworks met all of my requirements.  Two obvious options are modifying an existing framework or creating a new one.   Either way, I would like to be able unit test the new or existing unit testing framework, and I would prefer my unit tests for the required unit test framework have the same style as those for production code that utilize the required unit test framework.  Thus the idea of C++ unit test framework adapter was born.

Follows is a simple test of the verbose macros.  Several sets of macro aliases providing much short names are also included, but only the underlying verbose macros are tested.  In reviewing the code, you will note the lack of an identifying parameter on the *BEGIN_TESTS and *DEFINE_TEST macros, even though all four of the frameworks require identifiers be provided.  The adapter macros make use of a combination of the C processor, anonymous namespaces, and/or C++ template metaprogramming to automatically create unique identifiers for a test and groups of tests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
G42_TEST_BEGIN_TESTS()

G42_TEST_DEFINE_TEST()
{
}

G42_TEST_DEFINE_TEST()
{
	G42_TEST_IF_NOT_DEFAULT(true);
#if G42_TEST_IF_NOT_REPORT_ONLY
	G42_TEST_IF_NOT_REPORT_ONLY(true);
#endif
#if G42_TEST_IF_NOT_REPORT_TEST_ABORT
	G42_TEST_IF_NOT_REPORT_TEST_ABORT(true);
#endif
}

G42_TEST_END_TESTS()

G42_TEST_BEGIN_TESTS()
G42_TEST_DEFINE_TEST()
{
	G42_TEST_IF_NOT_DEFAULT(true);
}
G42_TEST_END_TESTS()

The technique of generating the id varies with each test, but the most difficult proved to be providing the compile-time constant integer values needed by TUT for individual tests since they are specified as specializations of a TUT defined template function.  The task was made more difficult when constraining the solution to utilize only standard language features and no implementation specific compiler extensions (e.g. __if_exists and __counter__).  This test code is repeated in two separate C++ files for each of the four frameworks.  This ensures there are not any multiply defined symbol errors between compilation units as well as with-in a compilation unit.

TUT is a header only library.  boost::test has a header that can be included in one and only one object file to avoid linking in a library.  Both gtest and UnitTest++ require linking in a library.  All four can be utilized in the same executable, although gtest and UnitTest++ do both define the TEST macro, but this isn’t terminal as the only object file both needed to be included in is the one with main (and even that is technically avoidable) and the macro can simple be undefined there if no tests are defined there.

All of the unit test frameworks except TUT produced errors formatted similar to compiler errors that allowed Visual Studio to navigate directly to source code line when the error line was double clicked in the output.  TUT doesn’t provide macros, so in turn it doesn’t generate errors with source filename and line number included in the output.  Even when a macro is created, the error message is appended to some TUT supplied text so it doesn’t conform to VS’s output parsing requirements.  Thus I did make one change to TUT code to put the error message on it’s own line.

Google Test provides verbose output about test execution even when there are no failures, where as both boost::test and UnitTest++ provide minimal output when there are no failures.

My next task is to see if the output of the 4 C++ unit testing frameworks can be normalized without modifying the frameworks.  I expect to utilize some of this error reporting code as the basis for a new logging framework that will in turn be used by the new unit testing framework that will be used to test the less foundational parts of the new logging framework.

Posted in About Me, Coding | Tagged , , , , , , , , , | 1 Comment

Project Cloud Hydra Indiegogo Postmortem

While my Cloud Hydra Indiegogo campaign isn’t over yet, it’s not only lacking traction but has gone off a cliff.  The first time I looked for the campaign by browsing through the Technology category, I found it on page 12.  The next day it was up to page 10, the next page 9, and then page 8 on the third and fourth full days.  However, on the fourth day of the campaign I update the brief and long descriptions and haven’t been able to find the project in the Technology category since then.  I actually went through all forty pages the first day and have check the first dozen several times since then.

As of now, the project has gotten 21 referrals through tracking links I’ve shared here, on Facebook, and Google+, but no direct views.  Since the campaign is no longer showing up when browsing the Technology category, I am assuming it will never get any direct views and in turn will not get any traction.

Lessons learned:

  • Regular updates (announcements) do appear to improve the positioning of a campaign in it’s category (I did an update of the first 4 days), as Indiegogo says it will.
  • Updating the brief and/or long descriptions may cause a campaign to no longer show up when browsing campaigns.
  • My social networking skills are insufficient to getting a campaign funded through my network.
Posted in Coding | Tagged , | Leave a comment