Ding! 5 years

Five year anniversary of this blog.  ‘nough said.

Posted in Uncategorized | 2 Comments

Grail42: test framework done enough for now

Now that the Grail42 C++ test framework has some basic documentation, it is time to move on to a new project.  The framework is by no means ‘done’ as I have numerous enhancements in mind for it and there is definitely some clean-up needed on the internal details.  That said, it is sufficiently done to allow easily creating C++ unit tests that are multi-threaded and multi-platform.  In turn I’m going to prioritize future enhancements based on the specific needs of other projects.

Posted in Coding | Tagged , , , | Leave a comment

Grail42: multi-threaded test support

Grail42 now includes support for multi-threaded tests.  A multi-threaded test is defined by multiple test part definitions that utilize the same test name.  The framework automatically executes each test part on its own thread and routes any error messages to stdout in a thread safe manner.

Next up is to update the unit_test template project to include a multi-threaded test and create some basic documentation.

Posted in Coding | Tagged , , , | Leave a comment

Grail42: single threaded test framework complete

Grail42 now provides its own independent implementation of the normalized API and output format that the TUT, gtest, and CppUTest test frameworks were adapted to provide by Grail42 already.  The unit_test template project also now defaults to utilizing Grail42’s own framework instead of TUT.

Since TUT and CppUTest are not thread safe and gtest is not thread safe on Windows platforms, the normalized API doesn’t account for threads, so the next step is to extend the API to provide the ability to create multi-threaded tests when using Grail42’s own framework rather than one of the three adapted frameworks.

A decent chunk of the required  implementation is already in place with only a few more C++ classes remaining to be implemented, and two of those aren’t required to be thread safe so they can be unit tested using Grail42’s own framework as a dogfooding exercise.

Posted in Coding | Tagged , , , | Leave a comment

Grail42: CppUTest adapter added

In addition to the Template Unit Test Framework (TUT) and Google Test (gtest), Grail42 Core now includes an adapter to normalize the use and output of CppUTest.  Since the point of the three adapters is to normalize the API and output of these C++ unit testing frameworks, the examples of usage and output provided on this post are relevant to CppUTest as well as TUT and gtest.

My next task is to add a template project to Grail42 Templates that starts each of the unit test frameworks on a separate thread while capturing their output and then serializes the output to stdout.  That template will enable creating a number of 3 threaded unit tests to used in the process of creating a new C++ unit testing framework that is thread safe.

Posted in Coding | Tagged , , , , | Leave a comment

Grail42: C++ Template Unit Test Framework Adapter

The Grail42 github project now contains adapters to normalize and simplify the use of the C++ unit testing frameworks Template Unit Test Framework (TUT) and Google Test (gtest) in Visual Studio.  The code has been tested to build for and execute on Windows 7, Android 2.2, and Chrome Native Client 23.  For the later two platforms the VS add-ins for those platforms were used, so the code is known to build with gcc as well as msvc.  An adapter for cpputest will be added shortly as well.

As TUT does not utilize any C preprocessor macros its test failure messages do not include filename and line number information.  On the other hand, the adapter macros and common reporter adapter that Grail42 supplies for TUT do include filenames and line numbers in failures messages.  The adapter macros are also result in more concise unit test code as demonstrated by the example below.

The benefits the adapter for gtest provide are less significant than for TUT.  Its primary purpose is actually just to facilitate the unit testing of a to be developed thread safe unit testing framework as mentioned here.

This is an example of the output provided by the Grail42 adapter for TUT or gtest for a single failing test:

*** Tests started ***
d:\code\prototypes\unit_test000\unit_test000.cpp(19): false
*** Tests complete: 0 passed, 1 failed, 0 skipped ***

versus the stock output of TUT for the same:

basic test: [1=F]

---> group: basic test, test: test
     problem: assertion failed

tests summary: failures:1 ok:0

This is the code using the Grail42 adapter for TUT or gtest to produce the above output:

#include <g42core/test/test.hpp>
#include <g42core/test/short_defines.hpp>
#include <g42core/test/main.hpp>

BEGIN_TESTS()

DEFINE_TEST()
{
    VERIFY(false);
}

END_TESTS()

int wmain(int, wchar_t* [])
{
    return G42CORE_TEST_RUN_TESTS();
}

versus code that directly uses TUT to accomplish the same:

#include <tut/tut.hpp>
#include <tut/tut_reporter.hpp>

namespace tut
{
    struct basic{};
    typedef test_group<basic> factory;
    typedef factory::object object;
}

namespace
{
    tut::factory tf("basic test");
}

namespace tut
{
    template<>
    template<>
    void object::test<1>()
    {
        ensure(false);
    }
}

int wmain(int argc, wchar_t* argv[])
{
    tut::reporter reporter;
    tut::test_runner_singleton::get().set_callback(&reporter);
    tut::test_runner_singleton::get().run_tests();
    return !reporter.all_ok();
}
Posted in Coding | Tagged , , , , , , , , , , , , | 1 Comment

Portability of 5 C++ Unit Test Frameworks to Android NDK and Chrome NaCl

After finishing up the Python script for cloning example projects that was mentioned in my last post, I was able to get back to cross-platform, multi-threaded C++ unit tests.  The next step was to 1) attempt building the 5 C++ unit testing frameworks in Visual Studio 2010 using the add-ins for the Android and Chrome Native Client (NaCl) platforms, and 2) get a simple test executing for each framework on each platform.

The detailed results follow.  Considering boost’s efforts toward wide portability, I was surprised to see that boost::unit_test was the only unit testing framework that failed to compile, link, and execute on Android.  I was also surprised to see that the last official source release of for Google Test didn’t run successfully on Chrome NaCl without a minor change.

To bootstrap the unit testing of the multi-threaded, multi-platform C++ unit test framework I am developing, I have goal of using a minimum of three frameworks on three platforms, since one framework is needed per thread.  When I added CppUTest to the set, it didn’t appear to be worthwhile.  However, CppUTest proved to be necessary in that it was the third of only three frameworks to work for the Chrome NaCl platform.

TUT / Template Unit Test Framework v2009-08-30

  • Android:  Successfully compiles, links and executes without changes.
  • Chrome Native Client:  Successfully compiles, links and executes without changes.

CppUTest v3.3

  • Android:  Successfully compiles, links and executes without changes.
  • Chrome Native Client:  Successfully compiles, links and executes without changes.

gtest / Google Test v1.6.0

  • Android:  Successfully compiles, links and executes without changes.
  • Chrome Native Client:  Official sources successfully compile and link without changes when targeting the newlib runtime, but fails to compile when targeting the glibc runtime.  There is a version the source on naclports, but I don’t know if that one builds successfully with the glibc runtime.  After the first test case is included, the resulting binary to execute due to the inability to get the current working directory (it executes fine when there are no test cases).  This can be resolved by excluding one block of code from a header file that appears specific to death tests, which is a feature of gtest I don’t need here.

UnitTest++ v1.4

  • Android:  Successfully compiles, links and executes without changes.
  • Chrome Native Client:  SignalTranslator.cpp doesn’t compile without pulling in some things from setjmp.h and signal.h as possibly because there aren’t enough instances of __native_client__ rather than __CYGWIN__.  However, once it compiled successfully it failed to link due to the missing symbol ‘sigaction’, which isn’t present in any of the newlib libraries, so the failure to compile looks to be accurate with respect to what the platform supports.

boost::unit_test / Boost Test Library v1.52.0

  • Android:  One minor change to a boost header was required to successfully compile, link, and execute.
  • Chrome Native Client:  Officially released sources do not compile successfully due to missing APIs.  There is a boost port in the naclports project but it doesn’t seem to be well supported, especially if the host OS isn’t isn’t Linux and apparently even if it is a version of Linux errors may still be encountered due to differences in the Linux variants.

General Notes

  • Only the project needed to build the framework’s library was updated to build the Android and NaCl platforms.
  • Only debug builds have been attempted thus far.  Typically debug builds use more APIs than release builds, so I’m currently assuming release will build if debug does.
  • For Android only the toolset “ARM 4.4.3 Toolchain (arm-linux-androideabi-4.4.3)” was used with a target architecture of “ARM Architecture v5te (armv5te)” with an API level of 9 (Android 2.3).
  • For Chrome Native Client only the newlib variant of the NaCl64 platform was attempted.  The one exception was gtest as noted above.  The tests were only executed using sel_ldr.
Posted in Coding | Tagged , , , , , , , , , , , , , | 2 Comments