Faux Console Application for Android NDK and Chrome Native Client SDK

While Android and the Chrome Native Client (NaCl) don’t actually provide support for console style applications, they are still useful for C++ unit testing.  As a second step towards developing cross-platform multi-threaded C++ unit tests, I developed a Visual Studio 2010 project that includes the platforms for Android and NaCl as well as the 32 and 64-bit versions of a Windows console application.  The VS project and source code can be directly browsed here.

To actually build and run the project, it is necessary to get the full Grail42 project and its subprojects Grail42 Core and Grail42 Templates, and install the Android NDK plug-in and the NaCl SDK plug-in for VS.  Note that the NaCl plug-in for VS requires a 64-bit version of Windows.

For NaCl platforms the project is configured so that it can run as a faux console application loaded into Chrome or be run standalone via the sel_ldr*.exe tool included in the NaCl SDK.

I completed the first step towards cross-platform multi-theaded unit tests, as explained here, after getting 5 threads running in a Windows console app where each thread was using a separate unit testing framework and all of the frameworks were configured to use a common thread safe results reporter.

Upon completing this first step, I thought it would be a good idea to ensure the code was portable and took a look the state of various cross platform build systems.  As I did not turn up any build systems that were generally better than the others*, and in my investigation I ran across the VS add-ins for the Android NDK and the Chrome Native Client SDK, I decided to stick with the build system I was most familiar (VS / msbuild.exe) to minimize the length of the detour I was taking away from developing the C++ unit testing framework.  It didn’t hurt that I had been considering investing in Android development anyway, and NaCl support OpenGL ES 2.0 so it provides the opportunity to write graphical code for Android, web pages, and possibly Windows as well (I suspect there may be an OpenGL ES 2.0 port for Windows, but I haven’t investigated yet).

The next step is to develop a Python script that can clone example projects so that they can be used as templates for new projects, and then I can get back to developing the framework for cross-platform, multi-threaded C++ unit tests.

* – certainly some build systems are better than others in specific situations or when measured against specific requirements.

Advertisements
Posted in Coding | Tagged , , , , , | 1 Comment

Google Test, pthreads, and Windows

As a quick follow-up to my last post, there is an existing library providing a pthreads implementation for Windows in the form of an OSS project named pthread-win32.  Unfortunately it appears a patch to Google Test is needed to successfully make use of pthread-win32 with Google Test on Windows and the issue tracking this hasn’t been resolved yet, so it would be necessary to modify Google Test to make it thread safe on Windows.

Posted in Coding | Tagged , , , | 1 Comment

C++ Unit Test Framework Adapter: Part 3

This is the third post in a 3 part series.  Part 2 is here.

I have completed reviewing all five C++ unit testing frameworks to determine if any could be made thread safe during the the execution of a test just by providing a thread safe custom formatter / reporter.  Unsurprisingly, none could and the details follow.

Google Test does have thread safety when pthreads are available on the target platform, which of course does not include Windows platforms.  Google Test does allow turning on pthreads support explicitly through a preprocessor define, so it might be possible to provide a compatible implementation of pthreads on Windows (or perhaps one exists already; Update 11/02/2012 – a Win32 pthreads library exists but gtest still needs some changes; details here).

If not, it will be necessary to modify one of the frameworks or create a new one in order to be able create multi-threaded unit tests in C++.  Either way, it should be possible to utilize one framework per thread to create multi-threaded tests that are constrained to one test per executable and 5 threads since all five framework can be utilized in the same module.  These limited multi-threaded unit tests can in turn used to test the custom multi-threaded formatter / reporter, the pthread implementation, and/or the modified or new thread safe unit testing framework.

I am also considering creating a new framework so that it can be designed to with threading as a first order feature, rather than merely being thread safe.  For example tests could be declared in parts with the logical id of the thread it should be executed on, rather than the tests needed to create threads directly themselves.

UnitTest++

  • Not thread safe:
    • The assertion macros utilize a single instance retrieved from a function that includes the instance as a static.  The reporter isn’t utilized in the function, so it has no opportunity to ensure the thread safety of the initialization of the static instance.
    • On failure, the assertion macros utilize member methods of the instance that modify a number of data members on the instance and reporter isn’t call in such a manner that locking in the reporter would serialize modification to these data members.

TUT / Template Unit Test Framework

  • Not multithreading compatible:
    • Assertion functions throw C++ exceptions that are caught by the framework method executing the test.
    • The test execution code of the framework does not provide a hook that could be utilized to propagate an exception from a child thread back to the main thread.

boost::unit_test / Boost Test Library

  • Not thread safe:
    • The assertion macros utilize a single instance of a class static data member to store file and line information on every usage (not just on assertion failure).
    • Member methods of the the above instance utilize a single instance retrieved from a function that includes the instance as a static.  The formatters aren’t utilized in the function, so it has no opportunity to ensure the thread safety of the initialization of the static instance.
    • The member methods of the instance retrieved from the function modify data members on the instance, but the formatters aren’t utilized, so it has no opportunity to ensure the thread safety of the modifications.

gtest / Google Test

  • Not thread safe on Windows platforms:
    • Has locks when pthreads are present and assumes pthreads are present on Linux, Mac, and HP-UX.
    • The threading related classes can not be overridden / supplied by the consumer of the framework.
  • There is function returning a static instance, so if the function isn’t called before any tests are run there is a possible threading issue there with respect to initialize the static.

CppUTest

 

  • Not thread safe:
    • The assertion macros utilize a instance that is either a data member of a instance that is a static in a function or a static class member.  The output instance is not utilized, so it has no opportunity to ensure the thread safety of the initialization of the static instances. The assertion macros utilize member methods on that instance that modify data members, and the output instance is not utilized in a manner that allows it to provide thread safety for the modifications to the members.
Posted in Coding | Tagged , , , , , , , , , | 4 Comments

C++ Unit Test Framework Adapter: Part 2

This is the second post in a 3 part series.  Part 1 is here and part 3 is here.

After reviewing the source for the four C++ unit testing frameworks with respect to if their output can be normalized without modifying the frameworks, I found that creating the normalized  output solution for Google Test would require parsing the error strings formed at a higher level of the framework’s stack.  In turn I decided to include CppUTest as a fifth framework, but it turned out that CppUTest actually had the same issue.

The code necessary to normalize the output for the 5 frameworks has been written and the points of interest regarding each framework follow.  Next up is to evaluate the implementation of assertions in four of the frameworks to determine if any of the framework’s can be made thread safe during the the execution of a test just by providing a thread safe custom formatter.  boost::unit_test has been evaluated already and it is necessary to modify that framework’s code for it to be the thread safe.

UnitTest++

  • Of the five frameworks, setting up the custom reporter for UnitTest++ was the quickest.
  • It has a simple reporter and the design of the reporter aligned well with my desired output format and content.

TUT / Template Unit Test Framework

  • Of the five frameworks, setting up the custom reporter for TUT was the second quickest.
  • Like UnitTest++ its reporter was simple, but its design was not aligned as closely with my desired output format and content.

boost::unit_test / Boost Test Library

  • Of the five frameworks, setting up the custom reporter for boost::unit_test was the most time consuming (by several factors).
  • It has two different formatters involved in output, and it allows overriding the stream that each formatter uses.
  • The current docs do not detail the use of custom formatters.  Since the code is also more complex, it was necessary to spend more time reviewing framework code.
  • Setting the formatter has to be done after other initialization or it will be overwritten by the framework.
  • It was not necessary to parse the assertion failure string reducing the fragility of the custom output solution.

gtest / Google Test

  • The custom printer solution has some fragility as it was necessary to parse the assertion failure strings to pull out the information of interest and disregard additional formatting added at a higher level in the framework’s stack.

CppUTest

  • The custom output solution has some fragility as it was necessary to parse the assertion failure strings to pull out the information of interest and disregard additional formatting added at a higher level in the framework’s stack.
Posted in Coding | Tagged , , , , , , , , , | 3 Comments

C++ Unit Test Framework Adapter: Part 1

This is the first post in a 3 part series.  Part 2 is here.

Earlier today I completed the implementation of an adapter that provides a common interface to four C++ unit testing frameworks:

I decided to begin work on Cloud Hydra in spite of Kickstarter not accepting the campaign and the Indiegogo campaign fizzling.  I have choosen C++ as the implementation language as it is the language, which I already know, that will provide portability across my target platforms, provide sufficiently high performance, provide sufficiently low resource utilization, doesn’t require productivity sacrifices that working in C requires, and is more desired for use in technical interviews.

As any production software needs a logging framework and I favor unit testing, a unit testing framework and mock object frameworks are needed first thing.  We were using boost::test at Mindspark, but that isn’t thread safe even in the success path and has no support for multi-processing.  Cloud Hydra will be multi-threaded and multi-processed.  After creating a list of requirements, I surveyed open source unit testing and logging frameworks.  While there aren’t numerous logging or mock object frameworks for C++, there are dozens of unit testing frameworks.

On a side note, I developed the unit testing and mock object framework that is part of the Visual Studio Library.  When development of VSL began Microsoft legal wouldn’t sign-off on including an open source unit testing framework in it, so it was necessary to create a new one.  The license terms for the Visual Studio SDK, of which VSL is  part, are ambiguous so it isn’t an option for me now and that is also the reason we didn’t use it at Mindspark.   Even if licensing weren’t an issue the unit testing framework in VSL doesn’t met all of my current requirements.

Unfortunately none of the OSS C++ unit testing frameworks met all of my requirements.  Two obvious options are modifying an existing framework or creating a new one.   Either way, I would like to be able unit test the new or existing unit testing framework, and I would prefer my unit tests for the required unit test framework have the same style as those for production code that utilize the required unit test framework.  Thus the idea of C++ unit test framework adapter was born.

Follows is a simple test of the verbose macros.  Several sets of macro aliases providing much short names are also included, but only the underlying verbose macros are tested.  In reviewing the code, you will note the lack of an identifying parameter on the *BEGIN_TESTS and *DEFINE_TEST macros, even though all four of the frameworks require identifiers be provided.  The adapter macros make use of a combination of the C processor, anonymous namespaces, and/or C++ template metaprogramming to automatically create unique identifiers for a test and groups of tests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
G42_TEST_BEGIN_TESTS()

G42_TEST_DEFINE_TEST()
{
}

G42_TEST_DEFINE_TEST()
{
	G42_TEST_IF_NOT_DEFAULT(true);
#if G42_TEST_IF_NOT_REPORT_ONLY
	G42_TEST_IF_NOT_REPORT_ONLY(true);
#endif
#if G42_TEST_IF_NOT_REPORT_TEST_ABORT
	G42_TEST_IF_NOT_REPORT_TEST_ABORT(true);
#endif
}

G42_TEST_END_TESTS()

G42_TEST_BEGIN_TESTS()
G42_TEST_DEFINE_TEST()
{
	G42_TEST_IF_NOT_DEFAULT(true);
}
G42_TEST_END_TESTS()

The technique of generating the id varies with each test, but the most difficult proved to be providing the compile-time constant integer values needed by TUT for individual tests since they are specified as specializations of a TUT defined template function.  The task was made more difficult when constraining the solution to utilize only standard language features and no implementation specific compiler extensions (e.g. __if_exists and __counter__).  This test code is repeated in two separate C++ files for each of the four frameworks.  This ensures there are not any multiply defined symbol errors between compilation units as well as with-in a compilation unit.

TUT is a header only library.  boost::test has a header that can be included in one and only one object file to avoid linking in a library.  Both gtest and UnitTest++ require linking in a library.  All four can be utilized in the same executable, although gtest and UnitTest++ do both define the TEST macro, but this isn’t terminal as the only object file both needed to be included in is the one with main (and even that is technically avoidable) and the macro can simple be undefined there if no tests are defined there.

All of the unit test frameworks except TUT produced errors formatted similar to compiler errors that allowed Visual Studio to navigate directly to source code line when the error line was double clicked in the output.  TUT doesn’t provide macros, so in turn it doesn’t generate errors with source filename and line number included in the output.  Even when a macro is created, the error message is appended to some TUT supplied text so it doesn’t conform to VS’s output parsing requirements.  Thus I did make one change to TUT code to put the error message on it’s own line.

Google Test provides verbose output about test execution even when there are no failures, where as both boost::test and UnitTest++ provide minimal output when there are no failures.

My next task is to see if the output of the 4 C++ unit testing frameworks can be normalized without modifying the frameworks.  I expect to utilize some of this error reporting code as the basis for a new logging framework that will in turn be used by the new unit testing framework that will be used to test the less foundational parts of the new logging framework.

Posted in About Me, Coding | Tagged , , , , , , , , , | 1 Comment

Project Cloud Hydra Indiegogo Postmortem

While my Cloud Hydra Indiegogo campaign isn’t over yet, it’s not only lacking traction but has gone off a cliff.  The first time I looked for the campaign by browsing through the Technology category, I found it on page 12.  The next day it was up to page 10, the next page 9, and then page 8 on the third and fourth full days.  However, on the fourth day of the campaign I update the brief and long descriptions and haven’t been able to find the project in the Technology category since then.  I actually went through all forty pages the first day and have check the first dozen several times since then.

As of now, the project has gotten 21 referrals through tracking links I’ve shared here, on Facebook, and Google+, but no direct views.  Since the campaign is no longer showing up when browsing the Technology category, I am assuming it will never get any direct views and in turn will not get any traction.

Lessons learned:

  • Regular updates (announcements) do appear to improve the positioning of a campaign in it’s category (I did an update of the first 4 days), as Indiegogo says it will.
  • Updating the brief and/or long descriptions may cause a campaign to no longer show up when browsing campaigns.
  • My social networking skills are insufficient to getting a campaign funded through my network.
Posted in Coding | Tagged , | Leave a comment

Talkback: Purposes for a Software and Internet Social Purpose Corporation

Briefly the new Washington State Social Purpose Corporations are similar to Benefits Corporations or B Corporations that now exist in 7 states, except that their social purpose isn’t defined by statute but rather can be customized in the corporate charter.  In turn, this provides the opportunity for an SPC to have social purposes not contemplated by the statutes for Benefits Corporations.  More information on WA SPCs can be found here, here, and here.

If you could create the purposes for a Software and Internet company that wanted to register as Social Purpose Corporatiodn (SPC) in Washington State, what would those purposes be?

Here are my initial thoughts on SPC charter terms with social purposes beneficial to customers:

  • A la carte pricing for all products and services.  Not required to to purchase the uber expensive SKU just to get one version not offered in the standard SKU.
  • At least two separate pricing models offered for every product and service.  Examples of pricing models being one-time fee, subscription, actual use fees, free but ad supported.
  • No hybrid pricing models.  No pricing model that includes both a one-time fee and ongoing subscription fees.  No pricing model that includes both a subscription and actual use fees for overages.  No ads when a subscription is being paid.
  • All software licenses and accounts are transferable.
  • Opt-in only and separately for all uses of customer data.  No blanket permission to use data via all or nothing Terms of Service.
  • No changes in Terms of Service without explicit user acknowledgement of the highlighted changes.
  • User service can not be terminated for 90 days after declining changes in Terms of Service.
  • Three strikes rule regarding violations of Terms of Service before service can be terminated.  The service can be suspended for a day, a week, and then a month at each strike.
  • All user services can not be suspected or terminated for violating the Terms of Service for one service.

Here are my initial thoughts on SPC charter terms with social purposes beneficial to employees:

  • Hourly pay for all employees even when not required by law.  No more unpaid overtime for salaried employees.
  • Flexible work hours and locations.
  • Separate sick leave and vacation pools.  No combined PTO that encourages people to come into the office when ill to save PTO days for vacation.
  • Health insurance for all employees including part-time employees.
  • Self-directed 401(k) for all employees.
  • No layoffs with out all of the C-level executives being included in the layoffs.
  • No benefits, forms of compensation, or severance packages made available to some employees and not others.  No golden parachutes just for executives.
  • Quantity of contingent staff, consultants, and vendor staff utilized limited to 10% the number of full-time employees.
Posted in Coding, Web development | Tagged , , , | 2 Comments