Boost logo

Boost Test Library: Unit Test Framework

The acceptance test makes the customer satisfied that the
software provides the business value that makes them willing
to pay for it. The unit test makes the programmer satisfied
that the software does what the programmer thinks it does

XP maxim

Home
Introduction
Getting Started
The Framework components

The Test Case
The Test Suite
The Test Result
The Test Log
The Test Monitor

The Framework integration
The Framework parameters
The Framework compilation
The Framework extensions
Example and Test Programs
Rationale
Design

Also see: Test Tools

Introduction

The Boost Test Library's Unit Test Framework provides facilities to simplify writing test cases using Test Tools and organizing them into test suites hierarchy. The framework relieves users from messy a error detection, reporting duties and parameter processing. It provided main() function that initialize the framework, setups parameters based on command line arguments and/or environment variables, calls the user-supplied function init_unit_test_suite(argc, argv) and than runs the provided test suite. The framework keeps track of all passed/failed Test Tools assertions, provides an ability to check the testing progress based on the amount of test cases run as part of the total amount of test cases and generates the result report in several different formats. The Unit Test Framework intended to be used both for a simple testing needs and a complex non trivial testing. It is not intended to be used with the production code, where the Program Execution Monitor could be used. This was one of the design rationale to make the library implemented offline vs. an inline implementation to allows us to speed up a compilation at an expense of a runtime efficiency. The Unit Test Framework should be preferred over Test Execution Monitor while working on creating new test programs. reference to the top

Getting Started

See Getting Started to use Unit Test Framework

The Framework components

The Unit Test Framework consists of several cooperating components. All components are located in the namespace boost::unit_test_framework. The Test Case component encapsulate the notion of a simple testing unit. The Test Suite component allows to unite several related testing units into one bundle that could be considered an a compound testing unit. To manage output generated during testing one could use the Test Log component. The Test Result component is responsible for testing results presentation.

The Test Case

The Unit Test Framework provides an ability for the user to create the Test Case based on a free function or a user's class method. There are four types of test cases: function_test_case, class_test_case, parametrized_function_test_case, parametrized_class_test_case. All of them implements test case interface defined in the abstract base class test_case.

Test case interface

Definition

defined in unit_test_suite.hpp

Synopsis
class test_case
{
public:
    void set_timeout( int timeout );
    void set_expected_failures( unit_test_counter exp_fail );

    void run();
};
Description

Abstract class test_case define the test case interface. Use method test_case:set_timeout(...) to set the timeout value for the test case. See method execution_monitor::execute(...) for more details about the timeout value. Use method test_case::set_expected_failures(...) to set the expected amount of Test Tools failures in the test case. In most cases it's more convenient to set these parameters while adding this test_case to a test_suite. See the method test_suite::add(...) for more details. Use method test_case::run() to start the test case processing.

Construction

You will never need to create instances of the class test_case.

reference to the top

Free function based test cases

Definition

defined in unit_test_suite.hpp

Synopsis
class function_test_case : public test_case
{
public:
    function_test_case( void (*function_type)(), char const* name );

    ... // Implementation
};
Description

Simplest and most widely used form of test cases. Instances of the class function_test_case are created by the framework for supplied by the user pointer to free function with the following specification: void (*fct)(). Note void return type. To report a testing results one should use the Test Tools instead of returing result code.

Construction

To create a test case based on your test function use the following macro:

BOOST_TEST_CASE( &free_function ).

BOOST_TEST_CASE creates a new instance of the class function_test_case and returns a pointer to the base class test_case. In most cases you will use it as an argument to the method test_suite::add(...).

Examples
void test_feature1()
{
    ...
}
...

ts->add( BOOST_TEST_CASE( &test_feature1 ) );
_____________________
#include <boost/test/unit_test.hpp>
using boost::unit_test_framework::test_suite;

// most frequently you implement test cases as a free functions
void free_test_function()
{
    BOOST_CHECK( 2 == 1 );
}

test_suite*
init_unit_test_suite( int argc, char* argv[] ) {
    test_suite* test= BOOST_TEST_SUITE( "Example" );

    test->add( BOOST_TEST_CASE( &free_test_function )

    return test;
}
reference to the top

Class member function based test cases

Definition

defined in unit_test_suite.hpp

Synopsis
template<class UserTestClass>
class class_test_case : public test_case
{
public:
    class_test_case( void (UserTestCase::*function_type)(),
                     char const* name,
                     boost::shared_ptr<UserTestCase>& user_test_case );

    ... // Implementation
};
Description

Instances of the class class_test_case are created by the framework for the supplied member function and instance of the user's test class with the following specification: void (UserTestClass::*fct)(). To allow sharing on the same instance, so that one test case could use results of another, class_test_case is constructed based on boost::shared_ptr to the user test case class instance. Note, though, that we could achive similar effect with use of free function like this:

void compount_test() {
    UserTestCase instance;

    instance.test1();
    instance.test2();
    ...
};
The only drawback of this version that you could not separate results of one test case from another. Another reason to use class member function based tast cases is to test non default constructible user test case. In other words, if iser test case need command line arguments or some other parameters to be constructed. As a general rule it is recommended to use the class_test_case only if you can't implement a test logic in a free function. Due to usage of templates in an implementation of the class_test_case, a compilation can be longer would you use the function_test_case.
Construction

To create an instance of the class class_test_case use the following macro:

BOOST_CLASS_TEST_CASE( function, shared_test_case_instance ).

BOOST_CLASS_TEST_CASE creates a new instance of the class class_test_case and returns a pointer to the base class test_case. In most cases you will use it as an argument to the method test_suite::add(...).

Examples
class my_complex_test {
public:
    void test_feature1() {
        ...
    }
};
...

ts->add( BOOST_TEST_CASE( &my_complex_test::test_feature1 ) );
_______________________________
class class_under_test {
public:
    // i should be positive; throw an exception otherwise
    explicit class_under_test( int i );
    ...
    // access methods
    int get_value() const;
};

class compound_test {
public:
    void test_construction() {
        BOOST_CHECK_THROW( new class_under_test( -1 ) );

        v = new class_under_test( 1 );

        BOOST_CHECK( v is valid );
        ...
    }

    void test_access_methods() {
        BOOST_CHECK_EQUAL( v->get_value(), 1 );
        ...
    }
private:
    class_under_test* v;
};
...

boost::shared_ptr<compound_test> instance( new compound_test );
ts->add( BOOST_TEST_CASE( &compound_test::constructor, instance ) );
ts->add( BOOST_TEST_CASE( &compound_test::test_access_methods, instance ) );
reference to the top

Parameterized free function test cases

Definition

defined in unit_test_suite.hpp

Synopsis
template <typename ParamIterator, typename ParameterType>
class parametrized_function_test_case : public test_case
{
    ... // Implementation
};
Description

Instances of the class parametrized_function_test_case are created by the framework for the supplied user's free function with the following specification: void (*fct)( ParameterType ).

Construction

To create an instance of the class parametrized_function_test_case use the following macro:

BOOST_PARAM_TEST_CASE( free_function, first_parameter, last_parameter ).

Here first_parameter and last_parameter are begin and end iterators for the list of parameters accordingly. BOOST_PARAM_TEST_CASE creates a new instance of the class parametrized_function_test_case and returns a pointer to the abstract class test_case. In most cases you will use it as an argument to the method test_suite::add(...). Be aware that the parametrized_function_test_case does not store list of parameters internally. The user should make sure that parameters list will not be destroyed until the test case is run. That's why it not recommended to create a parameters list as local variable in init_unit_test_suite. A simple way to handle a parameters list lifetime is to place it into a user defined test suite class.

Example
void test_file_reader( std::string const& file_name )
{
    ...
}

struct reader_testing : public boost::unit_test_framework::test_suite
{
    void reader_testing()
    {
        files_to_test.push_back( "file 1" );
        ...

        files_to_test.push_back( "file N" );

        add( BOOST_TEST_CASE( &test_file_reader,
                              files_to_test.begin(),
                              files_to_test.end() );
    }
    std::list<std::string> files_to_test;
};
...
ts->add( new reader_testing );
reference to the top

Parameterized class member function based test cases

Definition

defined in unit_test_suite.hpp

Synopsis
template<class UserTestClass, typename ParamIterator, typename ParameterType>

class parametrized_class_test_case : public test_case
{
    ... // Implementation
};
Description

Instances of the class parametrized_class_test_case are created by the framework for the supplied method of the user's test class with the following specification: void (UserTestClass::*fct)( ParameterType ). The parametrized_class_test_case is responsible for the creation and the destroying of the user's test class instance.

Construction

To create an instance of the class parameterized_class_test_case use the following macro:

BOOST_PARAM_TEST_CASE( test_class_method, first_parameter, last_parameter ).

Here first_parameter and last_parameter are begin and end iterators for the list of parameters accordingly. BOOST_TEST_CASE creates a new instance of the class parametrized_class_test_case and returns a pointer to the abstract class test_case. In most cases you will use it as an argument to the method test_suite::add(...). Be aware that the parametrized_class_test_case does not store list of parameters internally. The user should make sure that parameters list will not be destroyed until the test case is run. That's why it not recommended to create a parameters list as local variable in init_unit_test_suite. For example, you can place parameters list in file scope.

Example:
class my_complex_test
{
    void test_assignment( double tolerance )
    {
        ...
    }
};
...
std::list<double> possible_tolerances;
ts->add( BOOST_TEST_CASE( &my_complex_test::test_assignment,
                          possible_tolerances.begin(),
                          possible_tolerances.end() ) ); 
reference to the top

The Test Suite

Definition

defined in unit_test_suite.hpp

Synopsis
class test_suite : public test_case
{
public:
    void add( test_case* tc,
              unit_test_counter expected_failures = 0,
              int timeout = 0 );
    ... // Implementation
};
Description

The Unit Test Framework provides an ability for the user to combine created test cases into a test suite and create a hierarchy of test suites of any depth. To add a test_case to the test_suite use the method test_suite::add(...). The first argument is a pointer to a new test_case, second - expected_failured defines the amount of Test Tools assertions that are expected to fail in the test case, and third - timeout defines the timeout value for the test case. Note that the expected number of failures is calculated automatically for the test suites, so you do not need to specify them while adding test suites to a hierarchy. See the method execution_monitor::execute(...) for more details about the timeout value. Last two arguments are optional and will not be set if absent. In this case values defined in the test_case are used. reference to the top

Construction

To create instances of the class test_suite you can use the following macro:

BOOST_TEST_SUITE( test_suite_name ).

BOOST_TEST_SUITE creates an instance of the class test_suite and returns a pointer to it. test_suite is a test_case, that allows to generate a multilevel hierarchy.

The Test result

Definition

defined in unit_test_result.hpp

Synopsis
class unit_test_result
{
      static unit_test_result& instance();
      void                     confirmation_report( std::ostream& where_to );
      void                     short_report( std::ostream& where_to );
      void                     detailed_report( std::ostream& where_to );
      int                      result_code();
};
Description

The Unit Test Framework keeps testing results as the instances of the class unit_test_result. Class unit_test_result provides three methods for the result reporting. Use the method unit_test_result::confirmation_report(...) to report the success/failure confirmation massage only. Use the method unit_test_result::short_report(...) to report testing results for the current test_case. The report includes an amount of passed and failed tests and an amount of passed and failed Test Tools assertions. Use the method unit_test_result::detailed_report(...) to report testing results for the current test_case and all test cases it consist of. In most cases you will not need to use this interface directly. The framework provides a report for overall testing results automatically.

Construction

To access instance of class unit_test_result holding results for the current test case use static method unit_test_result::instance(). reference to the top

The Test log

Definition

defined in unit_test_log.hpp

Synopsis
class unit_test_log
{
     static unit_test_log& instance();
     void  set_log_stream( std::ostream& str );
     void  set_log_threshold_level_by_name( char const* lev );
};
Description

To manage the testing output stream the Unit Test Framework uses a singleton class unit_test_log. To change the test output stream use method unit_test_log::set_log_stream(...). The default stream is std::cout. To set a log level use the method unit_test_log::set_log_level_by_name(...). In most cases you will not need to use this interface directly. The framework provides a simple way to set the log level from out of the test program. For more details and for the list of acceptable values see Framework parameters.

Construction

To access the only instance of the class unit_test_log use method unit_test_log::instance(). reference to the top

The Test monitor

To monitor a test_case run the Unit Test Framework uses a class unit_test_monitor. The unit_test_monitor allows run test_case methods under control of a execution_monitor and extends the Execution Monitor functionality to translate execution_monitor exceptions into designated error codes. For more details about the class unit_test_monitor see the Boost Test Library Design. In most cases you will not need to utilize unit_test_monitor because the framework already monitor a test case run.

The Framework integration

Description

As mentioned above Unit Test Framework is responsible for supplying main function that initializing testing environment and taking care about results reporting. The main function also includes a hook for the function that should be supplied by the user. So, to integrate test program with the framework you should provide this function with the following specification:

boost::unit_test_framework::test_suite* init_unit_test_suite ( int argc, char* argv[] )

This function should create and initialize top level instance of the class test_suite. The NULL pointer returned by the function is treated as a non-initialized test_suite. In other case the framework runs provided instance of the class test_suite and then destroy it the end of program, so it should be allocated dynamically. Command line arguments passed to the function guarantied not to have any framework-specific command line arguments.

After test cases run the framework reports the results and returns the result code. Here the list of values returned by the testing programs integrated with the unit test framework:

Value Meaning
boost::exit_success returned if no errors occured during test or success result code was explicetly requested with return result code framework parameter
boost::exit_test_failure returned if non-fatal errors detected and no uncaught exceptions thrown
boost::exit_exception_failure returned if fatal errors detected or uncaught exceptions thrown
Example
test_suite*
init_unit_test_suite( int argc, char* argv[] )
{
    test_suite* test= BOOST_TEST_SUITE( "Master test suite" );

    test->add( BOOST_TEST_CASE( &my_test_function ) );

    return test;
}
reference to the top

The Framework parameters

The Unit Test Framework provides two ways to configure test program run parameters externally: using command line arguments and setting environment variables. The following tables provides a compiled information about the framework parameters. Note that default parameter's value (i.e. value of the parameter if it is not defined either in the environment or among command-line arguments) for each parameter is bolded.

Parameter Name: Log level
Environment variable name: BOOST_TEST_LOG_LEVEL
Command line argument name: --log_level=<value>
Acceptable Values:
all - report all log messages including the passed test notification
success - the same as all
test_suite - show test suite messages
messages - show user messages
warnings - report warnings issues by user
all_errors - report all error conditions
cpp_exception - report uncaught c++ exception
system_errors - report system originated non-fatal errors (for example, timeout or floating point exception)
fatal_errors - report only user or system originated fatal errors (for example, memory access violation)
progress - report only progress information: number of run test cases vs. overall number of test cases
nothing - does not report any information
Description: Set a log level for the testing in a range from a complete log when all successful tests are confirmed and all test suite messages are included to an empty log when nothing is put into a test output stream.

Parameter Name: [Not] return result code
Environment variable name: BOOST_TEST_RESULT_CODE
Command line argument name: --result_code=<value>
Acceptable Values: yes
no
Description: If set to no enforce framework to return zero result code always. Could be used for test programs integrated with GUI.

Parameter Name: Result report level
Environment variable name: BOOST_TEST_REPORT_LEVEL
Command line argument name: --report_level=<value>
Acceptable Values: no
confirm
short
detailed
Description: Set the level of result report, that the framework generates at the end of testing. Use "no" value to eliminate the result report.

Parameter Name: Save pattern
Environment variable name: BOOST_TEST_SAVE_PATTERN
Command line argument name: --save_pattern=yes
Acceptable Values: no
yes
Description: Could be used by a testing program to inquire a saving mode for the output_test_stream tool.

Parameter Name: Print build info
Environment variable name: BOOST_TEST_BUILD_INFO
Command line argument name: --build_info=yes
Acceptable Values: no
yes
Description: If specified in command line make the framework to print build information that include: platform, compiler, stl in use and boost version.

reference to the top

The Framework compilation

The Unit Test Framework is supplied as an offline library and should be compiled and linked with a test program. Following files, that are located in the Boost Test Library src directory, compose the framework:

execution_monitor.cpp
test_tools.cpp
unit_test_parameters.cpp
unit_test_log.cpp
unit_test_main.cpp
unit_test_monitor.cpp
unit_test_result.cpp
unit_test_suite.cpp

You also have a choice to include all files constituting the framework directly into your test module. Use <boost/test/included/unit_test_framework.hpp> for this porpose.

Example and Test Programsreference to the top

unit_test_example1
unit_test_example2
unit_test_example3
unit_test_example4
unit_test_example5
online_test
errors_handling_test
parameterized_test_test
result_report_test

Rationale

What is the first thing to do when you start working on new library/class/program? Exactly - we need the unit test module. Writing of the unit test module should be simple and obvious. On the other hand the framework should allow us to do a lot of non-trivial things. We want to be able to have a lot of small test cases. We want to be able to group them in test suites. At the beginning of the development we want to see as much descriptive error message as possible, while during regression testing we just want to know is there any errors at all. For small test programs a run time should prevail over a compilation time - who want to wait a 1 min to start the test that run a 1 sec?. For a long and complex test we want to be able to see the test progress.

Design

reference to the top

The Boost Test Library Design document describes the relationship between Boost Test Library components.