Boost logo

Boost Test Library: Test Execution Monitor

Home
Introduction
Benefits
Example
Compilation
Test/Example Programs
Rationale
Design

Also see: Unit Test Framework

Introduction

The Boost Test Library's Test Execution Monitor provides a main() function which calls a user-supplied test_main() function. The library supplied main() relieves users from messy error detection and reporting duties. The Test Execution Monitor is intended for fairly simple test programs or to dig a problem in an existent production code. Program Execution Monitor may be more suitable to monitor production (non-test) programs. Unit Test Framework may be more suitable for complex test programs.

Benefits

The Test Execution Monitor provides a simple framework for program testing.reference to the top

Example

The example program shows six different ways to detect and report an error in the add() function.

#include <boost/test/test_tools.hpp>

int add( int i, int j ) { return i+j; }

int test_main( int, char *[] )             // note the name!
{
    // six ways to detect and report the same error:
    BOOST_CHECK( add( 2,2 ) == 4 );        // #1 continues on error
    BOOST_REQUIRE( add( 2,2 ) == 4 );      // #2 throws on error
    if( add( 2,2 ) != 4 )
      BOOST_ERROR( "Ouch..." );            // #3 continues on error
    if( add( 2,2 ) != 4 )
      BOOST_FAIL( "Ouch..." );             // #4 throws on error
    if( add( 2,2 ) != 4 ) throw "Oops..."; // #5 throws on error

    return add( 2, 2 ) == 4 ? 0 : 1;       // #6 returns error code
}

Approach #1 uses the BOOST_CHECK tool, which displays an error message on std::cout that includes the expression that failed, the source file name, and the source file line number. It also increments an error count. At program termination, the error count will be displayed automatically by the Test Execution Monitor.

Approach #2 using the BOOST_REQUIRE tool, is similar to #1, except that after displaying the error, an exception is thrown, to be caught by the Test Execution Monitor. This approach is suitable when writing a explicit test program, and the error would be so severe as to make further testing impractical. BOOST_REQUIRE differs from the C++ Standard Library's assert() macro in that it is always generated, and channels error detection into the uniform Test Execution Monitor reporting procedure.

Approaches #3 and #4 are similar to #1 and #2 respectively, except that the error detection is coded separately. This is most useful when the specific condition being tested is not indicative of the reason for failure.

Approach #5 throws an exception, which will be caught and reported by the Test Execution Monitor. This approach is suitable for both production and test code, in libraries or not. The error message displayed when the exception is caught will be most meaningful if the exception is derived from std::exception, or is a char* or std::string.

Approach #6 uses a return value to inform the caller of the error. This approach is particularly suitable for integrating existing test code with the test tools library. Although it works fine with the Boost Program Execution Monitor or Test Execution Monitor libraries, and is very useful for running existing code under them, most C++ experts prefer using exceptions for error reporting. reference to the top

The Test Execution Monitor compilation

The Test Execution Monitor is supplied as an offline library and should be compiled and linked with a test program. Following files, that are located in the Boost Test Library src directory, compose the component:

execution_monitor.cpp
test_tools.cpp
unit_test_log.cpp
unit_test_parameters.cpp
unit_test_monitor.cpp
unit_test_result.cpp
unit_test_suite.cpp
test_main.cpp

You also have a choice to include all files constituting the Test Execution Monitor directly into your test module. Use <boost/test/included/test_exec_monitor.hpp> for this porpose.

Example and Test Programs

test_exec_example
test_exec_fail1
test_exec_fail2
test_exec_fail3
test_exec_fail4

reference to the top

Rationale

How should a test program report errors? Displaying an error message is an obvious possibility:

if( something_bad_detected )
  std::cout << "something bad has been detected" << std::endl;

But that requires inspection of the program's output after each run to determine if an error occurred. Since test programs are often run as part of a regression test suite, human inspection of output to detect error messages is too time consuming and unreliable. Test frameworks like GNU/expect can do the inspections automatically, but are overly complex for simple testing.

A better simple way to report errors is for the test program to return EXIT_SUCCESS (normally 0) if the test program completes satisfactorily, and EXIT_FAILURE if an error is detected. This allows a simple regression test script to automatically and unambiguous detect success or failure. Further appropriate actions such as creating an HTML table or emailing an alert can be taken by the script, and can be modified as desired without having to change the actual C++ test programs.

A testing protocol based on a policy of test programs returning EXIT_SUCCESS or EXIT_FAILURE does not require any supporting tools; the C++ language and standard library are sufficient. The programmer must remember, however, to catch all exceptions and convert them to program exits with non-zero return codes. The programmer must also remember to not use the standard library assert() macro for test code, because on some systems it results in undesirable side effects like a message requiring manual intervention.

The Test Execution Monitor automates those tasks, yet can be ignored by programmers who prefer to implement the zero return testing protocol themselves.

Design

The Boost Test Library Design document describes the relationship between Boost Test Library components. reference to the top