A test consists of a set of test cases. Each test case is implemented as an erlang function. An erlang module implementing one or more test cases is called a test suite.
A test specification is a specification of which test suites and test cases to run and which to skip. A test specification can also group several test cases into conf cases with init and cleanup functions (see section about configuration cases below). In a test there can be test specifications on three different levels:
The top level is a test specification file which roughly specifies what to test for a whole application. The test specification in such a file is encapsulated in a topcase command.
Then there is a test specification for each test suite, specifying which test cases to run within the suite. The test specification for a test suite is returned from the all(suite) function in the test suite module.
And finally there can be a test specification per test case, specifying sub test cases to run. The test specification for a test case is returned from the specification clause of the test case.
When a test starts, the total test specification is built in a tree fashion, starting from the top level test specification.
The following are the valid elements of a test specification. The specification can be one of these elements or a list with any combination of the elements:
A test specification file is a text file containing the top level test specification (a topcase command), and possibly one or more additional commands. A "command" in a test specification file means a key-value tuple ended by a dot-newline sequence.
The following commands are valid:
All test specification files shall have the extension ".spec". If special test specification files are needed for Windows or VxWorks platforms, additional files with the extension ".spec.win" and ".spec.vxworks" shall be used. This is useful e.g. if some test cases shall be skippped on these platforms.
Some examples for test specification files can be found in the Examples section of this user's guide.
If a group of test cases need the same initialization, a so called configuration or conf case can be used. A conf case consists of an initialization function, the group of test cases needing this initialization and a cleanup or finalization function.
If the init function in a conf case fails or returns {skip,Comment}, the rest of the test cases in the conf case (including the cleanup function) are skipped. If the init function succeeds, the cleanup function will always be called, even if some of the test cases in between failed.
Both the init function and the cleanup function in a conf case get the Config parameter as only argument. This parameter can be modified or returned as is. Whatever is returned by the init function is given as Config parameter to the rest of the test cases in the conf case, including the cleanup function.
If the Config parameter is changed by the init function, it must be restored by the cleanup function. Whatever is returned by the cleanup function will be given to the next test case called.
It is possible to skip certain test cases, for example if you know beforehand that a specific test case fails. This might be functionality which isn't yet implemented, a bug that is known but not yet fixed or some functionality which doesn't work or isn't applicable on a spesific platform.
There are several different ways to state that a test case should be skipped:
The latter of course means that the execution clause is actually called, so the author must make sure that the test case is not run. For more information about the different clauses in a test case, see the chapter about writing test cases.
When a test case is skipped, it will be noted as SKIPPED in the HTML log.