Tests and Test Suites

If you have a simple form with only a few fields, and little or no scripting, it is reasonably practicable to test it by hand, by doing all the things that the user would use the form for. But if the form is more complicated, particularly if there is a significant amount of scripting, then this becomes more problematic. This really rears its ugly head when you subsequently make a change to the form: has anything been broken as a result of the change?

Unit testing is a testing methodology based on isolating each part of a program or application, and providing tests that verify the functionality of each part. Tests are typically grouped together into test suites, in which the tests are run and from which errors are reported.

Starting with version 2.4.3, Rekall provides support for testing along the lines of unit testing. It is not strictly unit testing, since there is nothing to support isolation of the Rekall application into separate parts. However, it is related in the sense that it provides tests, test suites and error reporting.

By default, tests and test suites are disabled. To enable then, go to the advanced tab of the database connection dialog, and set the Show tests/test suites option. You can turn testing off; enable test (and their associated test menus); enable tests and test suites; or enable tests, test suites and test recording. The reason for the tests-only setting is that you can use tests to provide context menus.

Tests

In Rekall a test is effectively an event that is attached to an object. Tests are accessed from the object property dialogs, in the Events section. Double clicking the Tests entry will show a list of tests (in the same way as the list of slots) from where you can add, edit and drop tests. You can also create tests more conveniently by recording actions - this is described later - which can generate tests quite easily.

A rather trivial test is shown below, which simply verifies that the button to which the test is attached displays the text Click Me. This would be saved with a name. With the test set up and the form in data view, right clicking on the button will show a popup menu of tests, from which the test can be run. If the button text is Click Me when nothing will happen. If not then a warning will appear indicating that a test has failed, containing the location of the failure (which is not very interesting in this case, since you know which test you have run!).

def eventFunc (button) :

    RekallTest.test (button.text == "Click Me")
        

The RekallTest.test function takes an optional second argument, which is an error message. The function returns a boolean result which is the same as value passed as the first argument, ie., the validation expression.

def eventFunc (button) :

    RekallTest.test (button.text == "Click Me", "OOPS!")
        

As noted above, you could use tests to provide context menus, although there is no way at present to change the title that appears on the popup menu.

Test Suites

Test suites provide a method to run a set of one or more tests, and to report any errors that were detected by the tests. When run this way, errors detected by calls to RekallTest.test() will not display error popups. Instead they will be noted and presented when the test suite has completed.

Test suites are set up from the form (document) properties dialog, in the Other section. Like test setup, the dialog shows a list of current tests suites, from which they can be added, edited or dropped. Clicking add or edit brings up a dialog as shown below. The test suite name is set at the top; below this is a list of a tests in the suite. The lower section is an expandable tree of form objects and tests; this is trimmed to reduce the number of entries (specifically, it only shows objects that have child objects or tests). Tests are added and removed by selecting the appropriate test and clicking the Add or Remove buttons (or by double-clicking in the lower section); tests can be ordered using the Up and Down buttons.

In addition to the list of tests, there are four additional options. These also select specific tests (though only tests on the top-level form object, not throughout the form). The Setup and Teardown items correspond to the similarly named entities in unit tests; Setup is run before each test in the suite, while Teardown is run after each test. In both cases, since these tests are associated with the top-level form object, they run with that object as their argument (ie., the argument to their eventFunc function). These are intended to set up known conditions before a test, and to restore to known conditions after each test. Initialise and Reset are described below.

If enabled, test suites appear as a submenu on the form's File menu. When the test has been run, the results of all RekallTest.test() calls are shown in a dialog as below. This shows where the test call was made, the scripting language (the same syntax applies to javascript scripting as well as python), whether the test succeeded or failed, and the message, if any. In this example, all the tests worked - good!

Note that there is a difference in behaviour when running a test as part of a test suite. When run directly from a context menu, calls to RekallTest.test() which pass a false value as their first argument will return, so if you want to exit the test code at that point, you should check the result code:

def eventFunc (button) :

    if not RekallTest.test (button.text == "Click Me", "OOPS!"") :
        return

    # More tests follow ...
    #
        

However, when run as part of a test suite, such a test throws an exception and the remainder of the script is not executed. The exception is trapped, and the test suite moves immediately to the next test. So, at most one failed call to RekallTest.test() will be reported for each test.

Running Tests from the List of Forms

If test suites are enabled, then the context menu which appears when you right-click while holding down the control key ( Generating the extra menu entries can be time consuming if the form is in a remote database. Since accessing tests this way is not expected to be the usual thing to do, the control key is required, to save time in the usual case. ) on a form in the list of forms (the menu which normally displays things like Data View and Design View) will also show a list of test suites, plus an All Suites entry. These allow you to run one or all of the tests suites associated with the form.

When a test suite is run this way, the Initialise and Reset tests are used. Initialise is run once before the tests, and Reset once after (so, the sequence is initialise, setup, test1, teardown, setup, test2, teardown, ...., reset where test1 etc. are the tests in the test suite). The initialise and reset tests are unusual in that the former runs after the form document has been parsed but before the form is actually opened. Although it is restricted by this, it can be useful to set up some global state. Similarly, Reset runs after the form has logically been closed.

For instance, suppose that your form contains a script that gets the current time using python's time.time() function. This means that each time you run it, it will get a different value, which makes (say) checking that correct values are displayed more difficult. Ideally, you'd like to run it at the same time, every time! Well, here's how you can do it: below are the Initialise and Reset tests:

#  This is the Initialise test
#
def eventFunc (form) :
    import time
    time.x_time = time.time
    def ft() : return 1156503600.0
    time.time = ft
        

#  This is the Reset test
#
def eventFunc (form) :
    import time
    time.time = time.x_time
        

The Initialise test imports the time module, saves the time.time function (as time.x_time) and then replaces time.time with a function that returns a fixed value (which in this case is some time in Fri. 25th Aug. 2006). The Reset test restores time.time. So, any script code that runs between these tests will always see the same time!

Automatic Test Recording

Writing tests in python (or javascript) is powerful but time consuming, and, being realistic, however much we know that systems should be tested, the pressure is always on to get some new feature implemented. So, to make it easy to generate basic tests, Rekall has a test recording feature. If testing is enabled, the form file menu will show an extra Tests entry; if test recording is enabled, the Tests submenu will show four entries. These are Start recording, Start recording in transaction, Save recording and Cancel recording. Some of these will be disabled at any particular time.

To start recording, select Start recording (or Start recording in transaction, in which case a transaction will be started; this will be rolled back the recoding finishes). Once a recording has been started, Rekall will record most actions; things like mouse and key navigation withing the form, and changes to fields by the user. In addition, right-clicking within the form will generally bring up a menu which contains various validation checks. The exact set depends on where you right-clicked, but, for instance, a field has checks on the current field value and state (whether it is enabled and/or visible). Selecting one of these checks stores the check along with the recorded actions. To end a recording, select Save recording; this beings up a dialog as shown below.

In the dialog, you can specify a name and a comment. Generally, the save test to form checkbox should be set, so that the form will be saved with the new recording. Clicking OK saves the recording (and the form); clicking Cancel resumes recording. The actually kill a recording, select Cancel recording via the file menu.

A recording is used as for any other test, either directly via the file menu Test submenu (which will show all tests on the form object), or by incorporating it into a test suite. When the recording is run, it will replay all the actions, and check values and so forth as requested, and any error will be reported. As well as failed checks, a failure to find an object in the form will be reported. This can happen if, after making the recording, you remove or rename an object in the form.

Running Test Suites on all Forms

Right-clicking on the server item in the form list brings up a menu which, if test suites are enabled, includes a Execute tests ... item. This brings up a dialog as shown below. This allows all test suites on all forms, or a subset thereof, to be run. Running all test suites on all forms, without any errors, is the real target of testing.

Running Tests in Transactions

On problem with unit testing against databases is that a test which updates the database may not be repeatable successfully (precisely becuase the database has changed), and that the order in which tests are run (or exactly which set of tests are run) can affect results.

If your database supports transactions, then the Run in transaction option in the test suite dialog can be set. In this case, a database transaction is started immediately before a setup, test, teardown sequence; and the transaction is rolled back (ie., aborted) immediately afterwards. This means that any changes are lost, and the database returns to exactly the same state as existed before the test. This should remove the repeatability and ordering problems.

Note that this may not work if the form uses record locking, since this involves transactions.