e3.testsuite: Testsuite API

So far, this documentation focused on writing test drivers. Although these really are the meat of each testsuite, there are also testsuite-wide features and customizations to consider.

Test drivers

The Tutorial already covered how to register the set of test drivers in the testsuite, so that each testcase can chose which driver to use. Just creating TestDriver subclasses is not enough: testsuite must associate a name to each available driver.

This all happens in Testsuite.test_driver_map, which as usual can be either a class attribute or a property. It must contain/return a dict, mapping driver names to TestDriver subclasses:

from e3.testsuite import Testsuite
from e3.testsuite.driver import TestDriver


class MyDriver1(TestDriver):
    # ...
    pass


class MyDriver2(TestDriver):
    # ...
    pass


class MyTestsuite(Testsuite):
    test_driver_map = {"driver1": MyDriver1, "driver2": MyDriver2}

This is the only mandatory customization when creating a Testsuite subclass. A nice optional addition is the definition of a default driver: if most testcases use a single test driver, this will make it handier to create tests.

class MyTestsuite(Testsuite):
    test_driver_map = {"driver1": MyDriver1, "driver2": MyDriver2}

    # Testcases that don't specify a "driver" in their test.yaml file will
    # automatically run with MyDriver2.
    default_driver = "driver2"

Testsuite environment

Testsuite and TestDriver instances all have a self.env attribute. This holds a e3.env.BaseEnv instance: the testsuite originally creates it when starting and forwards it to test drivers.

This environment holds information about the platform for which tests are running (host OS, target CPU, … as well as parsed options from the command-line (see below). The testsuite is also free to add more information to this environment.

If a testsuite actually needs to deal with non-native targets, for instance running on GNU/Linux for x86_64 tests that involve programs for bare ARM ELF targets, then it’s useful to override the enable_cross_support class attribute/property to return true (it returns false by default):

class MyTestsuite(Testsuite):
    enable_cross_support = True

In this case, the testsuite will add --build, --host and --target command-line arguments. These have the same semantics as the homonym options in GNU configure scripts: see The GNU configure and build system. The testsuite will then use these arguments to build the appropriate environment in self.env, and thus for instance self.env.target.cpu.name will reflect the target CPU.

Command-line options

Note

This section assumes that readers are familiar with Python’s famous argparse standard package. Please read its documentation if this is the first time you hear about it.

Testsuites often have multiple operating modes. A very common mode is: does it run programs under Valgrind? Doing this has great value, as it helps finding invalid memory accesses, use of uninitialized values, etc. but comes at a great performance cost. So always using Valgrind is not realistic.

Adding a testsuite command-line option is a way to solve this problem: by default (for the most common cases: day-to-day development runs) Valgrind support is disabled, and the testsuite enables it when run with a --valgrind argument (used in continuous builders, for instance).

Adding testsuite options is very simple: in the Testsuite subclass, override the add_options method. It takes a single argument: the argparse.ArgumentParser instance that is responsible for parsing the testsuite command-line arguments. To implement the Valgrind example discussed above, we can have:

class MyTestsuite(Testsuite):
    def add_options(self, parser):
        parser.add_argument("--valgrind", action="store_true",
                            help="Run tests under Valgrind")

The result of command-line parsing, i.e. the result of parser.parse_args() is made available in self.env.options. This means that test drivers can then check for the presence of the --valgrind on the command line the following way:

class MyDriver(ClassicTestDriver):
    def run(self):
        argv = self.test_program_command_line

        # If the testsuite is run with the --valgrind option, run the test
        # program under Valgrind.
        if self.env.options.valgrind:
            argv = ["valgrind", "--leak-check=full", "-q"] + argv

        self.shell(argv)

Set up/tear down

Testsuites that need to execute arbitrary operations right before looking for tests and running them can override the Testsuite.set_up method. Similarly, testsuites that need to execute actions after all testcases ran to completion and after testsuite reports were emitted can override the Testsuite.tear_down method.

class MyTestsuite(Testsuite):
    def set_up(self):
        # Let the base class' set_up method do its job
        super().set_up()

        # Then do whatever is required before running testcases.
        # Note that by the time this is executed, command-line
        # options are parsed and the environment (self.env)
        # is fully initialized.

        # ...

    def tear_down(self):
        # Do whatever is required to after the testsuite has
        # run to completion.

        # ...

        # Then let the base class' tear_down method do its job
        super().tear_down()

Overriding tests subdirectory

As described in the tutorial, by default the testsuite looks for tests in the testsuite root directory, i.e. the directory that contains the Python script in which e3.testsuite.Testsuite is subclassed. Testsuites can override this behavior with the tests_subdir property:

class MyTestsuite(Testsuite):
    @property
    def tests_subdir(self):
        return "tests"

This property must return a directory name that is relative to the testsuite root: testcases are looked for in all of its subdirectories.

The next section describes how to go deeper and change the testcase discovery process itself.

Changing the testcase naming scheme

Testsuite require unique names for all testcases. These name must be valid filenames: no directory separator or special character such as : are allowed.

By default, this name is computed from the name of the testcase directory, relative to the tests subdirectory: directory separators are just replaced with __ (two underscores). For instance, the testcase a/b-c/d is assigned the a__b-c__d name.

Changing the naming scheme is as easy as overriding the test_name method, which takes the name of the test directory and must return the test name, conforming to the constraints described above:

class MyTestsuite(Testsuite):
    def test_name(self, test_dir):
        return custom_computation(test_dir)