Compatibility for AdaCore’s legacy testsuites¶

Although all the default behaviors in e3.testsuite presented in this documentation should be fine for most new projects, it is not realistic to require existing big testsuites to migrate to them. A lot of testsuites at AdaCore use similar formalisms (atomic testcases, dedicated test directories, …), but different formats: no test.yaml file, custom files for test execution control, etc.

These testsuites contain a huge number of testcases, and thus it is a better investment of time to introduce compatible settings in testsuite scripts rather than reformat all testcases. This section presents compatibility helpers for legacy AdaCore testsuites.

Test finder¶

The e3.testsuite.testcase_finder.AdaCoreLegacyTestFinder class can act as a drop-in test finder for legacy AdaCore testsuites: all directories whose name matches a TN (Ticket Number), i.e. matching the [0-9A-Z]{2}[0-9]{2}-[A-Z0-9]{3} regular expression, are considered as containing a testcase. Legacy AdaCore testsuites have only one driver, so this test finder always use the same driver. For instance:

@property
def test_finders(self):
    # This will create a testcase for all directories whose name matches a
    # TN, using the MyDriver test driver.
    return [AdaCoreLegacyTestFinder(MyDriver)]

Test control¶

AdaCore legacy testsuites rely on a custom file format to lead testcase execution control: test.opt files.

Similarly to the YAML-based control descriptions, this format provides a declarative formalism to describe settings depending on the environment, and more precisely on a set of discriminants: simple case insensitive names for environment specificities. For instance: linux on a Linux system, windows on a Windows one, x86 on Intel 32 bits architecture, vxworks when targetting a VxWorks is involved, etc.

A parser for such files is included in e3.testsuite (see the optfileparser module), and most importantly, a TestControlCreator subclass binds it to the rest of the testsuite framework: AdaCoreLegacyTestControlCreator, from the e3.testsuite.control module. Its constructor requires the list of discriminants used to selectively evaluate test.opt directives.

This file format not only controls test execution with its DEAD, XFAIL and SKIP commands: it also allows to control the name of the script file to run (CMD command), the name of the output baseline file (OUT), the time limit for the script (RLIMIT), etc. For this reason, AdaCoreLegacyTestControlCreator works best with the AdaCore legacy test driver: see the next section.

Test driver¶

All legacy AdaCore testsuites use actual/expected test output comparisons to determine if a test passes, so the reference test driver for them derives from DiffTestDriver: e3.testsuite.driver.adacore.AdaCoreLegacyTestDriver. This driver is coupled with a custom test execution control mechanism: test.opt files (see the previous section), and thus overrides the test_control_creator property accordingly.

This driver has two requirements for Testsuite subclasses using it:

Put a process environment (string dictionary) for subprocesses in self.env.test_environ. By default they can just put a copy of the testsuite’s own environment: dict(os.environ).
Put the list of discriminants (list of strings) in self.env.discs. For the latter, starting from the result of the e3.env.AbstractEnv.discriminants property can help, as it computes standard discriminants based on the current host/build/target platforms. Testsuites can then add more discriminants as needed.

For instance, imagine a testsuite that wants standard dircriminants plus the valgrind discriminant if the --valgrind command-line option is passed to the testsuite:

class MyTestsuite(Testsuite):
    def add_options(self, parser):
        parser.add_argument("--valgrind", action="store_true",
                            help="Run tests under Valgrind")

    def set_up(self):
        super(MyTestsuite, self).set_up()
        self.env.test_environ = dict(os.environ)
        self.env.discs = self.env.discriminants
        if self.env.options.valgrind:
            self.env.discs.append("valgrind")

There is little point describing precisely the convoluted behavior for this driver, so we will stick here to a summary, with a few pointers to go further:

All testcases must provide a script to run. Depending on testsuite defaults (AdaCoreLegacyTestControlCreator.default_script property) and the content of each test.opt testcase file, this script can be a Windows batch script (*.cmd), a Bourne-compatible shell script (*.sh) or a Python script (*.py).
It is the output of this script that is compared against the output baseline. To hide environment-specific differences, output refiners turn backslashes into forward slashes, remove .exe extensions and also remove occurences of the working directory.
On Unix systems, this driver has a very crude conversion of Windows batch script to Bourne-compatible scripts: text substitution remove some .exe extensions, replaces %VAR% environment variable references with $VAR, etc. See AdaCoreLegacyTestDriver.get_script_command_line. Note that subclasses can override this method to automatically generate a test script.

Curious readers are invited to read the sources to know the details: doing so is necessary anyway to override specific behaviors so that this driver fits the precise need of some testsuite. Hopefully, this documentation and inline comments have made this process easier.