e3.testsuite.testcase_finder
: Control testcase discovery¶
In Core concepts, the default format for testcases is described as: any
directory that contains a test.yaml
file. This section shows the mechanisms
to implement different formats.
Internally, the testsuite creates testcases from a list of
e3.testsuite.testcase_finder.ParsedTest
instances: precisely one testcase
per ParsedTest
object. This class is just a holder for the information
required to create a testcase, it contains the following attributes:
test_name
- Name for this testcase, generally computed from
test_dir
usingTestsuite.test_name
(see Changing the testcase naming scheme). Only one testcase can have a specific name, or put differently: test names are unique. driver_cls
TestDriver
subclass to instantiate for this testcase. When left toNone
, the testsuite will use the default driver (if available).test_env
- Dictionary for the test environment.
test_dir
- Name of the directory that contains the testcase.
test_matcher
- Optional “matching name”, for filtering purposes, i.e. to run the testsuite on a subset of tests. See below.
The next piece of code, responsible to create ParsedTest
instances, is the
e3.testsuite.testcase_finder.TestFinder
interface. This API is very simple:
TestFinder
objects must support a probe(testsuite, dirpath, dirnames,
filenames)
method, which is called for each directory that is a candidate to
be a testcase. The semantics for probe
arguments are:
testsuite
- Testsuite instance that is looking for testcases.
dirpath
- Absolute name for the candidate directory to probe.
dirnames
- Base names for
dirpath
subdirectories. filenames
- Basenames for files in
dirpath
.
When called, TestFinder.probe
overriding methods are supposed to look at
dirpath
, dirnames
and filenames
to determine whether this directory
contains testcases. It must return a list of ParsedTest
instances: each one
will later be used to instantiate a TestDriver
subclass for this testcase.
Note
For backwards compatibility, probe
methods can return None
instead
of an empty list when there is no testcase, and can return directly a
ParsedTest
instance instead of a list of one element when the probed
directory contains exactly one testcase.
The default TestFinder
instance that testsuites use come from the
e3.testsuite.testcase_finder.YAMLTestFinder
class. Its probe method is very
simple: consider there is a testcase iff there is test.yaml
is present in
filenames
. In that case, parse its YAML content, use the result as the test
environment and look for a driver
environment entry to fetch the
corresponding test driver.
The Testsuite.get_test_list
internal method is the one that takes care of
running the search for tests in the appropriate directories: in the testsuite
root directory, or in directories passed in argument to the testsuite, and
delegates the actual “testcase decoding” to TestFinder
instances.
Testsuites that need custom TestFinder
instances only have to override the
test_finders
property/class method in Testsuite
subclasses, to return,
as one would probably expect, the list of test finders that will probe
candidate directories. The default implementation is eloquent:
@property
def test_finders(self):
return [YAMLTestFinder()]
Note that when there are multiple test finders, they are used in the same order
as in the returned list: the first one that returns a ParsedTest
“wins”,
and the directory is ignored if all test finders returned None
.
The special case of directories with multiple tests¶
To keep reasonable performance when running a subset of testcases (i.e. when
passing the sublist
positional command line argument), the
Testsuite.get_test_list
method does not even try to call test finders on
directories that don’t match a requested sublist. For instance, with the given
tree of tests:
tests/
bar/
x.txt
y.txt
foo/
a.txt
b.txt
c.txt
The following testsuite run:
./testsuite.py tests/bar/
will call the TestFinder.probe
method only on the tests/bar/
directory
(and ignores tests/foo/
).
This is fine if each testcase has a dedicated directory, which is the
recommended strategy to encode tests. However, if indvidual tests are actually
encoded as single files (for instance *.txt
files in the example above,
which can happen with legacy testsuites), then the filtering of tests to run
can work in unfriendly ways:
./testsuite.py a.txt
will run no testcase: no directory matches a.txt
, so the testsuite will
never call TestFinder.probe
, and thus the testsuite will find no test.
In order to handle such cases, and thus force the matching machinery to consider filenames (possibly at the expanse of performance), you need to:
- override the
TestFinder.test_dedicated_directory
property to returnFalse
(it returnsTrue
by default); - make its
probe
method passParsedTest
’stest_matcher
constructor argument a string to be matched against sublists.
To continue with the previous example, let’s write a test finder that creates a
testcase for every *.txt
file in the test tree, using the
TextFileDriver
driver class:
class TextFileTestFinder(TestFinder):
@property
def test_dedicated_directory(self):
# We create one testcase per text file. There can be multiple text
# files in a single directory, ergo tests are not guaranteed to have
# dedicated test directories.
return False
def probe(self, testsuite, dirpath, dirnames, filenames):
# Create one test per "*.txt" file
return [
ParsedTest(
# Strip the ".txt" extension for the test name
test_name=testsuite.test_name(
os.path.join(dirpath, f[:-4])
),
driver_cls=TextFileDriver,
test_env={},
test_dir=dirpath,
# Preserve the ".txt" extension so that it matches "a.txt"
test_matcher=os.path.join(dirpath, f),
)
for f in filenames:
if not f.endswith(".txt")
]
Thanks to this test finder:
# Run tests/bar/x.txt and tests/bar/y.txt
./testsuite tests/bar
# Only run tests/bar/x.txt
./testsuite x.txt