e3.testsuite.driver.classic
: Common test driver facilities¶
The driver.classic
module’s main contribution is to provides a
TestDriver
convenience subclass: ClassicTestDriver
, that test driver
implementations are invited to derive from.
It starts with an assumption, considered to be common to most real world use
cases: testcases are atomic, meaning that the execution of each testcase is a
single chunk of work that produces a single test result. This assumption allows
to provide a simpler framework compared to the base TestDriver
API, so that
test drivers are easier to write.
First, there is no need to create fragments and handle
dependencies: the minimal requirement for ClassicTestDriver
subclasses is
to define a run
method. As you have probably guessed, its sole
responsibility is to proceed to testcase execution: build what needs to be
built, spawn subprocesses as needed, etc.
Working directory management¶
ClassicTestDriver
considers that most drivers will need to create the
temporary directories, and thus make it the default: before running the
testcase, this driver will copy the test directory to the working directory.
Subclasses can override this behavior overriding the copy_test_directory
property. For instance, to disable this copy unconditionally:
class MyDriver(ClassicTestDriver):
copy_test_directory = False
Alternatively, to disable it only if the test.yaml
file contains a
no-copy
entry:
class MyDriver(ClassicTestDriver):
@property
def copy_test_directory(self):
return not self.test_env.get("no-copy")
Output encodings¶
Although the concept of “test output” is not precisely defined here,
ClassicTestDriver
has provisions for the very common pattern of drivers
that build a string (the test output) and that, once the test has run, analyze
of the content of this output determines whether the testcase passed or failed.
For this reason, the self.output
attribute contains a Log
instance (see
Log).
Although drivers generally want to deal with actual strings (str
in
Python3, a valid sequence of Unicode codepoints), at the OS level, process
outputs are mere sequences of bytes (bytes
in Python3), i.e. binary data.
Such drivers need to decode the sequence of bytes into strings, and for that
they need to pick the appropriate encoding (UTF-8, ISO-8859-1, …).
The default_encoding
property returns the name of the default encoding used
to decode process outputs (as accepted by the str.encode()
method:
utf-8
, latin-1
, …). If it returns binary
, outputs are not decoded
and self.output
is set to a Log
instance that holds bytes.
The default implementation for this property returns the encoding
entry
from the self.test_env
dict. If there is no such entry, it returns
utf-8
(the most commonly used encoding these days).
Spawning subprocesses¶
Spawning subprocesses is so common that this driver class provides a convenience method to do it:
def shell(self, args, cwd=None, env=None, catch_error=True,
analyze_output=True, timeout=None, encoding=None):
This will run a subprocess given a list of command-line arguments (args
);
its standard input is redirected to /dev/null
while both its standard
output/error streams are collected as a single stream. shell
returns a
ProcessResult
instance once the subprocess exitted. ProcessResult
is
just a holder for process information: its status
attribute contains the
process exit code (an integer) while its out
attribute contains the
captured output.
Note that the shell
method also automatically appends a description of the
spawned subprocess (arguments, working directory, exit code, output) to the
test result log.
Its other arguments give finer control over process execution:
cwd
- Without surprise for people familiar with process handling APIs: this
argument controls the directory in which the subprocess is spawned. When
left to
None
, the processed is spawned in the working directory. env
- Environment variables to pass to the subprocess. If left to
None
, the subprocess inherit the Python interpreter’s environment. catch_error
- If true (the default),
shell
will check the exit status: if it is 0, nothing happen, however if it is anything else,shell
raises an exception to abort the testcase with a failure (see Exception-based execution control for more details). If set to false, nothing special happens for non-0 exit statuses. analyze_output
- Whether to append the subprocess output to
self.output
(see Output encodings). This is for convenience in test drivers based on output comparison (see e3.testsuite.driver.diff: Test driver for actual/expected outputs). timeout
Number of seconds to allow for the subprocess execution: if it lasts longer, the subprocess is aborted and its status code is set to non-zero.
If left to
None
, use instead the timeout that thedefault_process_timeout
property returns. TheClassicTestDriver
implementation for that property returns either thetimeout
entry fromself.test_env
(if present) or 300 seconds (5 minutes). Of course, subclasses are free to override this property if needed.encoding
- Name of the encoding used to decode the subprocess output. If left to
None
, use instead the encoding that thedefault_encoding
property returns (see Output encodings). Here, too, the default implementation returns theencoding
entry fromself.test_env
(if present) orutf-8
. Again, subclasses are free to override this property if needed. truncate_logs_threshold
- Natural number, threshold to truncate the subprocess output that
shell
logs in the test result log. This threshold is interpreted as half the number of output lines allowed before truncation, and 0 means that truncation is disabled. If left toNone
, use the testsuite’s--truncate-logs
option.
Set up/analyze/tear down¶
The common organization for test driver execution has four parts:
- Initialization: make sure input is valid: required files must be present (test program sources, input files), metadata is valid, start a server, and so on.
- Execution: the meat happens here: run the necessary programs, write the necessary files, …
- Analysis: look at the test output and decide whether the test passed.
- Finalization: free resources, shut down the server, ..
ClassicTestDriver
defines four overridable methods, one for each step:
set_up
, run
, analyze
and tear_down
. First, the set_up
method is called, then the run
one and then the analyze
one. So far,
any unhandled exception in these methods would prevent the next ones to run.
Except for the tear_down
method, which is called no matter what happens as
long as the set_up
method was called.
The following example shows how this is useful. Imagine a testsuite for a database server. We want some test drivers only to start the server (leaving the rest to testcases) while we want other test drivers to perform more involved server initialization.
class BaseDriver(ClassicTestDriver):
def set_up(self):
super().set_up()
self.start_server()
def run(self):
pass # ...
def tear_down(self):
self.stop_server()
super().tear_down()
class FixturesDriver(BaseDriver):
def set_up(self):
super(FixturesDriver, self).set_up()
self.install_fixtures()
The install_fixtures()
call has to happen after the start_server()
one,
but before the actual test execution (run()
). If initialization, execution
and finalization all happened in BaseDriver.run
, it would not be possible
for FixturesDriver
to insert the call at the proper place.
Note that ClassicTestDriver
provide valid default implementations for all
these methods except run
, which subclasses have to override.
The analyze
method is interesting: its default implementation calls the
compute_failures
method, which returns a list of error messages. If that
list is empty, it considers that there is no test failure, and thus that the
testcase passed. Otherwise, it considers that the test failed. In both cases,
it appropriately set the status/message in self.result
and pushes it to the
testsuite report.
That means that in practice, test drivers only need to override this
compute_failures
method in order to properly analyze test output. For
instance, let’s consider a test driver whose run
method spawns a supbrocess
and must consider that the test succeeds iff the SUCCESS
string appears in
the output. The following would do the job:
class FooDriver(ClassicTestDriver):
def run(self):
self.shell(...)
def compute_failures(self):
return (["no match for SUCCESS in output"]
if "SUCCESS" not in self.output
else [])
Metadata-based execution control¶
Deciding whether to skip a testcase, or expecting a test failure are both so
common that ClassicTestDriver
provides a mechanism which makes it possible
to control testcase execution thanks to metadata in that testcase.
By default, it is based on metadata from the test environment
(self.test_env
, i.e. from the test.yaml
file), but each driver can
customize this. This mechanism is described extensively in e3.testsuite.control: Control test execution.
Exception-based execution control¶
The e3.testsuite.driver.classic
module defines several exceptions that
ClassicTestDriver
subclasses can use to control the execution of testcases.
These exceptions are expected to be propagated from the set_up
, run
and
analyze
methods when appropriate. When they are, this stops the execution
of the testcase (next methods are not run). Please refer to
TestStatus for the meaning of test statuses.
TestSkip
- Abort the testcase and push a
SKIP
test result. TestAbortWithError
- Abort the testcase and push an
ERROR
test result. TestAbortWithFailure
- Abort the testcase and push a
FAIL
test result, orXFAIL
if a failure is expected (see e3.testsuite.control: Control test execution).
Colors¶
Long raw text logs can be difficult to read quickly. Light formatting (color,
brightness) can help in this area, revealing the structure of text logs. Since
it relies on the e3-core
project, e3-testsuite
already has the
colorama project in its dependencies.
ClassicTestDriver
subclasses can use self.Fore
and self.Style
attributes as “smart” shortcuts for colorama.Fore
and colorama.Style
:
if there is a single chance for text logs to be redirected to a text file
(rather than everything to be printed in consoles), colors support is disable
and these two attributes yield empty strings instead of the regular console
escape sequences.
The shell
method already uses them to format the logging of subprocesses in
self.result.log
:
self.result.log += (
self.Style.RESET_ALL + self.Style.BRIGHT
+ "Status code" + self.Style.RESET_ALL
+ ": " + self.Style.DIM + str(p.status) + self.Style.RESET_ALL
)
This will format Status code
in bright style and the status code in dim
style if formatting is enabled, and will just return Status code: 0`
without formatting when disabled.