testsuite

Generic testsuite framework.

Submodules

• testsuite._helpers
• testsuite.control
• testsuite.driver
• testsuite.event_notifications
• testsuite.find_skipped_tests
• testsuite.fragment
• testsuite.main
• testsuite.multiprocess_scheduler
• testsuite.optfileparser
• testsuite.report
• testsuite.result
• testsuite.running_status
• testsuite.testcase_finder
• testsuite.utils

Attributes

logger

Exceptions

TestAbort

Raise this to abort silently the execution of a test fragment.

Classes

TestsuiteCore	Testsuite Core driver.
Testsuite	Testsuite class.

Package Contents

testsuite.logger

exception testsuite.TestAbort

Bases: Exception

Raise this to abort silently the execution of a test fragment.

class testsuite.TestsuiteCore(root_dir: str | None = None, testsuite_name: str = 'Untitled testsuite')

Testsuite Core driver.

This class is the base of Testsuite class and should not be instantiated. It’s not recommended to override any of the functions declared in it.

See documentation of Testsuite class for overridable methods and variables.

root_dir = b'.'

Root directory for the testsuite, i.e. directory from which the test directory (see self.test_dir) is looked up.

test_dir

Root directory for the tree in which testcases are searched.

return_values: Dict[str, Any]

result_tracebacks: Dict[str, List[str]]

testsuite_name = 'Untitled testsuite'

running_status: e3.testsuite.running_status.RunningStatus

Object to report testsuite execution status to users and to manage abortion in case there are too many failures.

use_multiprocessing = False

Whether to use multi-processing for tests parallelism.

Beyond a certain level of parallelism, Python’s GIL contention is too high to benefit from more processors. When we reach this level, it is more interesting to use multiple processes to cancel the GIL contention.

The actual value for this attribute is computed once the DAG is built, in the “compute_use_multiprocessing” method.

_test_counter() → int

_test_status_counters() → Dict[e3.testsuite.result.TestStatus, int]

_results() → Dict[str, e3.testsuite.result.TestStatus]

property test_counter: int

Return the number of test results in the report.

Warning: this method is obsolete and will be removed in the future.

property test_status_counters: Dict[e3.testsuite.result.TestStatus, int]

Return test result counts per test status.

Warning: this method is obsolete and will be removed in the future.

property results: Dict[str, e3.testsuite.result.TestStatus]

Return a mapping from test names to results.

Warning: this method is obsolete and will be removed in the future.

abstractmethod compute_use_multiprocessing() → bool

Return whether to use multi-processing for tests parallelism.

See docstring for the “use_multiprocessing” attribute. Subclasses are free to override this to take control of when multiprocessing is enabled. Note that this will disregard the “–force-multiprocessing” command line option.

testsuite_main(args: List[str] | None = None) → int

Main for the main testsuite script.

Parameters:

args – Command line arguments. If None, use sys.argv.

Returns:

The testsuite status code (0 for success, a positive for failure).

get_test_list(sublist: List[str]) → List[e3.testsuite.testcase_finder.ParsedTest]

Retrieve the list of tests.

Parameters:

sublist – A list of tests scenarios or patterns.

add_test(dag: e3.collection.dag.DAG, parsed_test: e3.testsuite.testcase_finder.ParsedTest) → None

Register a test to run.

Parameters:

• dag – The DAG of test fragments to execute for the testsuite.

• parsed_test – Test to instantiate.

dump_testsuite_result() → None

Log a summary of test results.

Subclasses are free to override this to do whatever is suitable for them.

collect_result(fragment: e3.testsuite.fragment.TestFragment) → None

Import test results from fragment into testsuite reports.

Parameters:

fragment – Test fragment (just completed) from which to import test results.

add_result(item: e3.testsuite.driver.ResultQueueItem) → None

Add a test result to the result index and log it.

Parameters:

item – Result queue item for the result to add.

add_test_error(test_name: str, message: str, tb: str | None = None) → None

Create and add an ERROR test status.

Parameters:

• test_name – Prefix for the test result to create. This adds a suffix to avoid clashes.

• message (str) – Error message.

• tb – Optional traceback for the error.

setup_result_dirs() → None

Create the output directory in which the results are stored.

run_standard_mainloop(dag: e3.collection.dag.DAG) → None

Run the main loop to execute test fragments in threads.

run_multiprocess_mainloop(dag: e3.collection.dag.DAG) → None

Run the main loop to execute test fragments in subprocesses.

property tests_subdir: str

Abstractmethod:

Return the subdirectory in which tests are looked for.

The returned directory name is considered relative to the root testsuite directory (self.root_dir).

property test_driver_map: Dict[str, Type[e3.testsuite.driver.TestDriver]]

Abstractmethod:

Return a map from test driver names to TestDriver subclasses.

Test finders will be able to use this map to fetch the test drivers referenced in testcases.

property default_driver: str | None

Abstractmethod:

Return the name of the default driver for testcases.

When tests do not query a specific driver, the one associated to this name is used instead. If this property returns None, all tests are required to query a driver.

abstractmethod test_name(test_dir: str) → str

Compute the test name given a testcase spec.

This function can be overridden. By default it uses the name of the test directory. Note that the test name should be a valid filename (not dir seprators, or special characters such as :, …).

property test_finders: List[e3.testsuite.testcase_finder.TestFinder]

Abstractmethod:

Return test finders to probe tests directories.

abstractmethod add_options(parser: argparse.ArgumentParser) → None

Add testsuite specific switches.

Subclasses can override this method to add their own testsuite command-line options.

Parameters:

parser – Parser for command-line arguments. See <https://docs.python.org/3/library/argparse.html> for usage.

abstractmethod set_up() → None

Execute operations before running the testsuite.

Before running this, command-line arguments were parsed. After this returns, the testsuite will look for testcases.

By default, this does nothing. Overriding this method allows testsuites to prepare the execution of the testsuite depending on their needs. For instance:

• process testsuite-specific options;

• initialize environment variables;

• adjust self.env (object forwarded to test drivers).

abstractmethod tear_down() → None

Execute operation when finalizing the testsuite.

By default, this cleans the working (temporary) directory in which the tests were run.

abstractmethod write_comment_file(comment_file: IO[str]) → None

Write the comment file’s content.

Parameters:

comment_file – File descriptor for the comment file. Overriding methods should only call its “write” method (or print to it).

property default_max_consecutive_failures: int

Abstractmethod:

Return the default maximum number of consecutive failures.

In some cases, aborting the testsuite when there are just too many failures saves time and costs: the software to test/environment is too broken, there is no point to continue running the testsuite.

This property must return the number of test failures (FAIL or ERROR) that trigger the abortion of the testuite. If zero, this behavior is disabled.

property default_failure_exit_code: int

Abstractmethod:

Return the default exit code when at least one test fails.

property auto_generate_text_report: bool

Abstractmethod:

Return whether to automatically generate a text report.

This is disabled by default (and controlled by the –generate-text-report command-line option) because the generation of this report can add non-trivial overhead depending on results.

abstractmethod adjust_dag_dependencies(dag: e3.collection.dag.DAG) → None

Adjust dependencies in the DAG of all test fragments.

Parameters:

• dag – DAG to adjust.

• fragments – Set of all fragments added so far to the DAG.

property multiprocessing_supported: bool

Abstractmethod:

Return whether running test fragments in subprocesses is supported.

When multiprocessing is enabled (see the “use_multiprocessing” attribute), test fragments are executed in a separate process, and the propagation of their return values is disabled (FragmentData’s “previous_values” argument is always an empty dict).

This means that multiprocessing can work only if test drivers and all code used by test fragments can be imported by subprocesses (for instance, class defined in the testsuite entry point are unavailable) and if test drivers don’t use the “previous_values” mechanism.

Testsuite authors can use the “–force-multiprocessing” testsuite option to check if this works.

class testsuite.Testsuite(root_dir: str | None = None, testsuite_name: str = 'Untitled testsuite')

Bases: TestsuiteCore

Testsuite class.

When implementing a new testsuite you should create a class that inherit from this class.

property tests_subdir: str

Return the subdirectory in which tests are looked for.

The returned directory name is considered relative to the root testsuite directory (self.root_dir).

property test_driver_map: Dict[str, Type[e3.testsuite.driver.TestDriver]]

Abstractmethod:

Return a map from test driver names to TestDriver subclasses.

Test finders will be able to use this map to fetch the test drivers referenced in testcases.

property default_driver: str | None

Return the name of the default driver for testcases.

When tests do not query a specific driver, the one associated to this name is used instead. If this property returns None, all tests are required to query a driver.

test_name(test_dir: str) → str

Compute the test name given a testcase spec.

This function can be overridden. By default it uses the name of the test directory. Note that the test name should be a valid filename (not dir seprators, or special characters such as :, …).

property test_finders: List[e3.testsuite.testcase_finder.TestFinder]

Return test finders to probe tests directories.

add_options(parser: argparse.ArgumentParser) → None

Add testsuite specific switches.

Subclasses can override this method to add their own testsuite command-line options.

Parameters:

parser – Parser for command-line arguments. See <https://docs.python.org/3/library/argparse.html> for usage.

set_up() → None

Execute operations before running the testsuite.

Before running this, command-line arguments were parsed. After this returns, the testsuite will look for testcases.

By default, this does nothing. Overriding this method allows testsuites to prepare the execution of the testsuite depending on their needs. For instance:

• process testsuite-specific options;

• initialize environment variables;

• adjust self.env (object forwarded to test drivers).

tear_down() → None

Execute operation when finalizing the testsuite.

By default, this cleans the working (temporary) directory in which the tests were run.

write_comment_file(comment_file: IO[str]) → None

Write the comment file’s content.

Parameters:

comment_file – File descriptor for the comment file. Overriding methods should only call its “write” method (or print to it).

property default_max_consecutive_failures: int

Return the default maximum number of consecutive failures.

In some cases, aborting the testsuite when there are just too many failures saves time and costs: the software to test/environment is too broken, there is no point to continue running the testsuite.

This property must return the number of test failures (FAIL or ERROR) that trigger the abortion of the testuite. If zero, this behavior is disabled.

property default_failure_exit_code: int

Return the default exit code when at least one test fails.

property auto_generate_text_report: bool

Return whether to automatically generate a text report.

This is disabled by default (and controlled by the –generate-text-report command-line option) because the generation of this report can add non-trivial overhead depending on results.

adjust_dag_dependencies(dag: e3.collection.dag.DAG) → None

Adjust dependencies in the DAG of all test fragments.

Parameters:

• dag – DAG to adjust.

• fragments – Set of all fragments added so far to the DAG.

property multiprocessing_supported: bool

Return whether running test fragments in subprocesses is supported.

When multiprocessing is enabled (see the “use_multiprocessing” attribute), test fragments are executed in a separate process, and the propagation of their return values is disabled (FragmentData’s “previous_values” argument is always an empty dict).

This means that multiprocessing can work only if test drivers and all code used by test fragments can be imported by subprocesses (for instance, class defined in the testsuite entry point are unavailable) and if test drivers don’t use the “previous_values” mechanism.

Testsuite authors can use the “–force-multiprocessing” testsuite option to check if this works.

compute_use_multiprocessing() → bool

Return whether to use multi-processing for tests parallelism.

See docstring for the “use_multiprocessing” attribute. Subclasses are free to override this to take control of when multiprocessing is enabled. Note that this will disregard the “–force-multiprocessing” command line option.