testsuite ========= .. py:module:: testsuite .. autoapi-nested-parse:: Generic testsuite framework. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/testsuite/_helpers/index /autoapi/testsuite/control/index /autoapi/testsuite/driver/index /autoapi/testsuite/event_notifications/index /autoapi/testsuite/find_skipped_tests/index /autoapi/testsuite/fragment/index /autoapi/testsuite/main/index /autoapi/testsuite/multiprocess_scheduler/index /autoapi/testsuite/optfileparser/index /autoapi/testsuite/report/index /autoapi/testsuite/result/index /autoapi/testsuite/running_status/index /autoapi/testsuite/testcase_finder/index /autoapi/testsuite/utils/index Attributes ---------- .. autoapisummary:: testsuite.logger Exceptions ---------- .. autoapisummary:: testsuite.TestAbort Classes ------- .. autoapisummary:: testsuite.TestsuiteCore testsuite.Testsuite Package Contents ---------------- .. py:data:: logger .. py:exception:: TestAbort Bases: :py:obj:`Exception` Raise this to abort silently the execution of a test fragment. .. py:class:: TestsuiteCore(root_dir: Optional[str] = None, testsuite_name: str = 'Untitled testsuite') Testsuite Core driver. This class is the base of Testsuite class and should not be instantiated. It's not recommended to override any of the functions declared in it. See documentation of Testsuite class for overridable methods and variables. .. py:attribute:: root_dir :value: b'.' Root directory for the testsuite, i.e. directory from which the test directory (see ``self.test_dir``) is looked up. .. py:attribute:: test_dir Root directory for the tree in which testcases are searched. .. py:attribute:: return_values :type: Dict[str, Any] .. py:attribute:: result_tracebacks :type: Dict[str, List[str]] .. py:attribute:: testsuite_name :value: 'Untitled testsuite' .. py:attribute:: running_status :type: e3.testsuite.running_status.RunningStatus Object to report testsuite execution status to users and to manage abortion in case there are too many failures. .. py:attribute:: use_multiprocessing :value: False Whether to use multi-processing for tests parallelism. Beyond a certain level of parallelism, Python's GIL contention is too high to benefit from more processors. When we reach this level, it is more interesting to use multiple processes to cancel the GIL contention. The actual value for this attribute is computed once the DAG is built, in the "compute_use_multiprocessing" method. .. py:method:: _test_counter() -> int .. py:method:: _test_status_counters() -> Dict[e3.testsuite.result.TestStatus, int] .. py:method:: _results() -> Dict[str, e3.testsuite.result.TestStatus] .. py:property:: test_counter :type: int Return the number of test results in the report. Warning: this method is obsolete and will be removed in the future. .. py:property:: test_status_counters :type: Dict[e3.testsuite.result.TestStatus, int] Return test result counts per test status. Warning: this method is obsolete and will be removed in the future. .. py:property:: results :type: Dict[str, e3.testsuite.result.TestStatus] Return a mapping from test names to results. Warning: this method is obsolete and will be removed in the future. .. py:method:: compute_use_multiprocessing() -> bool :abstractmethod: Return whether to use multi-processing for tests parallelism. See docstring for the "use_multiprocessing" attribute. Subclasses are free to override this to take control of when multiprocessing is enabled. Note that this will disregard the "--force-multiprocessing" command line option. .. py:method:: testsuite_main(args: Optional[List[str]] = None) -> int Main for the main testsuite script. :param args: Command line arguments. If None, use `sys.argv`. :return: The testsuite status code (0 for success, a positive for failure). .. py:method:: get_test_list(sublist: List[str]) -> List[e3.testsuite.testcase_finder.ParsedTest] Retrieve the list of tests. :param sublist: A list of tests scenarios or patterns. .. py:method:: add_test(dag: e3.collection.dag.DAG, parsed_test: e3.testsuite.testcase_finder.ParsedTest) -> None Register a test to run. :param dag: The DAG of test fragments to execute for the testsuite. :param parsed_test: Test to instantiate. .. py:method:: dump_testsuite_result() -> None Log a summary of test results. Subclasses are free to override this to do whatever is suitable for them. .. py:method:: collect_result(fragment: e3.testsuite.fragment.TestFragment) -> None Import test results from ``fragment`` into testsuite reports. :param fragment: Test fragment (just completed) from which to import test results. .. py:method:: add_result(item: e3.testsuite.driver.ResultQueueItem) -> None Add a test result to the result index and log it. :param item: Result queue item for the result to add. .. py:method:: add_test_error(test_name: str, message: str, tb: Optional[str] = None) -> None Create and add an ERROR test status. :param test_name: Prefix for the test result to create. This adds a suffix to avoid clashes. :param str message: Error message. :param tb: Optional traceback for the error. .. py:method:: setup_result_dirs() -> None Create the output directory in which the results are stored. .. py:method:: run_standard_mainloop(dag: e3.collection.dag.DAG) -> None Run the main loop to execute test fragments in threads. .. py:method:: run_multiprocess_mainloop(dag: e3.collection.dag.DAG) -> None Run the main loop to execute test fragments in subprocesses. .. py:property:: tests_subdir :type: str :abstractmethod: Return the subdirectory in which tests are looked for. The returned directory name is considered relative to the root testsuite directory (self.root_dir). .. py:property:: test_driver_map :type: Dict[str, Type[e3.testsuite.driver.TestDriver]] :abstractmethod: Return a map from test driver names to TestDriver subclasses. Test finders will be able to use this map to fetch the test drivers referenced in testcases. .. py:property:: default_driver :type: Optional[str] :abstractmethod: Return the name of the default driver for testcases. When tests do not query a specific driver, the one associated to this name is used instead. If this property returns None, all tests are required to query a driver. .. py:method:: test_name(test_dir: str) -> str :abstractmethod: Compute the test name given a testcase spec. This function can be overridden. By default it uses the name of the test directory. Note that the test name should be a valid filename (not dir seprators, or special characters such as ``:``, ...). .. py:property:: test_finders :type: List[e3.testsuite.testcase_finder.TestFinder] :abstractmethod: Return test finders to probe tests directories. .. py:method:: add_options(parser: argparse.ArgumentParser) -> None :abstractmethod: Add testsuite specific switches. Subclasses can override this method to add their own testsuite command-line options. :param parser: Parser for command-line arguments. See for usage. .. py:method:: set_up() -> None :abstractmethod: Execute operations before running the testsuite. Before running this, command-line arguments were parsed. After this returns, the testsuite will look for testcases. By default, this does nothing. Overriding this method allows testsuites to prepare the execution of the testsuite depending on their needs. For instance: * process testsuite-specific options; * initialize environment variables; * adjust self.env (object forwarded to test drivers). .. py:method:: tear_down() -> None :abstractmethod: Execute operation when finalizing the testsuite. By default, this cleans the working (temporary) directory in which the tests were run. .. py:method:: write_comment_file(comment_file: IO[str]) -> None :abstractmethod: Write the comment file's content. :param comment_file: File descriptor for the comment file. Overriding methods should only call its "write" method (or print to it). .. py:property:: default_max_consecutive_failures :type: int :abstractmethod: Return the default maximum number of consecutive failures. In some cases, aborting the testsuite when there are just too many failures saves time and costs: the software to test/environment is too broken, there is no point to continue running the testsuite. This property must return the number of test failures (FAIL or ERROR) that trigger the abortion of the testuite. If zero, this behavior is disabled. .. py:property:: default_failure_exit_code :type: int :abstractmethod: Return the default exit code when at least one test fails. .. py:property:: auto_generate_text_report :type: bool :abstractmethod: Return whether to automatically generate a text report. This is disabled by default (and controlled by the --generate-text-report command-line option) because the generation of this report can add non-trivial overhead depending on results. .. py:method:: adjust_dag_dependencies(dag: e3.collection.dag.DAG) -> None :abstractmethod: Adjust dependencies in the DAG of all test fragments. :param dag: DAG to adjust. :param fragments: Set of all fragments added so far to the DAG. .. py:property:: multiprocessing_supported :type: bool :abstractmethod: Return whether running test fragments in subprocesses is supported. When multiprocessing is enabled (see the "use_multiprocessing" attribute), test fragments are executed in a separate process, and the propagation of their return values is disabled (FragmentData's "previous_values" argument is always an empty dict). This means that multiprocessing can work only if test drivers and all code used by test fragments can be imported by subprocesses (for instance, class defined in the testsuite entry point are unavailable) and if test drivers don't use the "previous_values" mechanism. Testsuite authors can use the "--force-multiprocessing" testsuite option to check if this works. .. py:class:: Testsuite(root_dir: Optional[str] = None, testsuite_name: str = 'Untitled testsuite') Bases: :py:obj:`TestsuiteCore` Testsuite class. When implementing a new testsuite you should create a class that inherit from this class. .. py:property:: tests_subdir :type: str Return the subdirectory in which tests are looked for. The returned directory name is considered relative to the root testsuite directory (self.root_dir). .. py:property:: test_driver_map :type: Dict[str, Type[e3.testsuite.driver.TestDriver]] :abstractmethod: Return a map from test driver names to TestDriver subclasses. Test finders will be able to use this map to fetch the test drivers referenced in testcases. .. py:property:: default_driver :type: Optional[str] Return the name of the default driver for testcases. When tests do not query a specific driver, the one associated to this name is used instead. If this property returns None, all tests are required to query a driver. .. py:method:: test_name(test_dir: str) -> str Compute the test name given a testcase spec. This function can be overridden. By default it uses the name of the test directory. Note that the test name should be a valid filename (not dir seprators, or special characters such as ``:``, ...). .. py:property:: test_finders :type: List[e3.testsuite.testcase_finder.TestFinder] Return test finders to probe tests directories. .. py:method:: add_options(parser: argparse.ArgumentParser) -> None Add testsuite specific switches. Subclasses can override this method to add their own testsuite command-line options. :param parser: Parser for command-line arguments. See for usage. .. py:method:: set_up() -> None Execute operations before running the testsuite. Before running this, command-line arguments were parsed. After this returns, the testsuite will look for testcases. By default, this does nothing. Overriding this method allows testsuites to prepare the execution of the testsuite depending on their needs. For instance: * process testsuite-specific options; * initialize environment variables; * adjust self.env (object forwarded to test drivers). .. py:method:: tear_down() -> None Execute operation when finalizing the testsuite. By default, this cleans the working (temporary) directory in which the tests were run. .. py:method:: write_comment_file(comment_file: IO[str]) -> None Write the comment file's content. :param comment_file: File descriptor for the comment file. Overriding methods should only call its "write" method (or print to it). .. py:property:: default_max_consecutive_failures :type: int Return the default maximum number of consecutive failures. In some cases, aborting the testsuite when there are just too many failures saves time and costs: the software to test/environment is too broken, there is no point to continue running the testsuite. This property must return the number of test failures (FAIL or ERROR) that trigger the abortion of the testuite. If zero, this behavior is disabled. .. py:property:: default_failure_exit_code :type: int Return the default exit code when at least one test fails. .. py:property:: auto_generate_text_report :type: bool Return whether to automatically generate a text report. This is disabled by default (and controlled by the --generate-text-report command-line option) because the generation of this report can add non-trivial overhead depending on results. .. py:method:: adjust_dag_dependencies(dag: e3.collection.dag.DAG) -> None Adjust dependencies in the DAG of all test fragments. :param dag: DAG to adjust. :param fragments: Set of all fragments added so far to the DAG. .. py:property:: multiprocessing_supported :type: bool Return whether running test fragments in subprocesses is supported. When multiprocessing is enabled (see the "use_multiprocessing" attribute), test fragments are executed in a separate process, and the propagation of their return values is disabled (FragmentData's "previous_values" argument is always an empty dict). This means that multiprocessing can work only if test drivers and all code used by test fragments can be imported by subprocesses (for instance, class defined in the testsuite entry point are unavailable) and if test drivers don't use the "previous_values" mechanism. Testsuite authors can use the "--force-multiprocessing" testsuite option to check if this works. .. py:method:: compute_use_multiprocessing() -> bool Return whether to use multi-processing for tests parallelism. See docstring for the "use_multiprocessing" attribute. Subclasses are free to override this to take control of when multiprocessing is enabled. Note that this will disregard the "--force-multiprocessing" command line option.