e3.testsuite.driver =================== .. py:module:: e3.testsuite.driver Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/e3/testsuite/driver/adacore/index /autoapi/e3/testsuite/driver/classic/index /autoapi/e3/testsuite/driver/diff/index Attributes ---------- .. autoapisummary:: e3.testsuite.driver.ResultQueue Classes ------- .. autoapisummary:: e3.testsuite.driver.GAIAResultFiles e3.testsuite.driver.TestResult e3.testsuite.driver.TestResultSummary e3.testsuite.driver.CleanupMode e3.testsuite.driver.ResultQueueItem e3.testsuite.driver.TestDriver e3.testsuite.driver.BasicTestDriver Functions --------- .. autoapisummary:: e3.testsuite.driver.dump_result_logs_if_needed Package Contents ---------------- .. py:class:: GAIAResultFiles Filenames for a given result in a GAIA report. In a GAIA testsuite report, each result is described as one entry (line) in the "results" text file, and several optional files. This object contains the names for the files, when present, corresponding to a given result. .. py:attribute:: result :type: Optional[str] .. py:attribute:: log :type: Optional[str] .. py:attribute:: expected :type: Optional[str] .. py:attribute:: out :type: Optional[str] .. py:attribute:: diff :type: Optional[str] .. py:attribute:: time :type: Optional[str] .. py:attribute:: info :type: Optional[str] .. py:function:: dump_result_logs_if_needed(env: e3.env.Env, result: e3.testsuite.result.TestResult, output_dir: str) -> Optional[GAIAResultFiles] Shortcut to call dump_result_logs if a GAIA report is requested. .. py:class:: TestResult(name: str, env: Optional[dict] = None, status: Optional[TestStatus] = None, msg: str = '') Bases: :py:obj:`yaml.YAMLObject` Represent a result for a given test. .. py:attribute:: yaml_loader .. py:attribute:: yaml_tag :value: '!e3.testsuite.result.TestResult' .. py:attribute:: test_name .. py:attribute:: env :value: None .. py:attribute:: status .. py:attribute:: msg :type: Optional[str] :value: '' .. py:attribute:: log .. py:attribute:: processes :type: list :value: [] .. py:attribute:: failure_reasons :type: Set[FailureReason] .. py:attribute:: expected :type: Optional[Log] :value: None .. py:attribute:: out :type: Optional[Log] :value: None .. py:attribute:: diff :type: Optional[Log] :value: None .. py:attribute:: time :type: Optional[float] :value: None .. py:attribute:: info :type: Dict[str, str] .. py:method:: set_status(status: TestStatus, msg: Optional[str] = '') -> None Update the test status. :param status: New status. Note that only test results with status set to ERROR can be changed. :param msg: Optional short message to describe the result. Note that multiline strings are turned into single-line strings. .. py:method:: __str__() -> str .. py:method:: save(results_dir: str) -> str Write this test results as a YAML file. :param results_dir: Name of the directory in which to write the test result. When writing a testsuite report, this corresponds to the report's ``results_dir`` (see ``e3.testsuite.report.index.ReportIndex``). :return: The base filename of the file written. It is generated from the testname. .. py:property:: summary :type: TestResultSummary .. py:class:: TestResultSummary Basic data about a test result. This class holds a subset of the information available in ``TestResult`` instances. This subset is meant to be small enough to be included in result indexes while enabling efficient common processings such as "show logs for tests that failed": no need to load all the test results for tests that succeeded. .. py:attribute:: test_name :type: str .. py:attribute:: status :type: TestStatus .. py:attribute:: msg :type: Optional[str] .. py:attribute:: failure_reasons :type: Set[FailureReason] .. py:attribute:: time :type: Optional[float] .. py:attribute:: info :type: Dict[str, str] .. py:class:: CleanupMode(*args, **kwds) Bases: :py:obj:`enum.Enum` Mode for working space cleanups. .. py:attribute:: NONE .. py:attribute:: PASSING .. py:attribute:: ALL .. py:method:: default() -> CleanupMode :classmethod: .. py:method:: descriptions() -> Dict[CleanupMode, str] :classmethod: .. py:class:: ResultQueueItem Information to integrate a test result in a testsuite report. Test drivers create test results. They travel from there to the testsuite final report through various queues. This class gathers all the information needed for the various stages of this pipeline. .. py:attribute:: result :type: e3.testsuite.result.TestResultSummary Summary for this test result. .. py:attribute:: filename :type: str Name of the file that contains test result data. .. py:attribute:: traceback :type: List[str] Stack trace for this result's push time. This stack trace corresponds to the code that led the TestResult instance to be included in the testsuite report. .. py:attribute:: gaia_results :type: Optional[e3.testsuite.report.gaia.GAIAResultFiles] GAIA files for this result. This is None if no GAIA report is requested. .. py:data:: ResultQueue .. py:class:: TestDriver(env: e3.env.Env, test_env: Dict[str, Any]) Bases: :py:obj:`object` Testsuite Driver. All drivers declared in a testsuite should inherit from this class .. py:attribute:: Fore :type: Any .. py:attribute:: Style :type: Any .. py:attribute:: test_env_filename :type: str .. py:attribute:: env :type: e3.env.Env .. py:attribute:: testsuite_options :type: argparse.Namespace .. py:attribute:: test_env :type: Dict[str, Any] .. py:attribute:: test_name :type: str .. py:attribute:: result :type: e3.testsuite.result.TestResult .. py:attribute:: result_queue :type: ResultQueue :value: [] Queue of test results that this driver plans to integrate to the testsuite report. .. py:method:: push_result(result: Optional[e3.testsuite.result.TestResult] = None) -> None Push a result to the testsuite. This method should be called to push results to the testsuite report. :param result: A TestResult object to push. If None push the current test result. .. py:method:: add_fragment(dag: e3.collection.dag.DAG, name: str, fun: Optional[e3.testsuite.fragment.FragmentCallback] = None, after: Optional[List[str]] = None) -> None Add a test fragment. This function is a helper to define test workflows that do not introduce dependencies to other tests. For more complex operation use directly add_vertex method from the dag. See add_test method. :param dag: DAG containing test fragments. :param name: Name of the fragment. :param fun: Callable that takes two positional arguments: a mapping from fragment names to return values for already executed fragments, and a slot ID. If None looks for a method inside this class called ``name``. :param after: List of fragment names that should be executed before this one. .. py:method:: working_dir(*args: str) -> str Build a filename in the test working directory. .. py:method:: test_dir(*args: str) -> str Build a filename in the testcase directory. .. py:method:: add_test(dag: e3.collection.dag.DAG) -> None :abstractmethod: Create the test workflow. Amend a DAG with the test fragments that should be executed along with their dependencies. See BasicTestDriver for an example of workflow. .. py:property:: working_dir_cleanup_enabled :type: bool Return whether test drivers should cleanup their working directory. Unless this returns False, test drivers should delete their working directory when the test has completed, so that temporaries for the whole testsuite are removed incrementally. This is necessary to avoid creating huge temporary directories when executing big testsuites. .. py:class:: BasicTestDriver(env: e3.env.Env, test_env: Dict[str, Any]) Bases: :py:obj:`TestDriver` Testsuite Driver. All drivers declared in a testsuite should inherit from this class .. py:method:: add_test(dag: e3.collection.dag.DAG) -> None Create a standard test workflow. set up -> run -> analyze -> tear_down in which set up and tear_down are optional. .. py:method:: set_up(prev: Dict[str, Any], slot: int) -> None Execute operations before executing a test. .. py:method:: tear_up(prev: Dict[str, Any], slot: int) -> None Backwards-compatible name for the "set_up" method. .. py:method:: tear_down(prev: Dict[str, Any], slot: int) -> None Execute operations once a test is finished. .. py:method:: run(prev: Dict[str, Any], slot: int) -> None :abstractmethod: Execute a test. .. py:method:: analyze(prev: Dict[str, Any], slot: int) -> None :abstractmethod: Compute the test result.