.. _api_diff: ``e3.testsuite.driver.diff``: Test driver for actual/expected outputs ===================================================================== The ``driver.diff`` module defines ``DiffTestDriver``, a ``ClassicTestDriver`` subclass specialized for drivers whose analysis is based on output comparisons. It also defines several helper classes to control more precisely the comparison process. Basic API --------- The whole ``ClassicTestDriver`` API is available in ``DiffTestDriver``, and the overriding requirements are the same: * subclasses must override the ``run`` method; * they can, if need be, override the ``set_up`` and ``tear_down`` methods. Note however that unlike its parent class, it provides an actually useful ``compute_failures`` method override, which compares the test actual output and the output baseline: * The test actual output is what the ``self.output`` ``Log`` instance holds: this is where the ``analyze_output`` from the :ref:`shell method ` matters. * The output baseline, which we could also call the *test expected output*, is by default the content of the ``test.out`` file, in the test directory. As explained :ref:`below `, this defalut can be changed. Thanks to this subclass, writing real world test drivers requires little code. The following example just runs the ``my_program`` executable with arguments provided in the ``test.yaml`` file, and checks that its status code is 0 and that its output matches the content of the ``test.yaml`` file: .. code-block:: python from e3.testsuite.driver.diff import DiffTestDriver class MyTestDriver(DiffTestDriver): def run(self): argv = self.test_env.get("argv", []) self.shell(["my_program"] + argv) Output encodings ---------------- See :ref:`classic_output_encodings` for basic notions regarding string encoding/decoding concerns in ``ClassicTestDriver`` and all its subclasses. In binary mode (the ``default_encoding`` property returns ``binary``), ``self.output`` is initialized to contain a ``Log`` instance holding ``bytes``. The ``shell`` method doesn't decode process outputs: they stay as ``bytes`` and thus their concatenation to ``self.output`` is valid. In addition, the baseline file (``test.out`` by default) is read in binary mode, so in the end, ``DiffTestDriver`` only deals with ``bytes`` instances. Conversely, in text mode, ``self.output`` is a ``Log`` instance holding ``str`` objects, which the ``shell`` method extends with decoded process outputs, and finally the baseline file is read in text mode, decoded using the same string encoding. Handling output variations -------------------------- In some cases, program outputs can contain unpredictible parts. For instance, the following script: .. code-block:: python o = object() print(o) Can have the following output: .. code-block:: text $ python foo.py ... or the following: .. code-block:: text $ python foo.py Although it's theoretically possible to constrain the execution environment enough to make the printed address constant, it is hardly practical. There can be a lot of other sources of output variation: printing the current date, timing information, etc. ``DiffTestDriver`` provides two alternative mechanisms to handle such cases: match actual output against regular expressions, or refine outputs before the comparison. Regexp-based matching ********************* Instead of providing a file that contains byte-per-byte or codepoint-per-codepoint expected output, the baseline can be considered as a regular expression. With this mechanism, the following ``test.out``: .. code-block:: text will match the output of the ``foo.py`` example script above. This relies on Python's standard ``re`` module: please refer to `its documentation `_ for the syntax reference and the available regexp features. In order to switch to regexp-matching on a per-testcase basis, just add the following to the ``test.yaml`` file: .. code-block:: yaml baseline_regexp: True .. _api_diff_output_refining: Output refining *************** Another option to match varying outputs is to refine them, i.e. perform substitutions to hide varying parts from the comparison. Applied to the previous example, the goal is to refine such outputs: .. code-block:: text To a string such as following: .. code-block:: text To achieve this goal, the ``driver.diff`` module defines the following abstract class: .. code-block:: python class OutputRefiner: def refine(self, output): raise NotImplementedError Subclasses must override the ``refine`` method so that it takes the original output (``output`` argument) and return the refined output. Note that depending on the encoding, ``output`` can be either a string (``str`` instance) or binary data (``bytes`` instance): in each case it must return an object that has the same type as the ``output`` argument. Several very common subclasses are available in ``driver.diff``: ``Substitute(substring, replacement="")`` Replace a specific substring. For instance: .. code-block:: python # Just remove occurences of # (replace them with an empty string) Substitute("") # Replace occurences of with Substitute("", "") ``ReplacePath(path, replacement="")`` Replace a specific filename: ``path`` itself, the corresponding absolute path or the corresponding Unix-style path. ``PatternSubstitute(pattern, replacements="")`` Replace anything matching the ``pattern`` regular expression. Using output refiners from ``DiffTestDriver`` instances is very easy: just override the ``output_refiners`` property in subclasses to return a list of ``OutputRefiner`` to apply on actual outputs before comparing them with baselines. To complete the ``foo.py`` example above, thanks to the following overriding: .. code-block:: python @property def output_refiners(self): return [PatternSubstitute("0x[0-9a-f]+", "[HEX-ADDR]")] All refined outputs from ``foo.py`` would match the following baseline: .. code-block:: text Note that even though refiners only apply to actual outputs by default, it is possible to also apply them to baselines. To do this, override the ``refine_baseline`` property: .. code-block:: python @property def refine_baseline(self): return True This behavior is disabled by default because a very common refinment is to remove occurences of the working directory from the test output. In that case, baselines that contain the working directory (for instance ``/home/user/my-testsuite/tmp/my-test``) will be refined as expected with the setup of the original testcase author, but will not on another setup (for instance when the working directory is ``/tmp/testsuite-tmp-dir``). .. _api_diff_alternative_baselines: Alternative baselines --------------------- ``DiffTestDriver`` subclasses can override two properties in order to select the baseline to use as well as the output matching mode (equality vs. regexp): The ``baseline_file`` property must return a ``(filename, is_regexp)`` couple. The first item is the name of the baseline file (relative to the test directory), i.e. the file that contains the output baseline. The second one is a boolean that determines whether to use the regexp matching mode (if true) or the equality mode (if false). If, for some reason (for instance: extracting the baseline is more involved than just reading the content of a file) the above is not powerful enough, it is possible instead to override the ``baseline`` property. In that case, the ``baseline_file`` property is ignored, and ``baseline`` must return a 3-element tuple: 1. The absolute filename for the baseline file, if any, ``None`` otherwise. Only a present filename allows :ref:`baseline rewriting `. 2. The baseline itself: a string in text mode, and a ``bytes`` instance in binary mode. 3. Whether the baseline is a regexp. .. _api_diff_rewriting: Automatic baseline rewriting ---------------------------- Often, test baselines depend on formatting rules that need to evolve over time. For example, imagine a testsuite for a program that keeps track of daily min/max temperatures. The following could be a plausible test baseline: .. code-block:: text 01/01/2020 260.3 273.1 01/02/2020 269.2 273.2 At some point, it is decided to change the format for dates. All baselines need to be rewritten, so the above must become: .. code-block:: text 2020-01-01 260.3 273.1 2020-01-02 269.2 273.2 That implies manually rewriting the baselines of potentially a lot of tests. ``DiffTestDriver`` makes it possible to automatically rewrite baselines for all tests based on equality (not regexps). Of course, this is disabled by default: one needs to run it only when such pervasive output changes are expected, and baseline updates need to be carefully reviewed afterwards. Enabling this behavior is as simple as setting ``self.env.rewrite_baselines`` to True in the ``Testsuite`` instance. The APIs to use for this are properly introduced later, in :ref:`api_testsuite`. Here is a short example, in the meantime: .. code-block:: python class MyTestsuite(Testsuite): # Add a command-line flag to the testsuite script to allow users to # trigger baseline rewriting. def add_options(self, parser): parser.add_argument( "--rewrite", action="store_true", help="Rewrite test baselines according to current outputs" ) # Before running the testsuite, keep track in the environment of our # desire to rewrite baselines. DiffTestDriver instances will pick it up # automatically from there. def set_up(self): super(MyTestsuite, self).set_up() self.env.rewrite_baselines = self.main.args.rewrite Note that baseline rewriting applies only to tests that are not already expected to fail. Imagine for instance the situation described above (date format change), and the following testcase: .. code-block:: yaml # test.yaml control: - [XFAIL, "True", "Precision bug: max temperature is 280.1 while it should be 280.0"] .. code-block:: text # test.out 01/01/2020 270.3 280.1 The testsuite must not rewrite ``test.out``, otherwise the precision bug (``280.1`` instead of ``280.0``) will be recorded in the baseline, and thus the testcase will incorrectly start to pass (``XPASS``). But this is just a compromise: in the future, the testcase will fail not only because of the lack of precision, but also because of the bad date formatting, so in such cases, baselines must be manually updated.