testsuite.driver.diff ===================== .. py:module:: testsuite.driver.diff Classes ------- .. autoapisummary:: testsuite.driver.diff.OutputRefiner testsuite.driver.diff.RefiningChain testsuite.driver.diff.Substitute testsuite.driver.diff.CanonicalizeLineEndings testsuite.driver.diff.ReplacePath testsuite.driver.diff.PatternSubstitute testsuite.driver.diff.LineByLine testsuite.driver.diff.DiffTestDriver Module Contents --------------- .. py:class:: OutputRefiner Bases: :py:obj:`Generic`\ [\ :py:obj:`AnyStr`\ ] Interface to refine a test output before baseline and actual comparison. Sometimes, the way a library/tool works forces it to have outputs that depends on the environment (for instance: the location of the testsuite on the filesystem, the current date, etc.). Refiners makes it possible for a testsuite to "hide" these discrepancies during the diff computation. Note that output refiners might get called bytes strings when the test drivers operate in binary mode. .. py:method:: refine(output: AnyStr) -> AnyStr :abstractmethod: Refine a test/baseline output. :param output: Output to refine. .. py:class:: RefiningChain(refiners: List[OutputRefiner]) Bases: :py:obj:`OutputRefiner`\ [\ :py:obj:`AnyStr`\ ] Simple wrapper for a sequence of output refiners applied in chain. .. py:attribute:: refiners .. py:method:: refine(output: AnyStr) -> AnyStr Refine a test/baseline output. :param output: Output to refine. .. py:class:: Substitute(substring: AnyStr, replacement: Optional[AnyStr] = None) Bases: :py:obj:`OutputRefiner`\ [\ :py:obj:`AnyStr`\ ] Replace substrings in outputs. .. py:attribute:: substring :type: AnyStr .. py:attribute:: replacement :type: AnyStr .. py:method:: refine(output: AnyStr) -> AnyStr Refine a test/baseline output. :param output: Output to refine. .. py:class:: CanonicalizeLineEndings Bases: :py:obj:`OutputRefiner`\ [\ :py:obj:`AnyStr`\ ] Replace \r\n with \n in outputs. .. py:method:: refine(output: AnyStr) -> AnyStr Refine a test/baseline output. :param output: Output to refine. .. py:class:: ReplacePath(path: str, replacement: str = '') Bases: :py:obj:`RefiningChain`\ [\ :py:obj:`str`\ ] Return an output refiner to replace the given path. .. py:class:: PatternSubstitute(pattern: AnyStr, replacement: AnyStr | Callable[[re.Match], AnyStr] | None = None) Bases: :py:obj:`OutputRefiner`, :py:obj:`Generic`\ [\ :py:obj:`AnyStr`\ ] Replace patterns in outputs. .. py:attribute:: regexp :type: Pattern[AnyStr] .. py:attribute:: replacement :type: AnyStr | Callable[[re.Match], AnyStr] .. py:method:: refine(output: AnyStr) -> AnyStr Refine a test/baseline output. :param output: Output to refine. .. py:class:: LineByLine(refiner: OutputRefiner) Bases: :py:obj:`OutputRefiner`\ [\ :py:obj:`AnyStr`\ ] Wrapper to apply an output refine line by line. .. py:attribute:: refiner .. py:method:: refine(output: AnyStr) -> AnyStr Refine a test/baseline output. :param output: Output to refine. .. py:class:: DiffTestDriver(env: e3.env.Env, test_env: Dict[str, Any]) Bases: :py:obj:`e3.testsuite.driver.classic.ClassicTestDriver` Test driver to compute test output against a baseline. .. py:property:: rewrite_baseline :type: bool Return whether this driver should rewrite its baseline. .. py:property:: baseline_file :type: Tuple[str, bool] Return the test output baseline file. :return: The name of the text file (relative to test directories) that contains the expected test output and whether the baseline is a regexp. .. py:property:: baseline :type: Tuple[Optional[str], Union[str, bytes], bool] Return the test output baseline. Subclasses can override this method if they want to provide a baseline that does not come from a file, short-circuiting the baseline_file property. :return: The baseline absolute filename (if any), the baseline content, as a string or as a bytes string (depending on the default encoding), and whether the baseline is a regexp. The baseline filename is used to rewrite test output: leave it to None if rewriting does not make sense. .. py:property:: output_refiners :type: List[OutputRefiner] List of refiners for test baselines/outputs. This just returns a refiner to canonicalize line endings unless the test environment contains a "strict_line_endings" key associated to true. .. py:property:: refine_baseline :type: bool Whether to apply output refiners to the output baseline. .. py:property:: diff_ignore_white_chars :type: bool Whether to ignore white characters in diff computations. This returns whether the comparison between test output and baseline must ignore whitespaces (leading and trailing spaces, tabs and carriage returns on lines, and empty lines). Note that if we don't ignore them, we still canonicalize line separators (CRLF are replaced by LF before the comparison). Note that at some point, this mechanism should be unified with the ``output_refiners`` machinery. However, this relies on e3.diff's ignore_white_chars feature, which is not trivial to reimplement. .. py:property:: diff_context_size :type: int Positive number of context lines to include in diff computations. .. py:method:: set_up() -> None Run initialization operations before a test runs. Subclasses can override this to prepare testcase execution. Having a callback separate from "run" is useful when dealing with inheritance: overriding the "set_up" method in subclasses allows to append setup actions before the testcase execution actually takes place (in the "run" method). If everything happened in "run" method, that would not be possible unless re-implementing the "run" method in each subclass, with obvious code duplication issues. .. py:method:: compute_diff(baseline_file: Optional[str], baseline: AnyStr, actual: AnyStr, failure_message: str = 'unexpected output', ignore_white_chars: Optional[bool] = None, context_size: Optional[int] = None, truncate_logs_threshold: Optional[int] = None) -> List[str] Compute the diff between expected and actual outputs. Return an empty list if there is no diff, and return a list that contains an error message based on ``failure_message`` otherwise. :param baseline_file: Absolute filename for the text file that contains the expected content (for baseline rewriting, if enabled), or None. :param actual: Actual content to compare. :param failure_message: Failure message to return if there is a difference. :param ignore_white_chars: Whether to ignore whitespaces during the diff computation. If left to None, use ``self.diff_ignore_white_chars``. :param context_size: Positive number of context lines to include in diff computations. If left to None, use ``self.diff_context_size``. :param truncate_logs_threshold: Threshold to truncate the diff message in ``self.result.log``. See ``e3.testsuite.result.truncated``'s ``line_count`` argument. If left to None, use the testsuite's ``--truncate-logs`` option. .. py:method:: compute_regexp_match(regexp: Union[Pattern[AnyStr], AnyStr], actual: AnyStr, failure_message: str = 'output does not match expected pattern', truncate_logs_threshold: Optional[int] = None) -> List[str] Compute whether the actual output matches a regexp. Return an empty list if the acutal content matches, and return a list that contains an error message based on ``failure_message`` otherwise. :param regexp: Regular expression to use. :param actual: Actual content to match. :param failure_message: Failure message to return if there is a difference. :param truncate_logs_threshold: Threshold to truncate the diff message in ``self.result.log``. See ``e3.testsuite.result.truncated``'s ``line_count`` argument. If left to None, use the testsuite's ``--truncate-logs`` option. .. py:method:: compute_failures() -> List[str] Return a failure if ``self.output.log`` does not match the baseline. This computes a diff with the content of the baseline file, unless there is a "baseline_regexp" entry in the test environment that evaluates to true. Subclasses can override this if they need more involved analysis: for instance computing multiple diffs.