Multiprocessing: leveraging many cores¶
In order to take advantage of multiple cores on the machine running a
testsuite, e3.testsuite
can run several tests in parallel. By default, it
uses Python threads to achieve this, which is very simple to use both for the
implementation of e3.testsuite
itself, but also for testsuite implementors.
It is also usually more efficient than using separate processes.
However there is a disadvantage to this, at least with the most common Python implementation (CPython): beyond some level of parallelism, the contention on CPython’s GIL is too high to benefit from more processors. When we reach this level, it is more interesting to use multiple processes to cancel the GIL contention.
To work around this CPython caveat, e3.testsuite
provides a non-default way
to run tests in separate processes and avoid multithreading completely, which
removes GIL contention and thus allows testsuites to run faster with many
cores.
Limitations¶
Compared to the multithreading model, running tests in separate processes adds several constraints on the implementation of test drivers:
First, all code involved in test driver execution (
TestDriver
subclasses, and all the code called by them) must be importable from subprocesses: defined in a Python module, during its initialization.Note that this means that test drivers must not be defined in the
__main__
module, i.e. not in the Python executable script that runs the testsuite, but in separate modules. This is probably the most common gotcha: the meaning of__main__
is different between the testsuite main script (for instancerun_testsuite.py
) and the internal script that will only run the test driver (e3-run-test-fragment
, built ine3.testsuite
).Test environments and results (i.e. all data exchanged between the testsuite main and the test drivers) must be compatible with Python’s standard pickle module.
There are two additional limitations that affect only users of the low level test driver API:
- Return value propagation between tests is disabled: the
previous_values
argument in the fragment callback is always the empty dict. Conversely, the fragment callback return values are always ignored. - Test driver instances are not shared between testsuite mains (when
add_test
is invoked) and each fragment: all live in separate processes and the test driver classes are re-instantiated in each process.
Enabling multiprocessing¶
The first thing to do is to check that your testsuite works despite the
limitations described above. The most simple way to check this is to pass the
--force-multiprocessing
command line flag to the testsuite. As its name
implies, it forces the use of separate processes to run test fragments (no
matter the level of parallelism).
Once this works, in order to communicate to e3.testsuite
that it can
automatically enable multiprocessing (this is done only when the parallelism
level is considered high enough for this strategy to run faster), you have to
override the Testsuite.multiprocessing_supported
property so that it
returns True
(it returns False
by default).
Advanced control of multiprocessing¶
Some testsuites may have test driver code that does not work in multithreading contexts (use of global variables, environment variables, and the like). For such testsuites, multiprocessing is not necessarily useful for performance, but is actually needed for correct execution.
These testsuites can override the Testsuite.compute_use_multiprocessing
method to override the default automatic behavior (using multiprocessing
beyond some CPU cores threshold), and always enable it. Note that this will
make the --force-multiprocessing
command line option useless.
Note that this possibility is a workaround for test driver code architectural issues, and should not be considered as a proper way to deal with parallelism.