Incompatible with pytest-xdist / thread safety

[rassie]

Describe the bug

I'm using pytest-xdist to run my tests in parallel (mostly a single test parameterized for a vast amount of different test data). Since I wanted to try snapshot-based testing, I've added syrupy to the mix. After initial creation of __snapshots__/mytest.ambr (using the --snapshot-update), it disappears from disk at some point during the test run. Without pytest-xdist (i.e. without using -n auto option) this doesn't happen.

To reproduce

I don't have a clean reproduction, only an observation. My setup is a single test for a bit of processing, parameterized with a couple of hundreds different input files. While the .ambr file appear initially, after about 80% of tests done, the file disappears.

Expected behavior

Snapshot files get updated with correct data without disappearing, even when used with pytest-xdist or pytest-parallel.

Environment:

• OS: Ubuntu 20.04
• Syrupy Version: 1.4.0
• Python Version: 3.8

Additional context

I assume the problem happens because each test execution writes to the file individually and thus a race condition happens. Not sure what can be done about it, maybe buffer the snapshot file somewhere and write it only at the end of the test run? There are surely better ideas. I assume that e.g. the JUnit reporter works correctly with pytest-xdist, maybe check what it does?

Either way, I wanted to bring this to your attention, would be happy if there was a short- to middle-term solution. If not, I'll run the suite without parallelization, it's not that big of a deal at the moment, at least for me :)

[noahnu]

I'll need to dig into how pytest-xdist operates. Currently when in "write mode" (snapshot update flag passed) we write immediately when making the assertion. Buffering and deferring this to the session end hook where we already delete unused snapshots shouldn't be too difficult.

[noahnu]

I'm re-examining where we do the reads/writes in an effort to solve #539. I'm hoping that work will also solve the concurrency issue with pytest-xdist.

[haakenlid]

I'm also using pytest-xdist and want to use syrupy in my test suite. I run into a different problem.

Just having syrupy installed will break pytest when using the pytest-xdist option --looponfail.

-f, --looponfail       run tests in subprocess, wait for modified files and re-
                       run failing test set until all pass.

If I uninstall syrupy the error goes away. I have not added any syrupy-specific code to any of my tests.

$ pytest --looponfail
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1400, in _save
    dispatch = self._dispatch[tp]
KeyError: <class 'abc.ABCMeta'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 162, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/usr/local/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/local/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.9/site-packages/xdist/looponfail.py", line 35, in pytest_cmdline_main
    looponfail_main(config)
  File "/usr/local/lib/python3.9/site-packages/xdist/looponfail.py", line 45, in looponfail_main
    remotecontrol.loop_once()
  File "/usr/local/lib/python3.9/site-packages/xdist/looponfail.py", line 115, in loop_once
    self.setup()
  File "/usr/local/lib/python3.9/site-packages/xdist/looponfail.py", line 77, in setup
    self.channel = channel = self.gateway.remote_exec(
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway.py", line 135, in remote_exec
    gateway_base.dumps_internal((source, file_name, call_name, kwargs)),
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1371, in dumps_internal
    return _Serializer().save(obj)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1389, in save
    self._save(obj)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1407, in _save
    dispatch(self, obj)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1494, in save_tuple
    self._save(item)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1407, in _save
    dispatch(self, obj)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1490, in save_dict
    self._write_setitem(key, value)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1484, in _write_setitem
    self._save(value)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1407, in _save
    dispatch(self, obj)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1490, in save_dict
    self._write_setitem(key, value)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1484, in _write_setitem
    self._save(value)
  File "/usr/local/lib/python3.9/site-packages/execnet/gateway_base.py", line 1405, in _save
    raise DumpError("can't serialize {}".format(tp))
execnet.gateway_base.DumpError: can't serialize <class 'abc.ABCMeta'>

[noahnu]

There's been some more attention around this issue. It's been prioritized as the next item to work on -- actually moved to In Progress. I'll post updates as they come up.

[Sharu95]

@noahnu, I'm coming from pytest-dev/pytest#9600, where I experienced the same issues as @rassie. Didn't quite manage to isolate the issue to whether it was syrupy, pytest or pytest-xdist at first, but good that I found this thread 👍 Do you have any updates of the progress on this issue/anywhere you left off that we might pick up to resolve this? :)

[noahnu]

I haven't had a chance to work on this just yet. I previously refactored some code to boost performance. Buffering the writes should still be a feasible fix here.

If you're interested in taking this on, I think we'd want to batch and then defer this operation https://github.com/tophat/syrupy/blob/ec4cbac3fa71eb7dad276005f9433ce26c2f1f9a/src/syrupy/assertion.py#L197 (the write_snapshot call), collect them for after the tests finish running, and flush them out to disk at the same time in some coordinated fashion (still need to look into pytest-xdist to see how that may be possible). The hooks are located in the __init__.py file. -- If you are interested, feel free to join our discord and I'll be available as a resource. Otherwise, I will get to this at some point. I ended up prioritizing the JSON extension over this issue, but as that's out of the way, this is definitely the next most pressing issue.

Might actually take a stab at it now.

[noahnu]

I started digging into the xdist implementation, and this won't be as straightforward as I hoped. Definitely open to help/contributions here. It looks like each worker manages its own session. So where/when should we write snapshots? Can two workers get nodes from the same test file? In which case, we might have 2 workers trying to write to the same file?

Even if we overcame that obstacle, we run into a problem with unused snapshot detection. We need to somehow defer our snapshot analysis until after all the workers have been cleaned up.

[Sharu95]

Sounds great, thanks for the update. I haven't dived into pytest-xdist, or syrupy yet, as this is my first dive into the such details and OSS contributions! I'll gladly explore this, but might take a bit more time to get into. Will join the Discord as well, for discussions, which is great anyway!

Any starting point I can base off from, when you looked at yesterday? Writer/Reader/session handler? I might contribute with some constructive ideas if I dive a bit more into the pytest-xdist details

[noahnu]

Will try take a deeper look this weekend but what needs to be clarified:

• How do we know when xdist is done and wrapping up? Is there a pytest hook?
• How can we communicate the snapshots we want to write to a single worker/controller? Is there a built in communication mechanism?

If I can't solve it immediately, I'll do some refactoring in syrupy to hopefully make it easier to reason about. I've been meaning to do some serious refactoring to the syrupy internals for a while and have been planning it for a v2 release since it'll affect plugin development.

[Sharu95]

From the looks of it, it looks like worker_collectionfinish, (triggered by the pytest_sessionfinish hook), at least what is mentioned in the docstrings, handles adding the collected items to the scheduler and initiating the schedule call:

https://github.com/pytest-dev/pytest-xdist/blob/29c4cce1d45fe317f78778b8eb0d7de681cb7290/src/xdist/dsession.py#L251-L262

from what I can see in e.g. load.py, in which the emitted behaviour is the same for the other scheduling approaches, the schedule first confirms whether all workers have collected the same test items before they are distributed:

https://github.com/pytest-dev/pytest-xdist/blob/29c4cce1d45fe317f78778b8eb0d7de681cb7290/src/xdist/scheduler/load.py#L237-L269

So with a RR schedule and an initial batch, the test items are distributed equally to each worker/node, with the runtests event. The event is caught on the other side by the pytest hook pytest_runtestloop:

https://github.com/pytest-dev/pytest-xdist/blob/29c4cce1d45fe317f78778b8eb0d7de681cb7290/src/xdist/remote.py#L74-L96

Could we maybe use the return value for this to figure out which tests are done?

The runtest_protocol_complete event sends signals for one node, but for all the nodes to be finished, I'm still digging through the source

[noahnu]

I've started some refactoring to defer the snapshot writes until after all tests finish running. Previously we were writing snapshots as part of the test case assertions. At least for writes, I think we can grab a file lock in the temp directory belonging to the xdist controller, essentially following the docs from https://pytest-xdist.readthedocs.io/en/latest/how-to.html#making-session-scoped-fixtures-execute-only-once.

Ensuring the "unused" snapshot logic still works will be a bit trickier. One naive solution would be to have each worker write the unused snapshot names to some temp file and then have the controller somehow do the cleanup by merging the unused files (i.e. a test file is only considered unused if it exists in exactly all worker files, so we essentially do the insersection of all unused).

[mcataford]

Taking a stab at this over the weekend! ⚒️

[noahnu]

Taking a stab at this over the weekend! ⚒️

@mcataford you'll want to work off of the next branch. I was running into performance issues which will need to be tackled first. We're running benchmarks against the main branch, so you can use those benchmarks as a reference.

[mcataford]

Circling back from post-weekend updates here --

I've experimented and done some diagramming to explore and understand the order-of-operation, hooks and where changes should fit in. Haven't reached a point where I have changes I can push up, but working up to it.

[philippefutureboy]

Heya 👋

Just wanted to share some hacky ideas in case they can be of help:

(1)
Could it be possible to circumvent the issue altogether by having one file per test that requests the snapshot fixture?

I understand that this would be a breaking change; this being said, a migration path that reads existing snapshots files and splits them to the new file-per-test would probably be easier than exploring the innards of the pytest-xdist plugins.

(2)
Spawn a python process using the multiprocessing python lib and using that process to coordinate writing. That way you can treat whatever happens in the tests as a black box and just write when all the listeners are done.

--

Ngl, it seems to me that the python testing tools ecosystem is fragmented to a point where it hinders the maintenance and stabilization of a rich & coherent suite of tools like in Javascript. Treating other plugins as a black box to me seem like a way to reduce headaches.

[noahnu]

Working my way through this.

#535 (comment) is addressed by #667 (going into the next branch / v4.0.0). It looks like pytest-xdist expects all pytest options to be serializable. This is not a requirement of pytest itself so I'd argue it's really a pytest-xdist limitation. That being said, it's easy enough to fix so I've added the change to the v4 release. It's a bit too niche to cover with a test but may loop back to figure out how to prevent regressions. I've added a comment in the code for now.

[noahnu]

Heya 👋

Just wanted to share some hacky ideas in case they can be of help:

(1) Could it be possible to circumvent the issue altogether by having one file per test that requests the snapshot fixture?

I understand that this would be a breaking change; this being said, a migration path that reads existing snapshots files and splits them to the new file-per-test would probably be easier than exploring the innards of the pytest-xdist plugins.

The JSON extension works like this but I suspect it also doesn't work too well with pytest-xdist. There's a use case for 1 file per test case, but I can see good arguments against it, for example consider the use of parametrization. You probably don't want 1000s of snapshot files per test case. Especially since these are expected to be committed to version control.

I'm trying my best to see if syrupy can be refactored/improved to the point where minimal work is required for pytest-xdist compatibility. Using this as an opportunity to also cleanup the architecture a bit.

(2) Spawn a python process using the multiprocessing python lib and using that process to coordinate writing. That way you can treat whatever happens in the tests as a black box and just write when all the listeners are done.

pytest-xdist already has the concept of a controller process. I'm hoping to piggyback on this. We'll see. Whatever we do to ensure pytest-xdist support, it shouldn't negatively impact our non-xdist users.

Ngl, it seems to me that the python testing tools ecosystem is fragmented to a point where it hinders the maintenance and stabilization of a rich & coherent suite of tools like in Javascript. Treating other plugins as a black box to me seem like a way to reduce headaches.

Syrupy itself doesn't leverage any "hacks" to do its job. It 100% respects the pytest plugin API. It's pytest-xdist which is doing some odd things to add parallelization to pytest. This breaks some assumptions when working with the pytest API. This is why we can't just treat pytest-xdist as a black box, because it does in fact change the way tests are run.

[noahnu]

I believe the changes I've made in #668, #669 and #605 should mostly resolve this. Once #605 is merged, I'll close this thread, but feel free to comment to re-open if there's still some incompatibility. There are some limitations which I've noted in the README.