Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support python 3.9 #1720

Merged
merged 7 commits into from
Jan 5, 2021
Merged

Support python 3.9 #1720

merged 7 commits into from
Jan 5, 2021

Conversation

benclifford
Copy link
Collaborator

@benclifford benclifford commented May 20, 2020

This PR makes changes so that CI can pass with Python 3.9.

No changes to core parsl code seem to be necessary to pass CI.

  • This PR brings in upgrades for test/build dependencies that were not working with 3.9: mypy, sphinx.

  • One test used platform, but datatypes from platform cannot be passed around any more, so that test is changed to not use platform.

@yadudoc yadudoc added this to the 1.1 milestone May 21, 2020
@yadudoc
Copy link
Member

yadudoc commented May 21, 2020

Tagging this for next release.

@benclifford benclifford changed the title Test parsl using (only) python 3.9 development version Support python 3.9 Jan 2, 2021
@benclifford
Copy link
Collaborator Author

This was failing mypy. I upgraded mypy to a more recent version to address this python 3.9 breakage: python/mypy#8627

@benclifford
Copy link
Collaborator Author

The test suite hangs on parsl/tests/site_tests/test_provider.py.

Switching to LSST parsl which has better debugging info, I get this error:

ERROR    parsl.process_loggers:process_loggers.py:25 exception wrapper: caught an exception for thread HTEX-Queue-Management-Thread
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 20, in wrapped
    r = fn(*args, **kwargs)
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/executor.py", line 404, in _queue_management_worker
    result = deserialize(msg['result'])
  File "/home/benc/parsl/src/parsl/parsl/serialize/facade.py", line 121, in deserialize
    result = self.methods_for_code[header].deserialize(payload)
  File "/home/benc/parsl/src/parsl/parsl/serialize/concretes.py", line 28, in deserialize
    data = pickle.loads(chomped)
TypeError: <lambda>() takes 6 positional arguments but 7 were given

This leads to https://bugs.python.org/issue42189 which suggest there is some cross version change to platform, which is used by that test case - in the linked bug this is to do with deepcopy, but in the above error it is to do with pickle. It's likely those are related.

I don't think there's any particular need to use platform in this test case, so a fix would probably be to do something more trivial there.

@benclifford
Copy link
Collaborator Author

This now fails the doc build repeably:

Warning, treated as error:
error while formatting arguments for parsl.app.errors.MissingOutputs: type object 'list' has no attribute '_special'
Makefile:52: recipe for target 'html' failed
make: *** [html] Error 2
The command "make SPHINXOPTS=-W html" exited with 2.

@benclifford
Copy link
Collaborator Author

…rallel test dependency which is apparently needed now for doc build
@benclifford benclifford marked this pull request as ready for review January 5, 2021 09:52
@yadudoc yadudoc self-assigned this Jan 5, 2021
Copy link
Member

@yadudoc yadudoc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes look good to go.

@yadudoc yadudoc merged commit 49099c5 into master Jan 5, 2021
@yadudoc yadudoc deleted the benc-python-3.9-dev branch January 5, 2021 17:25
benclifford added a commit that referenced this pull request Apr 22, 2021
Python 3.9 has been supported by parsl since #1720
benclifford added a commit that referenced this pull request Apr 28, 2021
Python 3.9 has been supported by parsl since #1720
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants