Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stash tests fail on Windows #16938

Open
gsnedders opened this issue May 21, 2019 · 1 comment
Open

stash tests fail on Windows #16938

gsnedders opened this issue May 21, 2019 · 1 comment

Comments

@gsnedders
Copy link
Member

The tests from #14024 fail on Windows because multiprocessing on Windows uses its "spawn" start method which relies on being able to pickle everything (as there is no fork on Windows), and they currently can't be pickled.

Found in #16936, filing mostly to have something to link to from xfail definitions.

e.g.,

______________________________ test_delayed_dict ______________________________

add_cleanup = <function add at 0x0000000005E61B38>

    def test_delayed_dict(add_cleanup):
        """Ensure that delays in proxied `dict` retrieval do not interfere with
        initialization in parallel threads."""
    
        class SlowDict(BaseManager):
            pass
    
        request_lock = multiprocessing.Lock()
        response_lock = multiprocessing.Lock()
    
        queue = multiprocessing.Queue()
    
        # This request handler allows the caller to delay execution of a thread
        # which has requested a proxied representation of the "get_dict" property.
        def mutex_dict_request():
            """This request handler allows the caller to delay execution of a
            thread which has requested a proxied representation of the `get_dict`
            property, simulating a "slow" interprocess communication channel."""
            request_lock.release()
            response_lock.acquire()
            return {}
    
        SlowDict.register("get_dict", callable=mutex_dict_request)
        SlowDict.register("Lock", callable=lambda: threading.Lock())
    
        slowdict = SlowDict(("localhost", 4543), b"some key")
>       slowdict.start()

wptserve\tests\test_stash.py:121: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Python27amd64\Lib\multiprocessing\managers.py:524: in start
    self._process.start()
C:\Python27amd64\Lib\multiprocessing\process.py:130: in start
    self._popen = Popen(self)
C:\Python27amd64\Lib\multiprocessing\forking.py:277: in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
C:\Python27amd64\Lib\multiprocessing\forking.py:199: in dump
    ForkingPickler(file, protocol).dump(obj)
C:\Python27amd64\Lib\pickle.py:224: in dump
    self.save(obj)
C:\Python27amd64\Lib\pickle.py:331: in save
    self.save_reduce(obj=obj, *rv)
C:\Python27amd64\Lib\pickle.py:425: in save_reduce
    save(state)
C:\Python27amd64\Lib\pickle.py:286: in save
    f(self, obj) # Call unbound method with explicit self
C:\Python27amd64\Lib\pickle.py:655: in save_dict
    self._batch_setitems(obj.iteritems())
C:\Python27amd64\Lib\pickle.py:687: in _batch_setitems
    save(v)
C:\Python27amd64\Lib\pickle.py:286: in save
    f(self, obj) # Call unbound method with explicit self
C:\Python27amd64\Lib\multiprocessing\forking.py:67: in dispatcher
    self.save_reduce(obj=obj, *rv)
C:\Python27amd64\Lib\pickle.py:401: in save_reduce
    save(args)
C:\Python27amd64\Lib\pickle.py:286: in save
    f(self, obj) # Call unbound method with explicit self
C:\Python27amd64\Lib\pickle.py:554: in save_tuple
    save(element)
C:\Python27amd64\Lib\pickle.py:286: in save
    f(self, obj) # Call unbound method with explicit self
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <multiprocessing.forking.ForkingPickler instance at 0x000000000578AF08>
obj = <class 'test_stash.SlowDict'>, name = 'SlowDict'
pack = <built-in function pack>

    def save_global(self, obj, name=None, pack=struct.pack):
        write = self.write
        memo = self.memo
    
        if name is None:
            name = obj.__name__
    
        module = getattr(obj, "__module__", None)
        if module is None:
            module = whichmodule(obj, name)
    
        try:
            __import__(module)
            mod = sys.modules[module]
            klass = getattr(mod, name)
        except (ImportError, KeyError, AttributeError):
            raise PicklingError(
                "Can't pickle %r: it's not found as %s.%s" %
>               (obj, module, name))
E           PicklingError: Can't pickle <class 'test_stash.SlowDict'>: it's not found as test_stash.SlowDict

C:\Python27amd64\Lib\pickle.py:754: PicklingError
gsnedders added a commit that referenced this issue May 21, 2019
gsnedders added a commit that referenced this issue May 22, 2019
gsnedders added a commit that referenced this issue May 22, 2019
moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this issue Jun 19, 2019
…ittests, a=testonly

Automatic update from web-platform-tests
Disable EOL conversion for all of WPT

In various tests, we expect files to be byte-for-byte identical to
what git stores, and therefore we cannot apply any conversion that
core.autocrlf or core.eol would apply.

--
Fix test_path_lints.py on Windows

--
Alter manifest test generation on Windows

Specifically, don't generate paths with a mixture of / and \

--
Mark stash tests as xfail on Windows

This is web-platform-tests/wpt#16938

--
Update Windows xfail declarations

--
Fix OpenSSL on Python 3 on Windows

--
Manifest items should always have os.path.sep separators

Previously, when loading from JSON, we'd always use the path
separators in the JSON file (always "/").

--
Use cmd to run pip on Windows

Powershell throws an exception if anything is written to stderr,
for example the Python 2.7 deprecation notice or the cache failing

--
Add Windows 2019 test jobs for Python unittests

--

wp5At-commits: 2aef503a5b83fd08f57b7277533293773176b445, 1c74776c231fe455f366ca6b1511dc40ac4317c5, c2f4e9f574dd2647c5cfbc17226c2f66b05ccbc3, e2e4b9f5e5093e9a7b761cedb5937af9f462f2c9, 08d35775c3550c41e9d834a848dd5de77452068e, eaa285dde18846df518b8081b0d1f86997e06580, 6cee5d181a8424bd88ecaefc1a7e4507471de484, cc2cf1a7ae0c77b8ec825792819a07cde228ee55, 97127be8f797a9a26fb141ae091a2a0957de55d0
wpt-pr: 16936
xeonchen pushed a commit to xeonchen/gecko that referenced this issue Jun 19, 2019
…ittests, a=testonly

Automatic update from web-platform-tests
Disable EOL conversion for all of WPT

In various tests, we expect files to be byte-for-byte identical to
what git stores, and therefore we cannot apply any conversion that
core.autocrlf or core.eol would apply.

--
Fix test_path_lints.py on Windows

--
Alter manifest test generation on Windows

Specifically, don't generate paths with a mixture of / and \

--
Mark stash tests as xfail on Windows

This is web-platform-tests/wpt#16938

--
Update Windows xfail declarations

--
Fix OpenSSL on Python 3 on Windows

--
Manifest items should always have os.path.sep separators

Previously, when loading from JSON, we'd always use the path
separators in the JSON file (always "/").

--
Use cmd to run pip on Windows

Powershell throws an exception if anything is written to stderr,
for example the Python 2.7 deprecation notice or the cache failing

--
Add Windows 2019 test jobs for Python unittests

--

wp5At-commits: 2aef503a5b83fd08f57b7277533293773176b445, 1c74776c231fe455f366ca6b1511dc40ac4317c5, c2f4e9f574dd2647c5cfbc17226c2f66b05ccbc3, e2e4b9f5e5093e9a7b761cedb5937af9f462f2c9, 08d35775c3550c41e9d834a848dd5de77452068e, eaa285dde18846df518b8081b0d1f86997e06580, 6cee5d181a8424bd88ecaefc1a7e4507471de484, cc2cf1a7ae0c77b8ec825792819a07cde228ee55, 97127be8f797a9a26fb141ae091a2a0957de55d0
wpt-pr: 16936
marcoscaceres pushed a commit that referenced this issue Jul 23, 2019
@gsnedders
Copy link
Member Author

These also seem to fail on macOS on Python 3.8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant