AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func' #276

prokher · 2020-07-25T14:48:50Z

Dear Colleagues,

enabling this plugin on the DjangoChannelsGraphqlWs gives me the traceback below. It is worth mentioning that it has worked fine some months ago. Unfortunately I could not track what particular change made this happen, probably Python version upgrade or PyLInt version upgrade, I am not sure. Anyway, currently I cannot make it running and had to disable pylint-django otherwise PyLint fails.

Traceback (most recent call last):
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/lib/python3.8/site-packages/pylint/__init__.py", line 22, in run_pylint
    PylintRun(sys.argv[1:])
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/lib/python3.8/site-packages/pylint/lint/run.py", line 344, in __init__
    linter.check(args)
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 874, in check
    check_parallel(
  File "/Volumes/Dev/DjangoChannelsGraphqlWs/.venv/lib/python3.8/site-packages/pylint/lint/check_parallel.py", line 92, in check_parallel
    with multiprocessing.Pool(jobs, initializer=initializer, initargs=[linter]) as pool:
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/prokher/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func'

Here is pip freeze output:

pip freeze 
WARNING: Could not find setup.py for directory /Volumes/Dev/DjangoChannelsGraphqlWs (tried all parent directories)
aiohttp==3.6.2
aioredis==1.3.1
aniso8601==7.0.0
apipkg==1.5
appdirs==1.4.4
appnope==0.1.0
asgiref==3.2.10
astroid==2.4.2
async-timeout==3.0.1
attrs==19.3.0
autobahn==20.7.1
Automat==20.2.0
backcall==0.2.0
black==19.10b0
cffi==1.14.0
cfgv==3.0.0
channels==2.4.0
channels-redis==3.0.1
chardet==3.0.4
click==7.1.2
constantly==15.1.0
coverage==5.2.1
cryptography==3.0
daphne==2.5.0
decorator==4.4.2
distlib==0.3.1
Django==3.0.8
-e git+https://gitlab.sd.datadvance.net/sd/pseven/DjangoChannelsGraphqlWs.git@ce8d5936a4dda3489c3d8951380f24745264106a#egg=django_channels_graphql_ws
execnet==1.7.1
filelock==3.0.12
graphene==2.1.8
graphene-django==2.12.1
graphql-core==2.3.2
graphql-relay==2.0.1
hiredis==1.1.0
hyperlink==19.0.0
identify==1.4.25
idna==2.10
incremental==17.5.0
ipython==7.16.1
ipython-genutils==0.2.0
isort==4.3.21
jedi==0.17.2
lazy-object-proxy==1.4.3
mccabe==0.6.1
more-itertools==8.4.0
msgpack==1.0.0
multidict==4.7.6
mypy==0.782
mypy-extensions==0.4.3
nodeenv==1.4.0
packaging==20.4
parso==0.7.1
pathspec==0.8.0
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
plumbum==1.6.9
pre-commit==2.1.1
promise==2.3
prompt-toolkit==3.0.3
ptyprocess==0.6.0
py==1.9.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pydocstyle==5.0.2
Pygments==2.6.1
PyHamcrest==2.0.2
pylint==2.5.3
pylint-django==2.2.0
pylint-plugin-utils==0.6
pylint-quotes==0.2.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
pytest==5.4.3
pytest-asyncio==0.14.0
pytest-cov==2.10.0
pytest-django==3.9.0
pytest-forked==1.2.0
pytest-pythonpath==0.7.3
pytest-xdist==1.33.0
pytz==2020.1
PyYAML==5.3.1
regex==2020.7.14
Rx==1.6.1
service-identity==18.1.0
singledispatch==3.4.0.3
six==1.15.0
snowballstemmer==2.0.0
sqlparse==0.3.1
toml==0.10.1
tox==3.18.0
traitlets==4.3.3
Twisted==20.3.0
txaio==20.4.1
typed-ast==1.4.1
typing-extensions==3.7.4.2
Unidecode==1.1.1
virtualenv==20.0.28
wcwidth==0.2.5
wrapt==1.12.1
yarl==1.4.2
zope.interface==5.1.0

The text was updated successfully, but these errors were encountered:

ivasic · 2020-08-17T13:21:21Z

Setting jobs=1 in .pylintrc seems to help, as a temporary workaround.

tizbecool · 2020-11-01T18:02:12Z

Looks like it's coming from https://github.com/PyCQA/pylint-plugin-utils/blob/master/pylint_plugin_utils/__init__.py#L52

the inner function augment_func() is not pickable and with python3.8 on macOS, spawn mode is now enable by default instead of fork previously for multiprocessing
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

See: pylint-dev/pylint-django#276

dineshtrivedi · 2021-12-23T13:15:33Z

Hi, @tizbecool any suggestions on how to go forward with this fix?
I wouldn't mind trying it, but I am not sure yet how to start

dineshtrivedi · 2021-12-24T10:38:19Z

I started working on this PR - pylint-dev/pylint-plugin-utils#21
I would really appreciate any feedback

carlio · 2021-12-28T15:29:07Z

pylint-plugin-utils version 0.7 just got pushed to PyPI which has the fix that @dineshtrivedi made in it - so that should hopefully fix this issue.

I'll close it because I am an optimist, but please re-open if it's not fixed or if there are other side-effects that turn up :-)

dineshtrivedi · 2022-01-21T14:26:42Z

Thank you @carlio
It seems that the release fixed the issue partially. There are two other problems

The first:

Traceback (most recent call last):
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/__init__.py", line 24, in run_pylint
    PylintRun(sys.argv[1:])
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/run.py", line 398, in __init__
    linter.check(args)
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1000, in check
    check_parallel(
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/parallel.py", line 113, in check_parallel
    pool = multiprocessing.Pool(  # pylint: disable=consider-using-with
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'generic_is_view_attribute.<locals>.is_attribute'

After fixing it, there is a second one:

Traceback (most recent call last):
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/__init__.py", line 24, in run_pylint
    PylintRun(sys.argv[1:])
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/run.py", line 398, in __init__
    linter.check(args)
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1000, in check
    check_parallel(
  File "/Users/Aatiqah/Documents/Lumkani/lumkani-v2/.lumkani_v2/lib/python3.8/site-packages/pylint/lint/parallel.py", line 113, in check_parallel
    pool = multiprocessing.Pool(  # pylint: disable=consider-using-with
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/Aatiqah/.pyenv/versions/3.8.7/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'is_class.<locals>.<lambda>'

Pierre-Sassoulas · 2022-01-22T08:43:05Z

In pylint we had to add a dependency to dill to be able to pickle in threads, see pylint-dev/pylint#5609

dineshtrivedi · 2022-01-24T07:21:49Z

Hello @Pierre-Sassoulas

Thank you for your response. So in your point of view, there is no value in accepting the PR I made as it will be fixed once pylint 2.13.0 is released? I am fine with it if that is the case.

Pierre-Sassoulas · 2022-01-24T13:20:11Z

So in your point of view, there is no value in accepting the PR I made as it will be fixed once pylint 2.13.0 is released?

I'm not saying that, just sharing the way we handled a similar problem. Seeing your comment before mine it seemed the problem was only partially resolved. Maybe the link provided in pylint-dev/pylint#5609 would be helpful ?

Also now that you say it, I'm thinking that starting from pylint 2.13 using dill to make your life easier would make sense as you'll have it as a dependency through pylint anyway.

carlio · 2022-01-24T13:30:36Z

It does suggest that the pylint 2.13 would have fixed this problem already, but I think that the change in the pickling on pylint-django (and pylint-plugin-utils) to use the class-based approach that @dineshtrivedi created is a good idea anyway. For backwards compatability if nothing else.

@dineshtrivedi I'll merge this in because I think it's forwards compatible with planned pylint changes.

@Pierre-Sassoulas dill sounds like it'd solve any other upstream issues but rather than rely on it, for what is a small change, might as well just be a bit more robust.

dineshtrivedi · 2022-01-25T10:16:00Z

Thank you @carlio

I have made one last change to include a unit test to make sure the plugin remains picklable after future changes. I thought could be a good idea. Maybe once pylint 2.13.0 is out we can consider using dill for the unit test instead of pickle.

The change:
https://github.com/PyCQA/pylint-django/pull/348/files#diff-0100e2ebad96c65eae259f46064a7f7488d6d3ee48ac911c805ed8534710f0a9R117

dineshtrivedi · 2022-01-27T06:24:13Z

Hi, @carlio Is there anything that I can do to assist you in getting this merged?

dineshtrivedi · 2022-02-16T09:31:51Z

I am sorry for the insistence, @carlio, and @Pierre-Sassoulas
Would be possible to merge the PR? I have been waiting for this to be merged so we can bump the python version in our project.

Pierre-Sassoulas · 2022-02-16T12:25:17Z

Sorry I'm not a maintainer in this project, I can't merge.

carlio · 2022-02-16T13:02:28Z

@dineshtrivedi Sorry for taking forever, I got distracted by life for a while. It's merged in now, and release 2.5.1 is now on PyPI with the fix.

dineshtrivedi · 2022-02-16T14:25:45Z

Amazing, thank you @carlio
I really appreciate it.
And no problem, I understand it takes time.

sergeyklay added a commit to sergeyklay/branch that referenced this issue Apr 18, 2021

Set pylint jobs to 1

2e20c5e

See: pylint-dev/pylint-django#276

dineshtrivedi mentioned this issue Dec 24, 2021

AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func' pylint-dev/pylint-plugin-utils#20

Closed

carlio closed this as completed Dec 28, 2021

dineshtrivedi mentioned this issue Jan 21, 2022

276 - Fixing Macos AttributeError: Can't pickle local object #348

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func' #276

AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func' #276

prokher commented Jul 25, 2020

ivasic commented Aug 17, 2020

tizbecool commented Nov 1, 2020 •

edited

Loading

dineshtrivedi commented Dec 23, 2021 •

edited

Loading

dineshtrivedi commented Dec 24, 2021

carlio commented Dec 28, 2021

dineshtrivedi commented Jan 21, 2022

Pierre-Sassoulas commented Jan 22, 2022

dineshtrivedi commented Jan 24, 2022

Pierre-Sassoulas commented Jan 24, 2022

carlio commented Jan 24, 2022 •

edited

Loading

dineshtrivedi commented Jan 25, 2022 •

edited

Loading

dineshtrivedi commented Jan 27, 2022

dineshtrivedi commented Feb 16, 2022

Pierre-Sassoulas commented Feb 16, 2022

carlio commented Feb 16, 2022

dineshtrivedi commented Feb 16, 2022

AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func' #276

AttributeError: Can't pickle local object 'augment_visit.<locals>.augment_func' #276

Comments

prokher commented Jul 25, 2020

ivasic commented Aug 17, 2020

tizbecool commented Nov 1, 2020 • edited Loading

dineshtrivedi commented Dec 23, 2021 • edited Loading

dineshtrivedi commented Dec 24, 2021

carlio commented Dec 28, 2021

dineshtrivedi commented Jan 21, 2022

Pierre-Sassoulas commented Jan 22, 2022

dineshtrivedi commented Jan 24, 2022

Pierre-Sassoulas commented Jan 24, 2022

carlio commented Jan 24, 2022 • edited Loading

dineshtrivedi commented Jan 25, 2022 • edited Loading

dineshtrivedi commented Jan 27, 2022

dineshtrivedi commented Feb 16, 2022

Pierre-Sassoulas commented Feb 16, 2022

carlio commented Feb 16, 2022

dineshtrivedi commented Feb 16, 2022

tizbecool commented Nov 1, 2020 •

edited

Loading

dineshtrivedi commented Dec 23, 2021 •

edited

Loading

carlio commented Jan 24, 2022 •

edited

Loading

dineshtrivedi commented Jan 25, 2022 •

edited

Loading