Crashed Worker tracking issue #35309
Labels
Comments
TomAugspurger
added
CI
I've encountered this one while working on #35214. It happened when the python version is 3.7.0. in my case. Bumping version to 3.7.1 fixed it. This specific test caused a Segmentation fault, therefore, crushed the worker. I reproduced it on my local machine using this environment. name: pandas-dev
channels:
- conda-forge
dependencies:
- python=3.7.0
# tools
- cython=0.29.16
- pytest=5.0.1
- pytest-xdist>=1.21
- hypothesis>=3.58.0
- pytest-azurepipelines
- psutil
# pandas dependencies
- beautifulsoup4=4.6.0
- bottleneck=1.2.1
- jinja2=2.10
- numba=0.46.0
- numexpr=2.6.8
- numpy=1.16.0
- openpyxl=2.5.7
- pytables=3.4.4
- python-dateutil=2.7.3
- pytz=2017.3
- scipy=1.2
- xlrd=1.1.0
- xlsxwriter=1.0.2
- xlwt=1.3.0
- html5lib=1.0.1(pandas-test) fangchenli@fangchenli-home-office:~/Workspace/pandas-VirosaLi$ pytest pandas/tests/io/json/test_json_table_schema.py::TestTableOrient
============================================================================================================================================= test session starts =============================================================================================================================================
platform linux -- Python 3.7.0, pytest-5.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/fangchenli/Workspace/pandas-VirosaLi, inifile: setup.cfg
plugins: forked-1.2.0, xdist-1.33.0, azurepipelines-0.8.0, hypothesis-5.19.1
collected 47 items
pandas/tests/io/json/test_json_table_schema.py ..........................................Fatal Python error: Segmentation fault
Current thread 0x00007f54ac64d740 (most recent call first):
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 166 in __getitem__
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 646 in _parse
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 426 in _parse_sub
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 816 in _parse
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 426 in _parse_sub
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_parse.py", line 930 in parse
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/sre_compile.py", line 764 in compile
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/re.py", line 286 in _compile
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/re.py", line 234 in compile
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/dataclasses.py", line 196 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 728 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/core/dtypes/inference.py", line 420 in is_dataclass
File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/core/frame.py", line 500 in __init__
File "/home/fangchenli/Workspace/pandas-VirosaLi/pandas/tests/io/json/test_json_table_schema.py", line 619 in test_timestamp_in_columns
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/python.py", line 165 in pytest_pyfunc_call
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/python.py", line 1451 in runtest
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 117 in pytest_runtest_call
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 192 in <lambda>
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 220 in from_call
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 192 in call_runtest_hook
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 167 in call_and_report
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 87 in runtestprotocol
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/runner.py", line 72 in pytest_runtest_protocol
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 278 in pytest_runtestloop
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 257 in _main
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 213 in wrap_session
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/main.py", line 250 in pytest_cmdline_main
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/home/fangchenli/miniconda3/envs/pandas-test/lib/python3.7/site-packages/_pytest/config/__init__.py", line 74 in main
File "/home/fangchenli/miniconda3/envs/pandas-test/bin/pytest", line 11 in <module>
Segmentation fault (core dumped) |
|
Interesting, thanks. The job at https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=39324&view=logs&j=266f9266-03bd-52fe-4891-1cfba8ecd83f&t=1190d2de-f7ad-53d5-9425-18b19f76e994 is using 3.7.0. |
TomAugspurger
added a commit
to TomAugspurger/pandas
that referenced
this issue
Jul 16, 2020
TomAugspurger
added a commit
that referenced
this issue
Jul 17, 2020
Closed
|
What's strange about this test is that if you look at the backtrace of a core dump the segfault seems to occur somewhere in the standard lib before ever executing any of the code. My guess is that there is a bug in 3.7.0 specifically with module initialization but haven't found anything conclusive. Strange that it would only happen with one test though... |
|
I encountered the second failure listed in the table while working on #35214. But in a different environment. https://travis-ci.org/github/pandas-dev/pandas/jobs/712331711 The environment is for the coverage test for python 3.7. name: pandas-dev
channels:
- defaults
- conda-forge
dependencies:
- python=3.7.*
# tools
- cython>=0.29.16
- pytest>=5.0.1,<6.0.0rc0
- pytest-xdist>=1.21
- hypothesis>=3.58.0
- pytest-cov # this is only needed in the coverage build
# pandas dependencies
- beautifulsoup4
- botocore>=1.11
- cython>=0.29.16
- dask
- fastparquet>=0.3.2
- fsspec>=0.7.4
- gcsfs>=0.6.0
- geopandas
- html5lib
- matplotlib
- moto
- nomkl
- numexpr
- numpy=1.16.*
- odfpy
- openpyxl
- pandas-gbq
- psycopg2
- pyarrow>=0.13.0
- pymysql
- pytables
- python-snappy
- pytz
- s3fs>=0.4.0
- scikit-learn
- scipy
- sqlalchemy
- statsmodels
- xarray
- xlrd
- xlsxwriter
- xlwt
- pip
- pip:
- brotlipy
- coverage
- pandas-datareader
- python-dateutil
- pyxlsb |
|
Looks like we've dropped 3.7 in our CI and the other builds haven't segfaulted in a while so I think we can close this issue for now. We can open a new issue if this occurs with another build |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've been seeing a lot of CI builds failing recently, but I have no idea what's causing them. Can maintainers add them to the table here when they occur?
TestTableOrientReader::test_read_json_table_orient[vals0-index]testst_unique.py::test_unique_bad_unicode[idx_or_series_w_bad_unicode0]The text was updated successfully, but these errors were encountered: