contourf issues with mpl3.8

[rcomer]

Description

I am sorry for raising this so late in the day, but I only today thought to test my "real work" against the upcoming releases. contourf seems occasionally very broken.

Code to reproduce

import cartopy.crs as ccrs
import matplotlib.pyplot as plt
import netCDF4
import numpy as np

fname = "z500_djf_bias.nc"

file2read = netCDF4.Dataset(fname,'r')
lons = file2read.variables['longitude'][:]
lats = file2read.variables['latitude'][:]
data = file2read.variables['unknown'][:]

fig, ax = plt.subplots(subplot_kw=dict(projection=ccrs.PlateCarree()))

levels = [-50, -40, -30, -20, -10, 10, 20, 30, 40, 50]
ax.contourf(lons, lats, data, levels=levels, cmap="RdBu_r", extend="both")
ax.coastlines()

z500_djf_bias.nc.zip

With Cartopy 0.21.1 and Matplotlib 3.7.2 I get

With Cartopy main and Matplotlib main I get

I have a variety of these plots and only some are affected, so I guess it's a function of exactly where the contours end up. Hence I attach a NetCDF file rather than attempting to create an example with dummy data.

Plotting the data array directly with Matplotlib looks sensible, and manually wrapping my data so the longitudes match the projection also gives me a sensible plot.

east = lons <= 180.
wrapped_lons = np.concatenate((lons[~east] - 360, lons[east]))
wrapped_data = np.concatenate((data[:, ~east], data[:, east]), axis=1)

fig, ax = plt.subplots(subplot_kw=dict(projection=ccrs.PlateCarree()))
ax.contourf(wrapped_lons, lats, wrapped_data, levels=levels, cmap="RdBu_r", extend="both")
ax.coastlines()

So I think this might be specific to Cartopy's transforms, though I've no idea whether the fix should be here or in Matplotlib.

Full environment definition

Operating system

RHEL7

Cartopy version

main

conda list

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alsa-lib                  1.2.9                hd590300_0    conda-forge
antlr-python-runtime      4.7.2           py311h38be061_1003    conda-forge
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
atk-1.0                   2.38.0               hd4edc92_1    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
blosc                     1.21.4               h0f2a231_0    conda-forge
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotli-python             1.0.9           py311ha362b79_9    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cairo                     1.16.0            hbbf8b49_1016    conda-forge
cairocffi                 1.6.0              pyhd8ed1ab_0    conda-forge
cartopy                   0.21.1.dev183+ga9412011          pypi_0    pypi
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cf-units                  3.2.0           py311h1f0f07a_0    conda-forge
cffi                      1.15.1          py311h409f033_3    conda-forge
cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
cftime                    1.6.2           py311h4c7f6c3_1    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
click                     8.1.5           unix_pyh707e725_0    conda-forge
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.0           py311h9547e67_0    conda-forge
coverage                  7.2.7           py311h459d7ec_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cython                    0.29.36         py311hb755f60_0    conda-forge
cytoolz                   0.12.0          py311hd4cff14_1    conda-forge
dask-core                 2023.7.0           pyhd8ed1ab_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
distlib                   0.3.7              pyhd8ed1ab_0    conda-forge
distributed               2023.7.0           pyhd8ed1ab_0    conda-forge
esmf                      8.4.2           nompi_ha7f9e30_1    conda-forge
esmpy                     8.4.2              pyhc1e730c_1    conda-forge
exceptiongroup            1.1.2              pyhd8ed1ab_0    conda-forge
execnet                   2.0.2              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
filelock                  3.12.2             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.41.0          py311h459d7ec_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
fsspec                    2023.6.0           pyh1a96a4e_0    conda-forge
gdk-pixbuf                2.42.10              h6b639ba_2    conda-forge
geos                      3.11.2               hcb278e6_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
glib                      2.76.4               hfc55251_0    conda-forge
glib-tools                2.76.4               hfc55251_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
graphviz                  8.1.0                h28d9a01_0    conda-forge
gst-plugins-base          1.22.4               hf7dbed1_1    conda-forge
gstreamer                 1.22.4               h98fc4e7_1    conda-forge
gtk2                      2.24.33              h90689f9_2    conda-forge
gts                       0.7.6                h977cf35_4    conda-forge
harfbuzz                  7.3.0                hdb3a94d_0    conda-forge
hdf4                      4.2.15               h501b40f_6    conda-forge
hdf5                      1.14.1          nompi_h4f84152_100    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
identify                  2.5.24             pyhd8ed1ab_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
imagehash                 4.3.1              pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib-resources       6.0.0              pyhd8ed1ab_1    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
importlib_resources       6.0.0              pyhd8ed1ab_1    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
ipython                   8.14.0             pyh41d4057_0    conda-forge
iris-sample-data          2.4.0              pyhd8ed1ab_0    conda-forge
jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py311h4dd048b_1    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.15                 haa2dc70_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcap                    2.67                 he9d0100_0    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libclang                  15.0.7          default_h7634d5b_2    conda-forge
libclang13                15.0.7          default_h9986a30_2    conda-forge
libcups                   2.3.3                h36d4200_3    conda-forge
libcurl                   8.1.2                h409715c_0    conda-forge
libdeflate                1.18                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgd                     2.3.3                hfa28ad5_6    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libgirepository           1.76.1               h8bf6c18_0    conda-forge
libglib                   2.76.4               hebfc3b9_0    conda-forge
libglu                    9.0.0             hac7e632_1002    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
libllvm15                 15.0.7               h5cf9203_2    conda-forge
libmo_unpack              3.1.2             hf484d3e_1001    conda-forge
libnetcdf                 4.9.2           nompi_he09a3a9_107    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.3                 hbcd7760_1    conda-forge
librsvg                   2.56.1               h98fae49_0    conda-forge
libsndfile                1.2.0                hb75c966_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libsystemd0               253                  h8c4010b_1    conda-forge
libtiff                   4.5.1                h8b53f26_0    conda-forge
libtool                   2.4.7                h27087fc_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp                   1.3.1                hbf2b3c1_0    conda-forge
libwebp-base              1.3.1                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxkbcommon              1.5.0                h5d7e998_3    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libxslt                   1.1.37               h0054252_1    conda-forge
libzip                    1.9.2                hc929e4a_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lxml                      4.9.3           py311h1a07684_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markupsafe                2.1.3           py311h459d7ec_0    conda-forge
matplotlib                3.8.0.dev1559+g3165eaf2f8          pypi_0    pypi
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mo_pack                   0.2.0           py311h4c7f6c3_1008    conda-forge
mpg123                    1.31.3               hcb278e6_0    conda-forge
msgpack-python            1.0.5           py311ha3edf6b_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.33               hf1915f5_2    conda-forge
mysql-libs                8.0.33               hca2cd23_2    conda-forge
nc-time-axis              1.4.1              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
netcdf-fortran            4.6.1           nompi_hec59055_101    conda-forge
netcdf4                   1.6.4           nompi_py311h9a7c333_101    conda-forge
nodeenv                   1.8.0              pyhd8ed1ab_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
numpy                     1.25.1          py311h64a7726_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.1.1                hd590300_1    conda-forge
owslib                    0.29.2             pyhd8ed1ab_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    2.0.3           py311h320fe9a_1    conda-forge
pango                     1.50.14              heaa33ce_1    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.4.0              pyhd8ed1ab_0    conda-forge
pathlib2                  2.3.7.post1     py311h38be061_2    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.0.0          py311h0b84326_0    conda-forge
pip                       23.2               pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
platformdirs              3.9.1              pyhd8ed1ab_0    conda-forge
pluggy                    1.2.0              pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
pooch                     1.7.0              pyha770c72_3    conda-forge
pre-commit                3.3.3              pyha770c72_0    conda-forge
proj                      9.2.1                ha643af7_0    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
psutil                    5.9.5           py311h2582759_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio-client         16.1                 hb77b528_4    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pybind11                  2.11.0          py311h9547e67_0    conda-forge
pybind11-global           2.11.0          py311h9547e67_0    conda-forge
pycairo                   1.24.0          py311h8feb60e_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
pygobject                 3.44.1          py311he2aa4b6_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyproj                    3.6.0           py311ha169711_1    conda-forge
pyqt                      5.15.7          py311ha74522f_3    conda-forge
pyqt5-sip                 12.11.0         py311hcafe171_3    conda-forge
pyshp                     2.3.1              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.4.0              pyhd8ed1ab_0    conda-forge
pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
pytest-mpl                0.16.1             pyhd8ed1ab_0    conda-forge
pytest-xdist              3.3.1              pyhd8ed1ab_0    conda-forge
python                    3.11.4          hab00c5b_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-stratify           0.3.0           py311h1f0f07a_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python-xxhash             3.2.0           py311h2582759_0    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pywavelets                1.4.1           py311hcb2cf0a_0    conda-forge
pyyaml                    6.0             py311hd4cff14_5    conda-forge
qt-main                   5.15.8              hf9e2b05_14    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
scipy                     1.11.1          py311h64a7726_0    conda-forge
scitools-iris             3.7.0.dev27              pypi_0    pypi
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
setuptools-scm            7.1.0              pyhd8ed1ab_0    conda-forge
setuptools_scm            7.1.0                hd8ed1ab_0    conda-forge
shapely                   2.0.1           py311h54d622a_1    conda-forge
sip                       6.7.9           py311hb755f60_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.42.0               h2c6b66d_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.3.2           py311h459d7ec_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
udunits2                  2.2.28               hc3e0081_0    conda-forge
ukkonen                   1.0.1           py311h4dd048b_3    conda-forge
urllib3                   2.0.3              pyhd8ed1ab_1    conda-forge
virtualenv                20.24.0            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
wheel                     0.40.0             pyhd8ed1ab_1    conda-forge
wxpython                  4.2.1           py311h5c9e8ae_0    conda-forge
xcb-util                  0.4.0                hd590300_1    conda-forge
xcb-util-image            0.4.0                h8ee46fc_1    conda-forge
xcb-util-keysyms          0.4.0                h8ee46fc_1    conda-forge
xcb-util-renderutil       0.3.9                hd590300_1    conda-forge
xcb-util-wm               0.4.1                h8ee46fc_1    conda-forge
xkeyboard-config          2.39                 hd590300_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xf86vidmodeproto     2.3.1             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xxhash                    0.8.1                h0b41bf4_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zict                      3.0.0              pyhd8ed1ab_0    conda-forge
zipp                      3.16.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.2                hfc55251_7    conda-forge

[greglucas]

Can you install cartopy==main and matplotlib==3.7.2 into that environment instead of main and see which result you get? I think that would tell us whether it needs to be addressed by Cartopy or Matplotlib.

[greglucas]

This only manifests itself with Matplotlib main, so it seems to be due to Matplotlib's contour update and not any of the Cartopy changes we added. (Cartopy 0.21 + matplotlib main also shows this issue)

I sort of wonder if we are getting the right path code splitting from the new compound paths? i.e. now we have one list with possibly multiple CLOSEPOLYs in the middle [1, 2, 2, 2, ..., 2, 79, 1, 2, 2, 2, 79, ...] instead of two separate paths with one CLOSEPOLY per path. But that is pure speculation on my part, and I won't have much time to dig into this. Maybe someone else has some ideas though...
https://github.com/SciTools/cartopy/blob/main/lib/cartopy/mpl/patch.py

I think this is worth holding up the release for as it seems like there is probably something larger going on with the new contouring updates than we were expecting.

[rcomer]

The problem contour is at 40: if we remove that then the plot looks fine. So a better example to see what's going on might be with levels = [30, 40], cmap="RdYlBu_r".

Good:

Bad:

It looks like the yellow patch is the inverse of what it should be - so is this a direction thing, clockwise vs counterclockwise?

[rcomer]

I think I found it. There is a tiny region around 82S, 160E which should be yellow in #2224 (comment). In the old PathCollection, this region is its own path and all the other yellow is another path. Now that these are all a single path, I think we need to modify the logic that decides "external" and "internal" here:

cartopy/lib/cartopy/mpl/patch.py

Lines 170 to 183 in 3ff2b13

	# If geom is a Polygon and is contained within the last geom in
	# collection, add it to its list of internal polygons, otherwise
	# simply append it as a new external geom.
	if geom.is_empty:
	pass
	elif (len(collection) > 0 and
	isinstance(collection[-1][0], sgeom.Polygon) and
	isinstance(geom, sgeom.Polygon) and
	collection[-1][0].contains(geom.exterior)):
	collection[-1][1].append(geom.exterior)
	elif isinstance(geom, sgeom.Point):
	other_result_geoms.append(geom)
	else:
	collection.append((geom, []))

I made a change there that fixes the current example. I need to tidy up and do some more testing but hopefully I will have a PR by the end of the day 🤞