python package incompatibilities to look for

[mnlevy1981]

Per a comment I made on #304, now that dask-jobqueue 0.4.0 is available I upgraded the recommended stack for working on cheyenne to reduce the number of ports we need to forward from the compute node.

I ran into two issues that were easily resolved, but I think it would be useful to mention these problems (and their solutions) here.

1. nbserverproxy: an issue that was documented on both the dask-jobqueue and the nbserverproxy boards: I was seeing a blank dashboard through http://localhost:8888/proxy/8787/status:

Even though forwarding port 8787 showed me a working dashboard on http://localhost:8787/status:

Long story short, if you are using nbserverproxy 0.8.3, you need to also use tornado 5.0 rather than the newer 5.1 release. The YAML file with the recommended conda environment for cheyenne currently contains

  - tornado=5.0.2
  - nbserverproxy=0.8.3

2. Cartopy: I was getting an error AttributeError: 'GeoAxesSubplot' object has no attribute '_hold' when trying to plot something. This traced back to an incompatibility between Cartopy 0.16.0 and matplotlib 3.0, and the solution is to specify versions for both of these packages as well:

# matplotlib 3.0.0 doesn't work with Cartopy 0.16.0, so specify 2.2.3
# see https://github.com/SciTools/cartopy/issues/1120 for details
  - matplotlib=2.2.3
  - Cartopy=0.16.0

This also led to a very old version of numpy being installed unless numpy was also included in the YAML file:

  - numpy
  - xarray
  - jupyterlab

(previous versions of pangeo-cheyenne.yaml did not include the first line in the above block).

---

What I'm really learning is that it seems vitally important to specify version numbers of all packages in these conda environment files, lest incompatibilities creep into previously-working environments. Prior to this update, I believe a new user cloning the NCAR/pangeo repository would have still encountered the Cartopy error despite the same environment working fine for me a couple months ago (before matplotlib 3.0 was released).

So it seems prudent for me to update that last block of code above to be

  - numpy=1.15.2
  - xarray=0.10.9
  - jupyterlab=0.35.1

at the added cost of needing to occasionally (monthly?) see if new releases of any packages are available / compatible with the rest of the stack.

[mrocklin]

Thank you for reporting these @mnlevy1981 !

It looks like the cartopy/matplotlib version mismatch is known. I'm somewhat concerned about the nbserverproxy situation though. We should probably try to track down if there is an issue upstream. If someone is looking for something to do it would be useful to install and run nbserverproxy locally with tornado 5.1 just to see if this also fails locally. If so, reporting the exception here would be welcome. I'd be happy to track down tornado bugs.

[mnlevy1981]

@mrocklin -- if I'm reading the conversations correctly, jupyterhub/jupyter-server-proxy#44 fixes this problem. I like your suggestion of testing this locally, though, and I should have time to do that after lunch today. So my plan of action:

1. set up a local environment with job-queue 0.4.0, nbserverproxy 0.8.3, and tornado 0.5.1; verify that the dashboard looks different when comparing :8888/proxy/8787 and :8787
2. update nbserverproxy to master, and see if the problem persists

If the second step fixes the issue, great! If not, the problem is far out of my area of expertise and it would be awesome if you could dig deeper. I'll keep you posted.

[rabernat]

It would be great if we had a test harness for pangeo clusters that would allow us to automate this process a bit. We would want to test, for example:

• Notebook and lab are accessible
• Dask dashboard is accessible
• Certain example notebooks run correctly

[mnlevy1981]

I can verify that

1. the nbserverproxy 0.8.3 / tornado 5.1 incompatibility is present on local machines, not just cheyenne
2. Updating nbserverproxy to the head of master fixes this incompatibility -- nbserverproxy commit 03449cc (Oct 10, 2018) + tornado 5.1 lets me view the dashboard where the 0.8.3 tag did not (presumably due to Only choose subprotocol if subprotocol list is non-empty jupyterhub/jupyter-server-proxy#44 though lots of other commits have come in since the last tag)

[mrocklin]

OK, so it sounds like we want to ask for a release of nbserverproxy ? Would you be interested in raising an issue on nbserverproxy with such a request, perhaps pointing to this issue as motivation?

…

On Wed, Oct 17, 2018 at 3:29 PM Michael Levy ***@***.***> wrote: I can verify that 1. the nbserverproxy 0.8.3 / tornado 5.1 incompatibility is present on local machines, not just cheyenne 2. Updating nbserverproxy to the head of master fixes this incompatibility -- nbserverproxy commit 03449cc (Oct 10, 2018) + tornado 5.1 lets me view the dashboard where the 0.8.3 tag did not (presumably due to jupyterhub/jupyter-server-proxy#44 <jupyterhub/jupyter-server-proxy#44> though lots of other commits have come in since the last tag) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#432 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszFyStliL4454un7cldl1lLM_okgyks5ul4UXgaJpZM4XiXZ4> .

[mnlevy1981]

Would you be interested in raising an issue on nbserverproxy

An issue already existed, but I left a comment referring back to here

[mrocklin]

Thanks @mnlevy1981 !

…

On Wed, Oct 17, 2018 at 4:36 PM Michael Levy ***@***.***> wrote: Would you be interested in raising an issue on nbserverproxy An issue already existed, but I left a comment referring back to here — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#432 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszDksgBgj1aE4hWgzBaeUViyMdvwmks5ul5S3gaJpZM4XiXZ4> .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.