Improve exception handling for listing Kubernetes resources

[josefhandl]

When the Kubernetes API fails to list Kubernetes resources in the reflector (for example too strict permissions), no relevant error message appears in the logs, and the program fails. I'm adding a check to see if the call is successful. If not, an error with the API is printed, and an exception is thrown.

This is what the previous logs look like (enable_user_namespaces = True):

[I 2024-05-02 13:20:03.726 JupyterHub spawner:188] Using user namespace: user-jhu
[E 2024-05-02 13:20:03.739 JupyterHub reflector:412] Initial list of pods failed
    Traceback (most recent call last):
      File "/opt/kubespawner/kubespawner/reflector.py", line 410, in start
        await self._list_and_update()
      File "/opt/kubespawner/kubespawner/reflector.py", line 249, in _list_and_update
        for p in initial_resources["items"]
    KeyError: 'items'
    
[E 2024-05-02 13:20:03.739 JupyterHub spawner:2411] Reflector with key ('pods', None) failed to start.
    Traceback (most recent call last):
      File "/opt/kubespawner/kubespawner/spawner.py", line 2409, in catch_reflector_start
        await func
      File "/opt/kubespawner/kubespawner/reflector.py", line 410, in start
        await self._list_and_update()
      File "/opt/kubespawner/kubespawner/reflector.py", line 249, in _list_and_update
        for p in initial_resources["items"]
    KeyError: 'items'

...and this now:

[I 2024-05-02 13:20:26.046 JupyterHub spawner:188] Using user namespace: user-jhu
[E 2024-05-02 13:20:26.059 JupyterHub reflector:237] Error when calling Kubernetes API. Status: 403 Forbidden. Message: pods is forbidden: User "system:serviceaccount:default:jupyterhub-spawner-sa" cannot list resource "pods" in API group "" in the namespace "user-jhu"
[E 2024-05-02 13:20:26.059 JupyterHub reflector:412] Initial list of pods failed
    Traceback (most recent call last):
      File "/opt/kubespawner/kubespawner/reflector.py", line 410, in start
        await self._list_and_update()
      File "/opt/kubespawner/kubespawner/reflector.py", line 240, in _list_and_update
        raise client.ApiException(
    kubernetes_asyncio.client.exceptions.ApiException: (403)
    Reason: Forbidden
    
    
[E 2024-05-02 13:20:26.059 JupyterHub spawner:2411] Reflector with key ('pods', None) failed to start.
    Traceback (most recent call last):
      File "/opt/kubespawner/kubespawner/spawner.py", line 2409, in catch_reflector_start
        await func
      File "/opt/kubespawner/kubespawner/reflector.py", line 410, in start
        await self._list_and_update()
      File "/opt/kubespawner/kubespawner/reflector.py", line 240, in _list_and_update
        raise client.ApiException(
    kubernetes_asyncio.client.exceptions.ApiException: (403)
    Reason: Forbidden

[welcome]

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct.

You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[yuvipanda]

Can you use self.log.exception here instead, and pass it the exception you're creating?

[josefhandl]

Yes, but as far as I know, it's not possible without a try-catch. I've pushed a fix

[consideRatio]

Based on python docs:

This method should only be called from an exception handler.

So I think you were forced to do that for this to make sense. But, doesn't this mean we get stack trace details from raising the error a few lines above, and that in turn doesn't help us get informed about the original error anyhow associated with getting a response where response.ok == False?

[consideRatio]

With that in mind, was the original thing better or not? I'm not sure - I'll defer to @yuvipanda or someone with more experience with this than me to make a decision and then we go for it.

[consideRatio]

@yuvipanda I tried catching up with differences of log.error and log.exception, but I've not come to a conclusion if 0f2555d was what you wanted - could you review this again?

[yuvipanda]

I think either works, and given that the ultimate goal is to figure out better error handling, I'm just going to merge this.

Thanks for your patience @josefhandl and thanks for the ping, @consideRatio