[Bug]: juju loses connection with namespaced resource

[beliaev-maksim]

Description

we see this error when try to remove application as soon as we add custom namespaced resource with CRD

as soon as we comment out the test test_seldon_predictor_server, then Juju works fine. If we keep it, then test_remove_with_resources_present fails with following trace

Traceback (most recent call last):
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 674, in transfer_data
    message = yield from self.read_message()
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 742, in read_message
    frame = yield from self.read_data_frame(max_size=self.max_size)
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 815, in read_data_frame
    frame = yield from self.read_frame(max_size)
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 880, in read_frame
    frame = yield from Frame.read(
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/websockets/framing.py", line 99, in read
    data = yield from reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/maksim/git/kubeflow/seldon-core-operator/tests/integration/test_charm.py", line 394, in test_remove_with_resources_present
    await ops_test.model.remove_application(app_name=APP_NAME, block_until_done=True)
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 908, in remove_application
    await self.applications[app_name].remove()
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/juju/application.py", line 233, in destroy
    return await app_facade.Destroy(application=self.name)
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 483, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/juju/client/_client13.py", line 1139, in Destroy
    reply = await self.rpc(msg)
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 656, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 605, in rpc
    await self._ws.send(outgoing)
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 462, in send
    yield from self.ensure_open()
  File "/home/maksim/git/kubeflow/seldon-core-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 656, in ensure_open
    raise ConnectionClosed(
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason

Urgency

Annoying bug in our test suite

Python-libjuju version

2.9.42.4

Juju version

2.9.42 and 2.9.43

Reproduce / Test

try to run `tox -e integration` on the main branch of https://github.com/canonical/seldon-core-operator/

[juanmanuel-tirado]

Some quick comments.

• Was this working before?
• Can this be reproduced using the juju CLI?
• I see you're using and python < 3.10 (i cannot running in my environment without installing old envs) which is using and old version of the websockets library. Not sure if that could be an issue, but that's not good for sure

[beliaev-maksim]

@juanmanuel-tirado
I think it was broken, not sure. But I tested pylibjuju 3.0.x and it also fails

I test on py3.8

no, when everything crashes and I still have model there. I just run juju remove-application X and it works fine

[beliaev-maksim]

@juanmanuel-tirado here is smaller reproducible, same tox instruction

https://github.com/canonical/seldon-core-operator/tree/minimal_for_juju

[github-actions]

This issue is marked as incomplete because it has been open 30 days with no activity. Please remove incomplete label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.