Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IncompleteRead error while streaming to watch some apis #540

Closed
steveum0105 opened this issue May 25, 2018 · 6 comments
Closed

IncompleteRead error while streaming to watch some apis #540

steveum0105 opened this issue May 25, 2018 · 6 comments
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@steveum0105
Copy link

steveum0105 commented May 25, 2018

I'm using 6.0.0 with Kubernetes 1.9.
I'm trying using watch API like this.

        w = watch.Watch()
        for event in w.stream(
                self.api_instance_ext_v1Beta1API.
                list_ingress_for_all_namespaces,
                **kwargs):
                 ...

At first loop, it works fine.
However, I got this error from the second loop when there is no change of watching Resources.

  File "/usr/local/lib/python3.6/site-packages/kubernetes/watch/watch.py", line 124, in stream
    for line in iter_resp_lines(resp):
  File "/usr/local/lib/python3.6/site-packages/kubernetes/watch/watch.py", line 45, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 626, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 320, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

I think it's because k8s disconnects watch api after 10 minutes.
kubernetes-client intentionally let me handle this?

@steveum0105 steveum0105 changed the title I got and error while streaming to watch some apis I got an error while streaming to watch some apis May 25, 2018
@steveum0105 steveum0105 changed the title I got an error while streaming to watch some apis I got IncompleteRead error while streaming to watch some apis May 25, 2018
@steveum0105 steveum0105 changed the title I got IncompleteRead error while streaming to watch some apis IncompleteRead error while streaming to watch some apis May 29, 2018
@abushoeb
Copy link

I had issues with Watch() while using k8 v1.7 and python client 6.0. Later I changed python client to 2.0 and it's working. What I observed is that the client documentation is not updated for the latest version which 6.0.

@steveum0105
Copy link
Author

steveum0105 commented Jun 19, 2018

Now.. I'm doing like this.

from urllib3.exceptions import ProtocolError
...
w = watch.Watch()
try:              
    for event in w.stream(list_resources, **kwargs):
...
except ProtocolError:
    print('skip this error... because kubelet disconnect client after default 10m...')  
...

Is it a known exception? then I will close the issue...

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 24, 2019
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 24, 2019
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

johnbradley added a commit to Duke-GCB/calrissian that referenced this issue Aug 16, 2019
Adds tenacity library and call to retry wait_for_completion
when a ProtocolError is received.

Example exception where we saw this issue:
```
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
at _error_catcher (/usr/local/lib/python3.6/site-packages/urllib3/response.py:380)
at __exit__ (/usr/local/lib/python3.6/contextlib.py:99)
at read_chunked (/usr/local/lib/python3.6/site-packages/urllib3/response.py:696)
at stream (/usr/local/lib/python3.6/site-packages/urllib3/response.py:492)
at follow_logs (/usr/local/lib/python3.6/site-packages/calrissian/k8s.py:122)
at wait_for_completion (/usr/local/lib/python3.6/site-packages/calrissian/k8s.py:143)
at wait_for_kubernetes_pod (/usr/local/lib/python3.6/site-packages/calrissian/job.py:372)
at run (/usr/local/lib/python3.6/site-packages/calrissian/job.py:593)
at runner (/usr/local/lib/python3.6/site-packages/cwltool/executors.py:264)
```
This error occurred while the k8s master was being updated.

Issue about this exception on python kubernetes module repo:
kubernetes-client/python#540
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

4 participants