when use exec api tail xxx.log error "'utf-8' codec can't decode bytes" #717

laixiaohuai · 2019-01-04T07:13:17Z

When I'am exec pod use tail -200f xxx.log.The following error is often reported

Exception in thread Thread-426:
Traceback (most recent call last):
File "/data/pyenv/versions/3.7.0/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/data/SeaOPS/Base/api/views_file/kubernetes.py", line 591, in run
requests[fileno] = self.chan.read_stdout()
File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 122, in read_stdout
return self.read_channel(STDOUT_CHANNEL, timeout=timeout)
File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 86, in read_channel
ret = self.peek_channel(channel, timeout)
File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 78, in peek_channel
self.update(timeout=timeout)
File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 182, in update
data = data.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4095-4096: unexpected end of data

The utf-8 decode error is currently seen on the source side

Here's some of my code
..........
while self.chan.is_open():
events = self.epoll.poll()
# print("events")
if not self.chan.is_open():
logger.info('container stream connect is closed')
self.websocket.send(bytes("container stream connect is closed", 'utf-8'))
self.websocket.close()
for fileno, event in events:
# print("adsfadsaf:", fileno, event)
if event & select.EPOLLIN:
requests[fileno] = self.chan.read_stdout()
# try:
# requests[fileno] = self.chan.read_stdout()
# except Exception as ee:
# logger.info(str(ee))
self.websocket.send(bytes(requests[fileno], 'utf-8'))
elif event & select.EPOLLHUP:
# print('------------EPOLLHUP-----------')
# 注销对此socket连接的关注
self.epoll.unregister(fileno)
# 关闭socket连
self.websocket.close()
............

Can anyone help me see why that is?Is it a Bug or something else?

leavest · 2019-01-11T03:37:23Z

I get the same error when execute vim command.
my code sample

from kubernetes import client, stream
self.core_api = client.CoreV1Api()
self.stream = stream.stream(
            self.core_api.connect_get_namespaced_pod_exec,
            name=name,
            namespace=namespace,
            container=container,
            command=command,
            stdin=True,
            stdout=True,
            stderr=True,
            tty=True,
            _preload_content=False
        )

error message:

Exception in thread Thread-180:
Traceback (most recent call last):
  File "/Users/zuili/project/apps/cloud/tools/terminal.py", line 38, in run
    if self.stream.peek_stderr():
  File "/Users/zuili/.pyenv/versions/env_opsoa/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 130, in peek_stderr
    return self.peek_channel(STDERR_CHANNEL, timeout=timeout)
  File "/Users/zuili/.pyenv/versions/env_opsoa/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 78, in peek_channel
    self.update(timeout=timeout)
  File "/Users/zuili/.pyenv/versions/env_opsoa/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 180, in update
    data = data.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 30: invalid start byte

I print the data in "kubernetes/stream/ws_client.py", line 180, and the last one raise UnicodeDecodeError

b'\x01' <class 'bytes'>
b'\x01' <class 'bytes'>
b'\x01root@test-f54787bd8-v9n26:/# ' <class 'bytes'>
b'\x01v' <class 'bytes'>
b'\x01i' <class 'bytes'>
b'\x01m' <class 'bytes'>
b'\x01\r\n' <class 'bytes'>
b'\x01\x1b[?1000h\x1b[?1049h\x1b[?1h\x1b=\x1b[2;1H\xbd\x1b[6n\x1b[2;1H  \x1b[1;1H' <class 'bytes'>

version
kubernetes==8.0.0
python==3.5.5
os==10.13.6

leavest · 2019-01-16T03:10:48Z

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。

# kubernetes/stream/ws_client.py  line 182
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")

laixiaohuai · 2019-01-16T03:12:54Z

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。
# kubernetes/stream/ws_client.py  line 82
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")

你是修改源码了是吧

leavest · 2019-01-16T03:13:37Z

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。
# kubernetes/stream/ws_client.py  line 82
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")
你是修改源码了是吧

是的

laixiaohuai · 2019-01-16T03:14:16Z

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。
# kubernetes/stream/ws_client.py  line 82
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")
你是修改源码了是吧
是的

好的，我也试试，测看看结果

laixiaohuai · 2019-01-16T03:23:23Z

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。
# kubernetes/stream/ws_client.py  line 82
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")
你是修改源码了是吧
是的
好的，我也试试，测看看结果

2019-01-16 11:23:02,712 - ERROR - kubernetes.py[line:713] - get - 'NoneType' object has no attribute 'fileno' Exception in thread Thread-430: Traceback (most recent call last): File "/data/pyenv/versions/3.7.0/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/data/SeaOPS/Base/api/views_file/kubernetes.py", line 587, in run self.chan.update(timeout=300) File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 181, in update data = data.decode(encode or "utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4096: unexpected end of data
我还会报错，纳闷

leavest · 2019-01-16T09:30:56Z

我还会报错，纳闷

你使用哪个 websocket 的包。
试一下换个 websocket 的包吧
我用 channels 、Flask-Sockets 都可以，用 dwebsocket 又有另外的问题

laixiaohuai · 2019-01-16T09:32:21Z

我还会报错，纳闷

你使用哪个 websocket 的包。
试一下换个 websocket 的包吧
我用 channels 、Flask-Sockets 都可以，用 dwebsocket 又有另外的问题

我用的就是dwebsocket

kubeoperater · 2019-03-19T09:41:09Z

麻烦请教，django如果使用channels怎么实现web terminal呢

gvalentinivmw · 2019-04-19T18:17:46Z

I ran into the same issue for the port-forward api. The proposed fix worked for me. Any chance to get it merged?

Giorgio

aaronjwood · 2019-04-19T18:20:17Z

The above patch also worked on my end with Python 3.7

fejta-bot · 2019-07-18T19:02:04Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2019-08-17T20:02:56Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2019-09-16T20:47:05Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2019-09-16T20:47:12Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

leavest mentioned this issue Apr 20, 2019

Fix stream decode error kubernetes-client/python-base#130

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 18, 2019

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 17, 2019

k8s-ci-robot closed this as completed Sep 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

when use exec api tail xxx.log error "'utf-8' codec can't decode bytes" #717

when use exec api tail xxx.log error "'utf-8' codec can't decode bytes" #717

laixiaohuai commented Jan 4, 2019

leavest commented Jan 11, 2019

leavest commented Jan 16, 2019 •

edited

Loading

laixiaohuai commented Jan 16, 2019

leavest commented Jan 16, 2019

laixiaohuai commented Jan 16, 2019

laixiaohuai commented Jan 16, 2019

leavest commented Jan 16, 2019 •

edited

Loading

laixiaohuai commented Jan 16, 2019

kubeoperater commented Mar 19, 2019

gvalentinivmw commented Apr 19, 2019

aaronjwood commented Apr 19, 2019

fejta-bot commented Jul 18, 2019

fejta-bot commented Aug 17, 2019

fejta-bot commented Sep 16, 2019

k8s-ci-robot commented Sep 16, 2019

when use exec api tail xxx.log error "'utf-8' codec can't decode bytes" #717

when use exec api tail xxx.log error "'utf-8' codec can't decode bytes" #717

Comments

laixiaohuai commented Jan 4, 2019

When I'am exec pod use tail -200f xxx.log.The following error is often reported

The utf-8 decode error is currently seen on the source side

leavest commented Jan 11, 2019

leavest commented Jan 16, 2019 • edited Loading

laixiaohuai commented Jan 16, 2019

leavest commented Jan 16, 2019

laixiaohuai commented Jan 16, 2019

laixiaohuai commented Jan 16, 2019

leavest commented Jan 16, 2019 • edited Loading

laixiaohuai commented Jan 16, 2019

kubeoperater commented Mar 19, 2019

gvalentinivmw commented Apr 19, 2019

aaronjwood commented Apr 19, 2019

fejta-bot commented Jul 18, 2019

fejta-bot commented Aug 17, 2019

fejta-bot commented Sep 16, 2019

k8s-ci-robot commented Sep 16, 2019

leavest commented Jan 16, 2019 •

edited

Loading

leavest commented Jan 16, 2019 •

edited

Loading