support websockify > 0.8 #2104

totaam · 2019-01-14T06:23:30Z

They've moved things around..

totaam · 2019-01-14T06:23:58Z

2019-01-14 06:23:58: antoine uploaded file `websockify-new-import.patch` (3.6 KiB)

work in progress, needs guess_type()

totaam · 2019-01-14T15:36:13Z

Preliminary work in r21365.

They've changed something I struggle to figure out.
Hitting the TCP port using wget:

with websockify 0.8.0:

2019-01-14 21:13:06,407 peer: (0, -1, -1)
2019-01-14 21:13:06,408 new_connection((1, <socket._socketobject object at 0x7fa88fbf8d00>)) sock=<socket._socketobject object at 0x7fa88ccc2980>, socket_info=('0.0.0.0', 10000), timeout=0.1, address=('127.0.0.1', 47214), peername=('127.0.0.1', 47214). timeout=0.1
2019-01-14 21:13:06,408 SocketConnection(<socket._socketobject object at 0x7fa88ccc2980>, ('127.0.0.1', 10000), ('127.0.0.1', 47214), ('127.0.0.1', 47214), 'tcp', {})
2019-01-14 21:13:06,408 Connection(('127.0.0.1', 47214), 'tcp', {})
2019-01-14 21:13:06,408 handle_new_connection(tcp socket: 127.0.0.1:10000 <- 127.0.0.1:47214, <socket._socketobject object at 0x7fa88ccc2980>, ('127.0.0.1', 47214), 'tcp', ('127.0.0.1', 47214), ('0.0.0.0', 10000)) sockname=('127.0.0.1', 10000), target=('127.0.0.1', 47214)
2019-01-14 21:13:06,659 socket tcp socket: 127.0.0.1:10000 <- 127.0.0.1:47214 peek: got 152 bytes
2019-01-14 21:13:06,659 socket peek='GET /index.html HTTP/1.1\r\nUser-Agent: Wget/1.20.1 (linux-gnu)\r\nAccept: */*\r\nAccept-Encoding: identity\r\nHost: localhost:10000\r\nConnection: Keep-Alive\r\n\r\n'
2019-01-14 21:13:06,659 socket peek hex=474554202f696e6465782e68746d6c20485454502f312e310d0a557365722d4167656e743a20576765742f312e32302e3120286c696e75782d676e75290d0a4163636570743a202a2f2a0d0a4163636570742d456e636f64696e673a206964656e746974790d0a486f73743a206c6f63616c686f73743a31303030300d0a436f
2019-01-14 21:13:06,659 socket peek line1='GET /index.html HTTP/1.1'
2019-01-14 21:13:06,659 may_wrap_socket(..) peek_data=474554202f696e6465782e68746d6c20485454502f312e310d0a557365722d4167656e743a20576765742f312e32302e3120286c696e75782d676e75290d0a4163636570743a202a2f2a0d0a4163636570742d456e636f64696e673a206964656e746974790d0a486f73743a206c6f63616c686f73743a31303030300d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a0d0a from 127.0.0.1:47214
2019-01-14 21:13:06,659 start_http_socket(tcp, tcp socket: 127.0.0.1:10000 <- 127.0.0.1:47214, False, ..) http proto=http, line1='GET /index.html HTTP/1.1'
2019-01-14 21:13:06,659 New http GET request received from 127.0.0.1:47214 for '/index.html'
2019-01-14 21:13:06,659 start_websockify(tcp, tcp socket: 127.0.0.1:10000 <- 127.0.0.1:47214, False, http GET, ('127.0.0.1', 47214)) www dir=/usr/share/xpra/www, headers dir=/usr/share/xpra/http-headers
2019-01-14 21:13:06,659 may_wrap_socket(..)=(False, tcp socket: 127.0.0.1:10000 <- 127.0.0.1:47214, None)
2019-01-14 21:13:06,660 send_head() script(/index.html)=None
2019-01-14 21:13:06,660 translate_path(/index.html)=/usr/share/xpra/www/index.html
2019-01-14 21:13:06,660 guess_type(/usr/share/xpra/www/index.html)=('text/html', None)
2019-01-14 21:13:06,660 accept-encoding=['identity']
2019-01-14 21:13:06,660 "GET /index.html HTTP/1.1" 200 -
2019-01-14 21:13:06,661 may_reload_headers() http headers time={}, mtime=1540297043.29
2019-01-14 21:13:06,661 may_reload_headers() loading from '/usr/share/xpra/http-headers/00_nocache.txt'
2019-01-14 21:13:06,661 may_reload_headers() loading from '/usr/share/xpra/http-headers/10_content_security_policy.txt'
2019-01-14 21:13:06,661 may_reload_headers() headers={'Expires': '0', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache, no-store, must-revalidate'}, mtime=1540297043.29
2019-01-14 21:13:06,661 handle_one_request()=None

with websockify trunk:

2019-01-14 22:35:38,578 peer: (0, -1, -1)
2019-01-14 22:35:38,578 new_connection((1, <socket._socketobject object at 0x7f2b006c0d00>)) sock=<socket._socketobject object at 0x7f2afd7d3130>, socket_info=('0.0.0.0', 10000), timeout=0.1, address=('127.0.0.1', 37130), peername=('127.0.0.1', 37130). timeout=0.1
2019-01-14 22:35:38,578 SocketConnection(<socket._socketobject object at 0x7f2afd7d3130>, ('127.0.0.1', 10000), ('127.0.0.1', 37130), ('127.0.0.1', 37130), 'tcp', {})
2019-01-14 22:35:38,578 Connection(('127.0.0.1', 37130), 'tcp', {})
2019-01-14 22:35:38,579 handle_new_connection(tcp socket: 127.0.0.1:10000 <- 127.0.0.1:37130, <socket._socketobject object at 0x7f2afd7d3130>, ('127.0.0.1', 37130), 'tcp', ('127.0.0.1', 37130), ('0.0.0.0', 10000)) sockname=('127.0.0.1', 10000), target=('127.0.0.1', 37130)
2019-01-14 22:35:38,829 socket tcp socket: 127.0.0.1:10000 <- 127.0.0.1:37130 peek: got 152 bytes
2019-01-14 22:35:38,829 socket peek='GET /index.html HTTP/1.1\r\nUser-Agent: Wget/1.20.1 (linux-gnu)\r\nAccept: */*\r\nAccept-Encoding: identity\r\nHost: localhost:10000\r\nConnection: Keep-Alive\r\n\r\n'
2019-01-14 22:35:38,830 socket peek hex=474554202f696e6465782e68746d6c20485454502f312e310d0a557365722d4167656e743a20576765742f312e32302e3120286c696e75782d676e75290d0a4163636570743a202a2f2a0d0a4163636570742d456e636f64696e673a206964656e746974790d0a486f73743a206c6f63616c686f73743a31303030300d0a436f
2019-01-14 22:35:38,830 socket peek line1='GET /index.html HTTP/1.1'
2019-01-14 22:35:38,830 may_wrap_socket(..) peek_data=474554202f696e6465782e68746d6c20485454502f312e310d0a557365722d4167656e743a20576765742f312e32302e3120286c696e75782d676e75290d0a4163636570743a202a2f2a0d0a4163636570742d456e636f64696e673a206964656e746974790d0a486f73743a206c6f63616c686f73743a31303030300d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a0d0a from 127.0.0.1:37130
2019-01-14 22:35:38,830 start_http_socket(tcp, tcp socket: 127.0.0.1:10000 <- 127.0.0.1:37130, False, ..) http proto=http, line1='GET /index.html HTTP/1.1'
2019-01-14 22:35:38,830 New http GET request received from 127.0.0.1:37130 for '/index.html'
2019-01-14 22:35:38,830 start_websockify(tcp, tcp socket: 127.0.0.1:10000 <- 127.0.0.1:37130, False, http GET, ('127.0.0.1', 37130)) www dir=/usr/share/xpra/www, headers dir=/usr/share/xpra/http-headers
2019-01-14 22:35:38,830 may_wrap_socket(..)=(False, tcp socket: 127.0.0.1:10000 <- 127.0.0.1:37130, None)
2019-01-14 22:35:38,830 127.0.0.1: Plain non-SSL (ws://) WebSocket connection
2019-01-14 22:35:38,830 127.0.0.1: Path: '/index.html'
2019-01-14 22:35:38,830 new_websocket_client(<xpra.server.websocket.WSRequestHandler instance at 0x7f2afd911368>) socket=<socket._socketobject object at 0x7f2afd7d3130>
2019-01-14 22:35:38,830 SocketConnection(<socket._socketobject object at 0x7f2afd7d3130>, ('127.0.0.1', 10000), ('127.0.0.1', 37130), ('127.0.0.1', 37130), 'ws', {})
2019-01-14 22:35:38,830 Connection(('127.0.0.1', 37130), 'ws', {})
2019-01-14 22:35:38,831 make_protocol(ws, ws websocket: 127.0.0.1:10000 <- 127.0.0.1:37130)
2019-01-14 22:35:38,831 send_head() script(/index.html)=None
2019-01-14 22:35:38,831 translate_path(/index.html)=/usr/share/xpra/www/index.html
2019-01-14 22:35:38,831 io_thread_loop(read, <bound method Protocol._read of Protocol(ws websocket: 127.0.0.1:10000 <- 127.0.0.1:37130)>) loop starting
2019-01-14 22:35:38,831 Error: read on ws websocket: 127.0.0.1:10000 <- 127.0.0.1:37130 failed: <type 'exceptions.AttributeError'>
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/net/protocol.py", line 591, in _io_thread_loop
    while not self._closed and callback():
  File "/usr/lib64/python2.7/site-packages/xpra/net/protocol.py", line 664, in _read
    buf = self._conn.read(READ_BUFFER_SIZE)
  File "/usr/lib64/python2.7/site-packages/xpra/server/websocket.py", line 352, in read
    bufs, closed_string = self.ws_handler.recv_frames()
  File "/usr/lib/python2.7/site-packages/websockify-0.8.0-py2.7.egg/websockify/websockifyserver.py", line 183, in recv_frames
    buf = self.request.recvmsg()
AttributeError: '_socketobject' object has no attribute 'recvmsg'
2019-01-14 22:35:38,831 guess_type(/usr/share/xpra/www/index.html)=('text/html', None)
2019-01-14 22:35:38,832 Protocol.close() closed=False, connection=ws websocket: 127.0.0.1:10000 <- 127.0.0.1:37130
2019-01-14 22:35:38,832 accept-encoding=['identity']
2019-01-14 22:35:38,832 Protocol.close() calling <bound method WebSocketConnection.close of ws websocket: 127.0.0.1:10000 <- 127.0.0.1:37130>
2019-01-14 22:35:38,832 process default packet connection-lost
2019-01-14 22:35:38,834 netifaces loaded sucessfully
2019-01-14 22:35:38,834 process_connection_lost(Protocol(None), ['connection-lost'])
2019-01-14 22:35:38,834 "GET /index.html HTTP/1.1" 200 -
2019-01-14 22:35:38,835 ws websocket: 127.0.0.1:10000 <- 127.0.0.1:37130.close() for socket={'fileno': 43, 'family': 2, 'proto': 0, 'timeout': 5000, 'type': 1, 'options': {'IP': {'IP_TTL': 64, 'IP_TOS': 0, 'IP_OPTIONS': 0}, 'SOCKET': {'SO_TYPE': 1, 'SO_SNDBUF': 2626560, 'SO_ERROR': 0, 'SO_RCVLOWAT': 1, 'SO_KEEPALIVE': 0, 'SO_RCVTIMEO': 0, 'SO_BROADCAST': 0, 'SO_OOBINLINE': 0, 'SO_REUSEPORT': 0, 'SO_DONTROUTE': 0, 'SO_RCVBUF': 1062000, 'SO_LINGER': 0, 'SO_SNDTIMEO': 0, 'SO_REUSEADDR': 1}, 'TCP': {'TCP_NODELAY': 1, 'TCP_MAXSEG': 21888```
2019-01-14 22:35:38,835 ws websocket: 127.0.0.1:10000 <- 127.0.0.1:37130.close() done
2019-01-14 22:35:38,835 terminate_queue_threads()
2019-01-14 22:35:38,835 cleanup_protocol(Protocol(None))
2019-01-14 22:35:38,836 may_reload_headers() http headers time={}, mtime=1540297043.29
2019-01-14 22:35:38,836 Protocol.close() done
2019-01-14 22:35:38,836 may_reload_headers() loading from '/usr/share/xpra/http-headers/00_nocache.txt'
2019-01-14 22:35:38,836 may_reload_headers() loading from '/usr/share/xpra/http-headers/10_content_security_policy.txt'
2019-01-14 22:35:38,836 may_reload_headers() headers={'Expires': '0', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache, no-store, must-revalidate'}, mtime=1540297043.29
}}}

totaam · 2019-01-17T15:55:30Z

2019-01-17 15:55:30: antoine uploaded file `websockify-latest.patch` (2.2 KiB)

some changes required to run against the latest websockify

totaam · 2019-01-22T12:03:32Z

Couldn't figure out what was going wrong, so bisected websockify instead:

Remove additional signal calls in websockify that causes novnc to hang: OK (0.7.0)
Split out basic WebSocket server template (had to create the missing include/base64.js file to be able to install it..): hangs
Disable Nagle for proxied connections: OK
Close connection after Websocket handshake: OK
Separate out raw WebSocket protocol handling: HTTP connections work, but websockify errors out on AttributeError: '_socketobject' object has no attribute 'recvmsg'
Next commit is:
Split out basic WebSocket server template - as per above, everything hangs with that one..

totaam · 2019-01-22T14:19:05Z

So the first breakage comes from the changes to websocket.py diff-653cb1e48180f6007df323a009690197.
This commit is huge, and it's not clear at all why calling socket.recv on the underlying tcp socket returns nothing... (even in blocking mode)

Meanwhile, this may have uncovered a bug in python-websocket-client:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 967, in connect_or_fail
    return connect_to(display_desc, opts)
  File "/usr/lib64/python2.7/site-packages/xpra/scripts/main.py", line 1098, in connect_to
    return websocket_client_connection(host, port, conn, dtype)
  File "/usr/lib64/python2.7/site-packages/xpra/net/websocket_connection.py", line 79, in websocket_client_connection
    ws = websocket.create_connection(url, SOCKET_TIMEOUT, subprotocols=subprotocols, socket=sock)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 502, in create_connection
    websock.connect(url, **options)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 222, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/usr/lib/python2.7/site-packages/websocket/_handshake.py", line 70, in handshake
    success, subproto = _validate(resp, key, options.get("subprotocols"))
  File "/usr/lib/python2.7/site-packages/websocket/_handshake.py", line 160, in _validate
    subproto = headers.get("sec-websocket-protocol", None).lower()
AttributeError: 'NoneType' object has no attribute 'lower'

Change requested: [https://github.com/websocket-client/websocket-client/compare/master...totaam:patch-1]

totaam · 2019-01-23T09:56:31Z

Found that there were typos in that changeset: self._sendmg should be self._sendmsg.

During testing, I also hit this one:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xpra/net/protocol.py", line 1042, in close
    c.close()
  File "/usr/lib64/python2.7/site-packages/xpra/server/websocket.py", line 342, in close
    SocketConnection.close(self)
  File "/usr/lib64/python2.7/site-packages/xpra/net/bytestreams.py", line 334, in close
    s.close()
  File "/usr/lib/python2.7/site-packages/websockify-0.8.0-py2.7.egg/websockify/websocket.py", line 489, in close
    self._close()
  File "/usr/lib/python2.7/site-packages/websockify-0.8.0-py2.7.egg/websockify/websocket.py", line 689, in _close
    self.socket.close()
AttributeError: 'NoneType' object has no attribute 'close'

r21460 adds compatibility up to and including Separate out raw WebSocket protocol handling.

r21461 better debug logging

totaam · 2019-01-23T17:29:52Z

Fixed in r21463.
Support added for websockify>0.8 in the performance test in r21465. (#1926)

Maybe we should propagate the EWOULDBLOCK write event. (and not just from websockets IO)

Will follow up in:

websocket-client vs websockify>0.8 error: Text frames are not supported #2120 websocket-client vs websockify>0.8 error: Text frames are not supported
get rid of websockify #2121 speedup websockify send / hybi

totaam closed this as completed Jan 23, 2019

totaam added the v2.4.x label Jan 22, 2021

This was referenced Jan 22, 2021

websocket-client vs websockify>0.8 error: Text frames are not supported #2120

Closed

get rid of websockify #2121

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support websockify > 0.8 #2104

support websockify > 0.8 #2104

totaam commented Jan 14, 2019 •

edited

Loading

totaam commented Jan 14, 2019

totaam commented Jan 14, 2019 •

edited

Loading

totaam commented Jan 17, 2019

totaam commented Jan 22, 2019 •

edited

Loading

totaam commented Jan 22, 2019 •

edited

Loading

totaam commented Jan 23, 2019 •

edited

Loading

totaam commented Jan 23, 2019 •

edited

Loading

support websockify > 0.8 #2104

support websockify > 0.8 #2104

Comments

totaam commented Jan 14, 2019 • edited Loading

totaam commented Jan 14, 2019

2019-01-14 06:23:58: antoine uploaded file websockify-new-import.patch (3.6 KiB)

totaam commented Jan 14, 2019 • edited Loading

totaam commented Jan 17, 2019

2019-01-17 15:55:30: antoine uploaded file websockify-latest.patch (2.2 KiB)

totaam commented Jan 22, 2019 • edited Loading

totaam commented Jan 22, 2019 • edited Loading

totaam commented Jan 23, 2019 • edited Loading

totaam commented Jan 23, 2019 • edited Loading

totaam commented Jan 14, 2019 •

edited

Loading

2019-01-14 06:23:58: antoine uploaded file `websockify-new-import.patch` (3.6 KiB)

totaam commented Jan 14, 2019 •

edited

Loading

2019-01-17 15:55:30: antoine uploaded file `websockify-latest.patch` (2.2 KiB)

totaam commented Jan 22, 2019 •

edited

Loading

totaam commented Jan 22, 2019 •

edited

Loading

totaam commented Jan 23, 2019 •

edited

Loading

totaam commented Jan 23, 2019 •

edited

Loading