Websocket problems, wrong PONG response

[DennisRutjes]

Julia 0.6.2
HTTP.jl 0.6.9
MbedTLS.jl 0.5.8

Trying to connect to Binance API : wss://stream.binance.com:9443/ws/trxbtc@kline_15m/xvgbtc@kline_15m/ncashbtc@kline_15m/iostbtc@kline_15m/stormbtc@kline_15m/zilbtc@kline_15m/adabtc@kline_15m/poebtc@kline_15m/xrpbtc@kline_15m/xlmbtc@kline_15m/funbtc@kline_15m/tnbbtc@kline_15m

HTTP.WebSockets.open(url; verbose = true) do io
while !eof(io);
put!(channel, r2j(readavailable(io)))
end

resulted in an obscure error on binance part.

I used a working java client with the same URL and proxied the socket results via an own java websocket server and tried to connect via localhost "ws://localhost:8887" to this ws server.

then the same error occured, but now on the "java" side it said
org.java_websocket.exceptions.InvalidFrameException: Control frame cant have fin==false set

Is there a way to debug/test the Websocket implementation to address this behaviour

[EricForgy]

What was the obscure error?

[DennisRutjes]

wsRead="websocket: close 1008 (policy violation): {"code":-1,"msg":"Illegal format ws or stream"}"

I guess it has to do with the PING/PONG control frame.

HTTP.WebSockets sends back a pong control frame which is not expected

[DennisRutjes]

with the java websocket I can connect to binance url perfectly with no errors, with my proxy in between I got the control frame error

proxy is running fine : https://websocket.org/echo.html with ws://localhost:8887 works also, got data without disconnecting

[DennisRutjes]

using HTTP
function test() 
    url = "wss://stream.binance.com:9443/ws/trxbtc@kline_15m/xvgbtc@kline_15m/ncashbtc@kline_15m/iostbtc@kline_15m/stormbtc@kline_15m/zilbtc@kline_15m/adabtc@kline_15m/poebtc@kline_15m/xrpbtc@kline_15m/xlmbtc@kline_15m/funbtc@kline_15m/tnbbtc@kline_15m"

    HTTP.WebSockets.open(url; verbose = false) do io
        while !eof(io);
            println(String(readavailable(io)))
        end
    end
end

@schedule test()

this happens eventually after 5 min or less.

"e":"kline","E":1525971419786,"s":"NCASHBTC","k":{"t":1525970700000,"T":1525971599999,"s":"NCASHBTC","i":"15m","f":4206342,"L":4206501,"o":"0.00000449","c":"0.00000447","h":"0.00000450","l":"0.00000447","v":"1317445.00000000","n":160,"x":false,"q":"5.89483970","V":"374184.00000000","Q":"1.67710449","B":"0"}}
{"e":"kline","E":1525971420099,"s":"XVGBTC","k":{"t":1525970700000,"T":1525971599999,"s":"XVGBTC","i":"15m","f":15844904,"L":15845708,"o":"0.00000793","c":"0.00000792","h":"0.00000794","l":"0.00000790","v":"3018827.00000000","n":805,"x":false,"q":"23.89244454","V":"1076908.00000000","Q":"8.52795351","B":"0"}}
ERROR (unhandled task failure): HTTP.WebSockets.WebSocketError(1011, "{\"code\":-2,\"msg\":\"Internal server error.\"}")
Stacktrace:
 [1] readframe(::HTTP.WebSockets.WebSocket{MbedTLS.SSLContext}) at /Users/drutjes/.julia/v0.6/HTTP/src/WebSockets.jl:261
 [2] readavailable(::HTTP.WebSockets.WebSocket{MbedTLS.SSLContext}) at /Users/drutjes/.julia/v0.6/HTTP/src/WebSockets.jl:228
 [3] (::##1#2)(::HTTP.WebSockets.WebSocket{MbedTLS.SSLContext}) at ./none:4
 [4] (::HTTP.WebSockets.##3#4{Bool,##1#2,String})(::HTTP.Streams.Stream{HTTP.Messages.Response,HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}) at /Users/drutjes/.julia/v0.6/HTTP/src/WebSockets.jl:108
 [5] macro expansion at /Users/drutjes/.julia/v0.6/HTTP/src/StreamRequest.jl:61 [inlined]
 [6] macro expansion at ./task.jl:302 [inlined]
 [7] #request#1(::Void, ::HTTP.WebSockets.##3#4{Bool,##1#2,String}, ::Int64, ::Array{Any,1}, ::Function, ::Type{HTTP.StreamRequest.StreamLayer}, ::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, ::HTTP.Messages.Request, ::Void) at /Users/drutjes/.julia/v0.6/HTTP/src/StreamRequest.jl:53
 [8] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::Type{HTTP.StreamRequest.StreamLayer}, ::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, ::HTTP.Messages.Request, ::Void) at ./<missing>:0
 [9] #request#1(::Void, ::Type{T} where T, ::Array{Any,1}, ::Function, ::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Void) at /Users/drutjes/.julia/v0.6/HTTP/src/ConnectionRequest.jl:45
 [10] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Void)at ./<missing>:0
 [11] #request#1(::Array{Any,1}, ::Function, ::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}, ::HTTP.URIs.URI, ::Vararg{Any,N} where N) at /Users/drutjes/.julia/v0.6/HTTP/src/ExceptionRequest.jl:19
 [12] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Void) at ./<missing>:0
 [13] (::Base.###49#50#52{ExponentialBackOff,HTTP.RetryRequest.##2#3{Bool,HTTP.Messages.Request},HTTP.#request})(::Array{Any,1}, ::Function, ::Type{T} where T, ::Vararg{Any,N} where N) at ./error.jl:139
 [14] (::Base.#kw###49#51)(::Array{Any,1}, ::Base.##49#51, ::Type{T} where T, ::Vararg{Any,N} where N) at ./<missing>:0
 [15] #request#1(::Int64, ::Bool, ::Array{Any,1}, ::Function, ::Type{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Void) at /Users/drutjes/.julia/v0.6/HTTP/src/RetryRequest.jl:44
 [16] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::Type{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Void) at ./<missing>:0
 [17] #request#1(::VersionNumber, ::String, ::Void, ::Function, ::Array{Any,1}, ::Function, ::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Void) at /Users/drutjes/.julia/v0.6/HTTP/src/MessageRequest.jl:44
 [18] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Void) at ./<missing>:0
 [19] #request#1(::Int64, ::Bool, ::Array{Any,1}, ::Function, ::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Void) at /Users/drutjes/.julia/v0.6/HTTP/src/RedirectRequest.jl:24
 [20] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Void) at ./<missing>:0
 [21] #request#7(::Array{Any,1}, ::Function, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Void) at /Users/drutjes/.julia/v0.6/HTTP/src/HTTP.jl:289
 [22] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Void) at ./<missing>:0
 [23] #request#8(::Array{Pair{String,String},1}, ::Void, ::Void, ::Array{Any,1}, ::Function, ::String, ::String, ::Array{Pair{String,String},1}, ::Void) at /Users/drutjes/.julia/v0.6/HTTP/src/HTTP.jl:303
 [24] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::String, ::String, ::Array{Pair{String,String},1}, ::Void) at ./<missing>:0
 [25] #open#9(::Array{Any,1}, ::Function, ::Function, ::String, ::String, ::Array{Pair{String,String},1}) at /Users/drutjes/.julia/v0.6/HTTP/src/HTTP.jl:328
 [26] (::HTTP.#kw##open)(::Array{Any,1}, ::HTTP.#open, ::Function, ::String, ::String, ::Array{Pair{String,String},1}) at ./<missing>:0
 [27] #open#2(::Bool, ::Bool, ::Array{Any,1}, ::Function, ::##1#2, ::String) at /Users/drutjes/.julia/v0.6/HTTP/src/WebSockets.jl:88
 [28] (::HTTP.WebSockets.#kw##open)(::Array{Any,1}, ::HTTP.WebSockets.#open, ::Function, ::String) at ./<missing>:0
 [29] test() at ./none:3
 [30] (::##3#4)() at ./event.jl:73

[hgeorgako]

Were you able to resolve this Dennis?

[DennisRutjes]

nope , got the date via timed rest call less optimal but working;

Can you verify that this is also happening on your end?

I am on a macbook pro btw

[hgeorgako]

Not able to verify this at the moment. But I did want to work on something very similar. I still am not sure which version of websockets to use. The one from HTTP.jl or Websockets.jl? The use case is similar to yours. Connect to binance, stream some quotes into a Julia stuct and read the value of that struct anytime a change occurs.

[DennisRutjes]

I will give websockets.jl a try :-)
This is serverside only, DandelionWebsockets.jl looks like an option

[DennisRutjes]

helas, DandelionWebsockets forces to use outdated TLS packages, so still an issue

[DennisRutjes]

closing issue because 1011 code wit close event is send via binance

[DennisRutjes]

dug a little deeper, the reponse PONG message is incorrect so it seems, after three ping messages the HTTP.jl gives up? Can it be thate the reponse applicationdata is incorrect with regards to the ping data, see specification 👍
5.5.3. Pong

The Pong frame contains an opcode of 0xA.

Section 5.5.2 details requirements that apply to both Ping and Pong
frames.

A Pong frame sent in response to a Ping frame must have identical
"Application data" as found in the message body of the Ping frame
being replied to.

WebSocket ➡️ WebSocketHeader(TEXT | FINAL, 311-byte payload)
➡️ "{"e":"kline","E":1534787820788,"s":"TRXBTC","k":{"t":1534787100000,"T":1534787999999,"s":"TRXBTC","i":"15m","f":32783430,"L":32783713,"o":"0.00000336","c":"0.00000336","h":"0.00000337","l":"0.00000335","v":"5360925.00000000","n":284,"x":false,"q":"17.99483273","V":"3618983.00000000","Q":"12.15932703","B":"0"}}"
2018-08-20T19:57:00.923 :
WebSocket ➡️ WebSocketHeader(PING | FINAL, )
WebSocket ⬅️ WebSocketHeader(PONG | FINAL, 311-byte payload, mask = c0518157)
⬅️ UInt8[0x2c, 0xa3, 0x34, 0xe2, 0x6d, 0xa3, 0x3a, 0xac, 0x3e, 0xef, 0x34, 0xe2, 0x7b, 0xa3, 0x14, 0xe2, 0x6d, 0xb0, 0x64, 0xf3, 0x63, 0xb6, 0x69, 0xf7, 0x6f, 0xb3, 0x61, 0xf7, 0x6f, 0xb9, 0x7d, 0xe2, 0x24, 0xa3, 0x6b, 0xe2, 0x03, 0xd3, 0x09, 0x82, 0x03, 0xc2, 0x73, 0xec, 0x75, 0xea, 0x73, 0xfa, 0x2c, 0xa3, 0x25, 0xe2, 0x6d, 0xb0, 0x64, 0xf3, 0x63, 0xb6, 0x69, 0xf7, 0x66, 0xb1, 0x61, 0xf0, 0x67, 0xb1, 0x7d, 0xe2, 0x03, 0xa3, 0x6b, 0xf1, 0x62, 0xb2, 0x65, 0xf7, 0x6f, 0xb6, 0x68, 0xf9, 0x6e, 0xb8, 0x68, 0xf9, 0x7b, 0xa3, 0x22, 0xe2, 0x6d, 0xa3, 0x05, 0x92, 0x0f, 0xc3, 0x05, 0x83, 0x75, 0xad, 0x73, 0xa9, 0x75, 0xbb, 0x73, 0xf1, 0x62, 0xec, 0x73, 0xec, 0x75, 0xe7, 0x73, 0xfa, 0x64, 0xb3, 0x66, 0xf8, 0x64, 0xb5, 0x62, 0xf0, 0x7b, 0xa3, 0x1d, 0xe2, 0x6d, 0xb2, 0x63, 0xf7, 0x6f, 0xb2, 0x66, 0xf1, 0x64, 0xad, 0x73, 0xaf, 0x75, 0xbb, 0x73, 0xf0, 0x79, 0xb1, 0x61, 0xf0, 0x67, 0xb1, 0x62, 0xf3, 0x61, 0xa3, 0x7d, 0xe2, 0x34, 0xa3, 0x6b, 0xe2, 0x67, 0xaf, 0x61, 0xf0, 0x67, 0xb1, 0x61, 0xf3, 0x64, 0xb7, 0x73, 0xec, 0x75, 0xe9, 0x73, 0xfa, 0x75, 0xb1, 0x7f, 0xf0, 0x67, 0xb1, 0x61, 0xf0, 0x64, 0xb2, 0x66, 0xe2, 0x7b, 0xa3, 0x3d, 0xe2, 0x6d, 0xa3, 0x61, 0xee, 0x67, 0xb1, 0x61, 0xf0, 0x67, 0xb2, 0x62, 0xf5, 0x75, 0xad, 0x73, 0xb6, 0x75, 0xbb, 0x73, 0xf5, 0x64, 0xb7, 0x61, 0xf9, 0x65, 0xb4, 0x7f, 0xf0, 0x67, 0xb1, 0x61, 0xf0, 0x67, 0xb1, 0x61, 0xe2, 0x7b, 0xa3, 0x3f, 0xe2, 0x6d, 0xb3, 0x69, 0xf4, 0x7b, 0xa3, 0x29, 0xe2, 0x6d, 0xe7, 0x30, 0xac, 0x24, 0xe4, 0x7d, 0xe2, 0x26, 0xa3, 0x6b, 0xe2, 0x66, 0xb6, 0x7f, 0xf9, 0x6e, 0xb5, 0x69, 0xf3, 0x65, 0xb6, 0x62, 0xe2, 0x7b, 0xa3, 0x07, 0xe2, 0x6d, 0xa3, 0x62, 0xf6, 0x66, 0xb9, 0x68, 0xf8, 0x64, 0xaf, 0x61, 0xf0, 0x67, 0xb1, 0x61, 0xf0, 0x67, 0xb1, 0x73, 0xec, 0x75, 0xd0, 0x73, 0xfa, 0x75, 0xb0, 0x63, 0xee, 0x66, 0xb4, 0x68, 0xf3, 0x65, 0xb6, 0x61, 0xf3, 0x75, 0xad, 0x73, 0x82, 0x75, 0xbb, 0x73, 0xf0, 0x75, 0xfc, 0x2c]
base64 binary data: V2ViU29ja2V0IOKeoe+4jyAgV2ViU29ja2V0SGVhZGVyKENMT1NFIHwgRklOQUwsIDQ0LWJ5dGUgcGF5bG9hZCkKICAgICAgICAgIOKeoe+4jyAgIgPzeyJjb2RlIjotMiwibXNnIjoiSW50ZXJuYWwgc2VydmVyIGVycm9yLiJ9IgpXZWJTb2NrZXQg4qyF77iPICBXZWJTb2NrZXRIZWFkZXIoQ0xPU0UgfCBGSU5BTCwgKQo=

base64data decoded 👍
WebSocket ➡️ WebSocketHeader(CLOSE | FINAL, 44-byte payload)
➡️ "�{"code":-2,"msg":"Internal server error."}"
WebSocket ⬅️ WebSocketHeader(CLOSE | FINAL, )

[DennisRutjes]

Bump

[quinnj]

I think the solution is to remove the WebSockets code from HTTP.jl. There have been failing tests for several months now and no one who was originally involved in adding support for that code has been able to respond. The WebSockets.jl package is fully functioning and better maintained at this point than the HTTP.WebSockets code, so I think we should delete it unless someone can step up and make it work. cc: @samoconnor @EricForgy

[samoconnor]

Hi @DennisRutjes, can you try out 5914a0e and let me know if that helps?

[DennisRutjes]

@samoconnor that fixed it!
The PING/PONG responses are correct:
2018-09-02T21:24:01.194 :
WebSocket ➡️ WebSocketHeader(PING | FINAL, )
WebSocket ⬅️ WebSocketHeader(PONG | FINAL, , mask = 12a2d605)
⬅️ UInt8[]
WebSocket ➡️ WebSocketHeader(TEXT | FINAL, 294-byte payload)

I am getting steady results

Thanks for addressing this issue, to me it was a real pain!

[christnp]

I know this is closed, but I just want to throw out my opinion. The error message is:
{"code":-1,"msg":"Illegal format ws or stream"}"
And the format of your ws is:
url = "wss://stream.binance.com:9443/ws/trxbtc@kline_15m/xvgbtc@kline_15m/ncashbtc@kline_15m/iostbtc@kline_15m/stormbtc@kline_15m/zilbtc@kline_15m/adabtc@kline_15m/poebtc@kline_15m/xrpbtc@kline_15m/xlmbtc@kline_15m/funbtc@kline_15m/tnbbtc@kline_15m"

Per the Binance API, for multiple streams the format instead needs to be:
wss://stream.binance.com:9443/stream?streams=trxbtc@kline_15m/...

I've received similar error response from Binance and it has always been a typo in my websocket URI.