Broken Sync / Timeout Behavior on slow connections

[pik]

On slow connections it seems that room-state cannot be effectively loaded (for a logged-in user, guest works fine).

Guest

Loading as guest seems to work:

https://matrix.org/_matrix/client/r0/sync?filter=%7B%7D&timeout=30000&since=s72314221_107161785_509881_9264024_74962_32558_235595&access_token=<access_token>

Logged-In User

On Login Sync request times-out:

Console Logging

SyncApi.stop
sync.js:49 Sync no longer running: exiting
index.js:121 newscreen login
Lifecycle.js:238 setLoggedIn => @pik:matrix.org (guest=false) hs=https://matrix.org
Lifecycle.js:260 Session persisted for @pik:matrix.org
sync.js:49 SyncApi.sync: starting with sync token null
index.js:121 newscreen 
client.js:2789 Got TURN URIs: turn:turn.matrix.org:3478?transport=udp,turn:turn.matrix.org:3478?transport=tcp,turns:turn-ssl.matrix.org:443?transport=tcp refresh in 86400 secs
sync.js:49 Got push rules
Presence.js:83 Presence: online
sync.js:550 /sync error MatrixError
{"errcode":"ORG.MATRIX.JSSDK_TIMEOUT","name":"ORG.MATRIX.JSSDK_TIMEOUT","message":"Locally timed out waiting for a response","data":{"error":"Locally timed out waiting for a response","errcode":"ORG.MATRIX.JSSDK_TIMEOUT","timeout":110000}}

Request

https://matrix.org/_matrix/client/r0/sync?filter=22&timeout=30000&_cacheBuster=1482684475006&access_token=<access_token>

Response Headers

HTTP/1.1 200 OK
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Sun, 25 Dec 2016 17:00:56 GMT
Server: Synapse/0.18.5 (b=hotfixes-v0.18.5,2998fa5)
Transfer-Encoding: chunked
Content-Encoding: gzip

Speculatively, this might have to do with chunk-size of the SYNC query? And maybe after time-out it should be reduced - also maybe just loading most recent room-state and than parsing back to get things like historical responses / etc. would help?

I'll be able to fill in more details after adding some artificial timeouts and trying this on localhost to narrow down the behavior...

[pik]

Is this duplicate of #2737 ? If so feel free to close.

[pik]

Here out of interest is the number of entries / byte size of a Riot Client-Sync (for the uncompressed JSON response):

Sync Stats

{'account_data': 3,
 'account_data_size': '2.9 KiB',
 'presence_events': 379,
 'presence_events_size': '48.7 KiB',
 'rooms_invite_size': '2.0 B',
 'rooms_join_size': '5.9 MiB',
 'rooms_leave_size': '2.0 B',
 'rooms_size': '5.9 MiB',
 'to_device': 0,
 'to_device_size': '2.0 B'}

Sync Stats per Room

{ '!DgvjtOljKujDBrxyHk:matrix.org': { 'account_data_size': '14.0 B',
                                      'ephemeral_events': 2,
                                      'ephemeral_size': '143.2 KiB',
                                      'state_events': 1756,
                                      'state_size': '613.5 KiB',
                                      'timeline_events': 20,
                                      'timeline_size': '6.3 KiB',
                                      'unread_notifications_size': '47.0 B'},
  '!XqBunHwQIXUiqCaoxq:matrix.org': { 'account_data_size': '14.0 B',
                                      'ephemeral_events': 2,
                                      'ephemeral_size': '72.9 KiB',
                                      'state_events': 1352,
                                      'state_size': '468.4 KiB',
                                      'timeline_events': 20,
                                      'timeline_size': '5.9 KiB',
                                      'unread_notifications_size': '47.0 B'},
  '!cURbafjkfsMDVwdRDQ:matrix.org': { 'account_data_size': '125.0 B',
                                      'ephemeral_events': 2,
                                      'ephemeral_size': '388.4 KiB',
                                      'state_events': 7943,
                                      'state_size': '2.7 MiB',
                                      'timeline_events': 20,
                                      'timeline_size': '6.8 KiB',
                                      'unread_notifications_size': '47.0 B'},
  '!gUtFRMMjnvcvTLCQwU:matrix.org': { 'account_data_size': '125.0 B',
                                      'ephemeral_events': 1,
                                      'ephemeral_size': '141.0 B',
                                      'state_events': 7,
                                      'state_size': '2.5 KiB',
                                      'timeline_events': 20,
                                      'timeline_size': '6.1 KiB',
                                      'unread_notifications_size': '47.0 B'},
  '!kxIpnPaYREeRkPOtuc:matrix.org': { 'account_data_size': '14.0 B',
                                      'ephemeral_events': 0,
                                      'ephemeral_size': '14.0 B',
                                      'state_events': 0,
                                      'state_size': '14.0 B',
                                      'timeline_events': 8,
                                      'timeline_size': '2.3 KiB',
                                      'unread_notifications_size': '2.0 B'},
  '!lfNdkoorQdjhxTjfZB:matrix.org': { 'account_data_size': '14.0 B',
                                      'ephemeral_events': 0,
                                      'ephemeral_size': '14.0 B',
                                      'state_events': 2,
                                      'state_size': '491.0 B',
                                      'timeline_events': 10,
                                      'timeline_size': '3.2 KiB',
                                      'unread_notifications_size': '2.0 B'},
  '!mKfbfylfmNRrjRGxCT:matrix.org': { 'account_data_size': '14.0 B',
                                      'ephemeral_events': 0,
                                      'ephemeral_size': '14.0 B',
                                      'state_events': 0,
                                      'state_size': '14.0 B',
                                      'timeline_events': 8,
                                      'timeline_size': '2.3 KiB',
                                      'unread_notifications_size': '2.0 B'},
  '!tItmtevQmRsOUnRHpN:matrix.org': { 'account_data_size': '14.0 B',
                                      'ephemeral_events': 0,
                                      'ephemeral_size': '14.0 B',
                                      'state_events': 0,
                                      'state_size': '14.0 B',
                                      'timeline_events': 6,
                                      'timeline_size': '1.8 KiB',
                                      'unread_notifications_size': '2.0 B'},
  '!xwuKXrGMSmrmvfauht:matrix.org': { 'account_data_size': '14.0 B',
                                      'ephemeral_events': 0,
                                      'ephemeral_size': '14.0 B',
                                      'state_events': 0,
                                      'state_size': '14.0 B',
                                      'timeline_events': 6,
                                      'timeline_size': '1.8 KiB',
                                      'unread_notifications_size': '2.0 B'},
  '!yhqiEdqNjyPbxtUjzm:matrix.org': { 'account_data_size': '14.0 B',
                                      'ephemeral_events': 2,
                                      'ephemeral_size': '89.8 KiB',
                                      'state_events': 4397,
                                      'state_size': '1.4 MiB',
                                      'timeline_events': 20,
                                      'timeline_size': '7.1 KiB',
                                      'unread_notifications_size': '47.0 B'}}

[pik]

And the associated request (1.37mb compressed):

curl 'https://matrix.org/_matrix/client/r0/sync?filter=25&timeout=0&_cacheBuster=1483482315381&access_token=<access_token>' -H 'Origin: http://127.0.0.1:8080' -H 'Accept-Encoding: gzip, deflate, sdch, br' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36' -H 'accept: application/json' -H 'Referer: http://127.0.0.1:8080/' -H 'Connection: keep-alive' -o /dev/null --dump-header - --compressed 

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:34 --:--:--     0HTTP/1.1 200 OK
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Thu, 05 Jan 2017 16:46:34 GMT
Server: Synapse/0.18.6-rc3 (b=matrix-org-hotfixes,345afd9)
Transfer-Encoding: chunked
Content-Encoding: gzip

100 1378k    0 1378k    0     0  37332      0 --:--:--  0:00:37 --:--:--  296k

[richvdh]

dup #2737