premature client disconnects

[bloodearnest]

We're seeing frequent client disconnects.

The dokku2 nginx logs show premature close, e.g.

2021/09/28 13:19:27 [error] 29908#29908: *1727134 upstream prematurely closed connection while reading upstream, client: 62.253.26.158

It looks like we are seeing this with larger uploads, but not always.

I noticed that uploads were being buffered, so I turned that off in proxy nginx config, but it doesn't seem to have worked.

It may be something todo with our double nginx thing. It might be worth redesigning the proxy as just nginx config that's dropped in to dokku's nginx, rather than running a separate nginx as a dokku app

It seems elevated starting this week (27/09/2021), and there were some network changes on the 26th, so have email TPP

[sebbacon]

Is it worth investing in a script to attempt to reproduce the problem? We could provide it to TPP to run, as well, then.

[bloodearnest]

So, it turns out I didn't turn off buffering properly, so I did that: https://github.com/opensafely-core/proxy/pull/19/files

I tested a manual push via the proxy, at file sizes up to 50mb (GH limit), and encounter no issues.

I successfully released the previously problematic 7.3mb html file, so this may be fixed, we'll see if users report any more issues.