206 - Partial Content received for slow connections or large files, most browsers fail to load

[mtraynham]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Accessing webpack content on a slow connection or against a large file (like a vendor.js file) can result in a 206 - Partial Content. Within Chrome (v. 60.0.3112.101) this results in a net::ERR_CONTENT_LENGTH_MISMATCH, as the content-length header doesn't match the downloaded file, failing the download completely and the page will fail to load.

Locally (using localhost, 127.0.0.1 or 0.0.0.0) it seems to fail much slower, at almost exactly 3 minutes 58 seconds (I'm using wget to rate limit the download). Externally to the machine, large files will fail anywhere from 5 - 20 seconds. The latter here is the problem, accessing the dev-server externally with larger bundles will almost certainly fail to load.

If the current behavior is a bug, please provide the steps to reproduce.
With any webpack-dev-server running, download a somewhat large file using Webpack.

Local

> wget 127.0.0.1:3000/vendor.js --limit-rate=20k

--2017-08-15 13:44:56--  http://127.0.0.1:3000/vendor.js
Connecting to 127.0.0.1:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5900535 (5.6M) [application/javascript]
Saving to: ‘vendor.js’  ---  4.64M  20.2KB/s    in 3m 58s

2017-08-15 13:48:54 (20.0 KB/s) - Connection closed at byte 4867381. Retrying.

--2017-08-15 13:48:55--  (try: 2)  http://127.0.0.1:3000/vendor.js
Connecting to 127.0.0.1:3000... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 5900535 (5.6M), 1033154 (1009K) remaining [application/javascript]
Saving to: ‘vendor.js’

External (obfuscated IP)

wget 8.8.8.8:3000/vendor.js --limit-rate=20k
--2017-08-15 17:29:48--  http://8.8.8.8:3000/vendor.js
Connecting to 8.8.8.8:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5900535 (5.6M) [application/javascript]
Saving to: “vendor.js” --- 374,074     20.0K/s   in 18s     

2017-08-15 17:30:06 (20.0 KB/s) - Connection closed at byte 374074. Retrying.

--2017-08-15 17:30:07--  (try: 2)  http://8.8.8.8:3000/vendor.js
Connecting to 8.8.8.8:3000... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 5900535 (5.6M), 5526461 (5.3M) remaining [application/javascript]
Saving to: “vendor.js” --- 775,430     20.2K/s   in 19s     

2017-08-15 17:30:27 (20.1 KB/s) - Connection closed at byte 775430. Retrying.

--2017-08-15 17:30:29--  (try: 3)  http://8.8.8.8:3000/vendor.js
Connecting to 8.8.8.8:3000... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 5900535 (5.6M), 5125105 (4.9M) remaining [application/javascript]
Saving to: “vendor.js” --- 1,176,786   20.2K/s   in 19s     

2017-08-15 17:30:48 (20.1 KB/s) - Connection closed at byte 1176786. Retrying.

--2017-08-15 17:30:51--  (try: 4)  http://8.8.8.8:3000/vendor.js
Connecting to 8.8.8.8:3000... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 5900535 (5.6M), 4723749 (4.5M) remaining [application/javascript]
Saving to: “vendor.js” --- 1,540,100   20.0K/s   in 18s     

2017-08-15 17:31:09 (20.0 KB/s) - Connection closed at byte 1540100. Retrying.

--2017-08-15 17:31:13--  (try: 5)  http://8.8.8.8:3000/vendor.js
Connecting to 8.8.8.8:3000... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 5900535 (5.6M), 4360435 (4.2M) remaining [application/javascript]
Saving to: “vendor.js” --- 1,903,414   20.0K/s   in 18s

Related

angular/angular-cli#7197

Some things I tried with the Server code

• Setting a huge timeout

    app.use(function(req, res, next){
        res.setTimeout(4800000, function(){
            console.log('Request has timed out.');
            res.send(408);
        });
        next();
    });

• Removing the Promise from Webpack-Dev-Middleware (didn't realize Express allowed promises)

What is the expected behavior?
It ships the full content.

Please mention your webpack and Operating System version.
Webpack: 3.5
Webpack-Dev-Server: 2.7
Linux Mint 18.3, Mac OSX Sierra

[shellscape]

Definitely an edge case, but one that's sprinkled around the web not with webpack-dev-server, but with express. It seems there's an open issue for this as well.

This may be with how we're using express with regard to large files, but this is probably going to be a nuanced solution specific to your environment.

[mtraynham]

@shellscape The info is much appreciated. I was wondering what you suspect be the best plan here?

Open a bug against express to figure out what a workaround would be? The bug you referenced seems to suspect an incorrect usage of response.end, whereas weback-dev-middleware seems to always call response.send, which is like response.json. dougwilson seems to believe response.json can handle large objects.

Or rather, avoid using express and switch to koa-webpack? Does koa suffer from the same issue?

Or avoid *-webpack-middleware altogether for external usage and only use it for localhost development?

Edit So I tried a pretty simple test circumventing webpack-dev-server and just using Express standalone to ship the large file using response.send. It did fail (as the dev-server fails), so this is likely not a problem with webpack-dev-server.

Alternatively, I did try using a createReadStream(file).pipe(response) and that does work. The content-length header would be unspecified, but it does ship the file in full.

So then I tried changing webpack-dev-middleware to:

				// server content
				// var content = context.fs.readFileSync(filename);
				// content = shared.handleRangeHeaders(content, req, res);
				res.setHeader("Content-Type", mime.lookup(filename) + "; charset=UTF-8");
				// res.setHeader("Content-Length", content.length);
				if(context.options.headers) {
					for(var name in context.options.headers) {
						res.setHeader(name, context.options.headers[name]);
					}
				}
				// Express automatically sets the statusCode to 200, but not all servers do (Koa).
				res.statusCode = res.statusCode || 200;
                context.fs.createReadStream(filename).pipe(res)
				// if(res.send) res.send(content);
				// else res.end(content);
				resolve();

And that seems to work properly. For some reason, if you (un-comment) and append the Content-Length header,

res.setHeader("Content-Length", content.length);

even with streaming, it reintroduces the 206 - Partial Content bug.

[mtraynham]

Welp, I think I'm going to close this, as the bug is not within Express/WebpackDevServer. This is a NodeJS bug since Node 8.0.0, per expressjs/express#3392.

[shellscape]

@mtraynham thanks for your diligence.