CheerioCrawler's gzip decompression fails on certain pages

[jancurn]

For example, when you try to crawl https://www.ebay.com/sch/sis.html?_nkw=Beechcraft it consistently fails with:

ERROR: BasicCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue {"url":"https://www.ebay.com/sch/i.html?_nkw=dodge","retryCount":1}
  Error: incorrect header check
    at Gunzip.zlibOnError (zlib.js:153:15)

When you try to open the same page in Firefox and set Accept-Encoding: gzip, the page loads fine.

[mnmkng]

Still not fixed, but at least it led me to a bug in http-request. We were swallowing these errors and requests were timing out, instead of throwing.

The trouble is that the incorrect header check pops in the middle of the response body. I tried all zlib decompress functions and none worked.

It could be fixed by retrying the request with accept-encoding: identity, but we would lose the browser headers in requestAsBrowser.

[mnmkng]

nodejs/node#7502 Not sure if it's even possible to do it anymore. But let's see.

[pocesar]

you don't need to go far, it happens with Node and the log API endpoint, I was having this exact problem when providing a manually crafted Range header. it works in the browser, but fails on Node... tried a lot of request libraries as well. unless you forge a zlib header then try again, you still get corrupt data after getting past the internal incorrect header check

[B4nan]

cc @szmarczak, looks like its the same with got, maybe you have some ideas?

incorrect header check
RequestError: incorrect header check
    at PassThrough.<anonymous> (/Users/adamek/htdocs/apify/apify-js/node_modules/got-cjs/dist/source/core/index.js:614:31)
    at Object.onceWrapper (node:events:476:26)
    at PassThrough.emit (node:events:381:22)
    at emitErrorNT (node:internal/streams/destroy:188:8)
    at emitErrorCloseNT (node:internal/streams/destroy:153:3)
    at processTicksAndRejections (node:internal/process/task_queues:81:21)
    at Zlib.zlibOnError [as onerror] (node:zlib:190:17)

[szmarczak]

The website improperly compressed the data, however it's still possible to decode this. This is against the spec so won't be fixed in Got. The solution is to get the buffer / stream and decode it with DeflateRaw instead.

[B4nan]

Closing, either its somehow fixed in crawlee or one of the dependencies, or that URL no longer handles compression wrongly.

Works fine with cheerio (got-scraping) as well as playwright in crawlee.

[szmarczak]

Yes, it's been fixed: https://github.com/apify/got-scraping/blob/8d9d4f1e6f0144cfd7993c958fcc71e6c61c00ff/src/hooks/fix-decompress.ts#L49