EOFError Heisenbug

[timholy]

Julia 1.x
HTTP.jl 0.8.x
MbedTLS.jl 0.6.x and 1.0.x

I'm sure this is my own fault, but I'm a newbie when it comes to HTTP requests and I'm experiencing a strange Heisenbug when running tests in my not-yet-registered package https://github.com/JuliaHealth/PubChemCrawler.jl. The issue is that whenever I run the tests with HTTP.request(args...; verbose=3) they pass, but when I run without the verbose setting they fail. EDIT: the failure only happens in CI, locally they always pass.

Examples:

• with verbose: https://github.com/JuliaHealth/PubChemCrawler.jl/actions/runs/358347994 (the failure on nightly is Change to typesubtract signature causes failure with Julia 1.6-dev JuliaData/CSV.jl#757)
• without verbose: https://github.com/JuliaHealth/PubChemCrawler.jl/actions/runs/358322402

The offending line is here (it's systematic), which calls https://github.com/JuliaHealth/PubChemCrawler.jl/blob/6bf9a3226132d0eff5381ce9be196d808af243fd/src/query.jl#L127. I've toggled between verbose=3 and not twice now, and gotten the same result, so I think this is not just bad luck.

It may also be worth pointing out that this service has request limits, see https://juliahealth.org/PubChemCrawler.jl/previews/PR3/#Before-you-start:-be-aware-of-search-limits.

Anyone have any ideas?

[quinnj]

So, I cloned the repo and ran the tests several times, but didnt' see any errors; is that expected?

[timholy]

Locally, for me it works every time, it just fails on CI. I wonder if it's because so many jobs are running in parallel and that ends up triggering PubChem's limits? But I've tried to add fairly generous delays, without benefit (so far, I could try something like 15s I suppose). The weirdest part, though, is the fact that they work on CI with verbose=3 but not without.

[timholy]

I worked around the issue (just so I could register the package) by checking for ENV["CI"] and use verbose=3 when found. But it's a weird phenomenon, so if anyone does figure this out I'd love to know the explanation.

[quinnj]

We've overhauled the request internals and connection pool logic, and there's a proposal to overhaul the debug logging architecture and what verbose controls, so I'm going to close this for now with the idea that the forthcoming 1.0 release should reasonably fix this.