Add asynchronous downloading to speed up signed URL downloading.

[jonathanmetzman]

The speed ups are up to 20x on my own machine.
It's even faster with one core than 16 parallel processes.

[jonathanmetzman]

TODO: Do this for other http methods.

[vitorguidi]

The speed ups are up to 20x on my own machine. It's even faster with one core than 16 parallel processes.

Do we have any idea of how long the downloading step takes from past profiling data, on fuzz task? (back when those profiling images were around)

[vitorguidi]

Are these dep bumps supported in our platforms?

There should be complaints from

clusterfuzz/src/local/butler/common.py

Line 295 in 044ec08

print(f'Did not find package for platform: {pip_platform}')

during butler deploy, if not.

[vitorguidi]

Might also need to rebuild docker, since httpaio is a new dep with C extensions to access async syscalls

[jonathanmetzman]

I'm pretty sure we're not using the C extensions.

[jonathanmetzman]

But I'll know what to look out for if I see a problem.

[jonathanmetzman]

The speed ups are up to 20x on my own machine. It's even faster with one core than 16 parallel processes.

Do we have any idea of how long the downloading step takes from past profiling data, on fuzz task? (back when those profiling images were around)

We got rid of profiling before we switched to this new method of downloading, but seeing as the new method can take up to 15 minutes, I think the savings will be substantial.

[jonathanmetzman]

Please take a another look.