Core: Batch the loading of CSF files for extract() etc

[tmeasday]

Issue: We've seen situations where extract() overwhelms browsers due to loading too many requests at once

(only replicated on lambda-based browsers running old version of Chrome)

What I did

Batch loadAllCsfFiles() so it's no opening too many resources on the browser at once. The browser will only open so many requests to any given host anyway.

How to test

Run extract. This should happen as part of the chromatic-sandboxes task.

[shilman]

LGTM. What's an appropriate batch size though? Isn't 100 pretty high?

[shilman]

LGTM!

[ndelangen]

@tmeasday There seems to be some problem with this code according to chromatic:

Do you have an error in your `preview.js`? Check your Storybook's browser console for errors.
    at PreviewWeb.extract (https://635781f3500dd2c49e189caf-pxsfegimbt.capture.chromatic.com/sb-preview/runtime.mjs:84:4851)
    at __puppeteer_evaluation_script__:86:33
    → Failed to publish build

[tmeasday]

Hmm, so that seems to be a weird flake, like the SB wasn't initialized properly when extract() was run. It's very odd given I was touching code inside extract, but following the code path, my changes wouldn't have influenced that error (i.e. the error comes earlier in the code path than loadAllCsfFiles().

Still it seems pretty suspicious. I ran the CI job again and it worked. I'll run it a couple more times to be sure...

[tmeasday]

Ok, reran it about half a dozen times without error. If anyone sees the above error again, please let me know.