[WebCryptoAPI] Stability job unable to run tests to completion

[jugglinmike]

The stability of the tests in the WebCryptoAPI/derive_bits_keys/ directory cannot be verified on Taskcluster because they require too much time to complete. This makes it difficult for contributors to submit modifications to these tests via the WPT repository directly.

@foolip reported this in gh-19904:

wpt-firefox-nightly-stability is failing on Taskcluster. There were only 4 affected tests and yet we got "Task timeout after 7200 seconds. Force killing container."

These are the actual tests that were run, after expanding multi-globals and variants:

/WebCryptoAPI/derive_bits_keys/ecdh_bits.https.any.html
/WebCryptoAPI/derive_bits_keys/ecdh_bits.https.any.worker.html
/WebCryptoAPI/derive_bits_keys/ecdh_keys.https.any.html
/WebCryptoAPI/derive_bits_keys/ecdh_keys.https.any.worker.html
/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?1-1000
/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?1001-2000
/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?2001-3000
/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?3001-last
/WebCryptoAPI/derive_bits_keys/hkdf.https.any.worker.html?1-1000
/WebCryptoAPI/derive_bits_keys/hkdf.https.any.worker.html?1001-2000
/WebCryptoAPI/derive_bits_keys/hkdf.https.any.worker.html?2001-3000
/WebCryptoAPI/derive_bits_keys/hkdf.https.any.worker.html?3001-last
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?1-1000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?1001-2000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?2001-3000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?3001-4000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?4001-5000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?5001-6000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?6001-7000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?7001-8000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?8001-last
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?1-1000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?1001-2000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?2001-3000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?3001-4000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?4001-5000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?5001-6000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?6001-7000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?7001-8000
/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.worker.html?8001-last

The live_backing.log.txt is a huge 975MB file with ~9.5 million lines...

cc @jimsch

[tniessen]

I am seeing this problem in #21141 and #21140.

[foolip]

This is a specific case of the more general problem in #7660. #16175 is another case that we often hit.

The root cause is simple: we try to run too many tests too many times and the CI system times out. We can't fix this by increasing the timeout because it would take way too long in the extreme case of thousands of files being touched or affected.

We'd have to make the stability jobs aware of the timeout internally to avoid running more iterations than there's time for.