Move functional tests to web container; get them working w/client-side playback

[rebeccacremona]

This overdue PR moves the Perma functional/end-to-end tests from a standalone playwright container into the main web container, and tweaks the tests and test setup so that it works with client-side playback.

If the playwright tests are run from a separate container, as with our current setup, then they have to be run against an independently spun up web server and database; the usual Django test server and test database are not available. That's not great: each test run was creating persistent artifacts in our local database; we didn't have access nice pytest fixtures, etc. By moving everything into a single suite, we ensure we have access to all our standard conveniences.

The setup is complicated by Perma's need to be served over SSL: between cross-origin iframes, services workers, cross-origin fetch requests... client-side replay simply does not work without a real SSL setup with trusted certs. Flags to disable the browser's SSL cert verification (i.e., ignore_https_errors) turn out to be insufficient. So, this PR does two things:

• adds a library with an SSL-enabled version of Django's LiveTestServer
• adds the certs generated by mkcert during the Perma local installation process as trusted certs to the web container

I'm adding the certs, including the rootCA cert, manually using certutil because I could not get it working using mkcert -install: even when I could see that that installed the rootCA cert installed in the nssdb, the other certs themselves were not being trusted; if I add just the certs without the rootCA, the browser complains about an untrusted CA.

I'm using the same certs as development rather than installing mkcert on the web container and using it to generate testing-specific certs because of minio: the minio container is already running with the dev certificate, and from within the test suite, we can't change that without significant shenanigans. (And if the minio cert is untrusted, then the fetch of the warc will fail.)

This is currently only working with Chromium: Firefox, also installed into the container, is still objecting to the SSL cert situation. All I can figure out is... it's not looking at the NSS database in $HOME/.pki/nssdb; it's looking somewhere else. I've Googled around and seen a couple of other people complaining about custom certs + playwright + Docker, but not found anything definitive... including one comment on a Github issue that I can no longer find, stating that their browser was looking in /tmp/.pki/nssdb.... I'd like to get this working with Firefox too, but don't want it to block any longer.

I also set things up so that, when functional tests fail, a full-screen screenshot will be put in the failed_test_files directory, like we used to have. I don't think we'll be able to inspect them with our current CI setup, but they can be helpful locally to see what's going on. That directory is automatically cleared out by Playwright on every test run.

Closes #3095 and #3086

[codecov]

Codecov Report

Merging #3117 (1d0af5f) into develop (b0a0092) will increase coverage by 0.29%.
The diff coverage is 78.94%.

@@             Coverage Diff             @@
##           develop    #3117      +/-   ##
===========================================
+ Coverage    82.32%   82.61%   +0.29%     
===========================================
  Files           53       53              
  Lines         5737     5793      +56     
===========================================
+ Hits          4723     4786      +63     
+ Misses        1014     1007       -7     

Impacted Files	Coverage Δ
perma_web/conftest.py	82.60% <78.94%> (-17.40%)	⬇️
perma_web/perma/models.py	88.15% <0.00%> (-3.22%)	⬇️
perma_web/perma/utils.py	70.04% <0.00%> (-1.00%)	⬇️
perma_web/api/serializers.py	93.26% <0.00%> (+0.48%)	⬆️
perma_web/perma/tasks.py	70.39% <0.00%> (+0.63%)	⬆️
perma_web/api/views.py	86.46% <0.00%> (+0.75%)	⬆️
perma_web/perma/views/common.py	73.59% <0.00%> (+1.98%)	⬆️
perma_web/perma/middleware.py	97.50% <0.00%> (+2.50%)	⬆️
perma_web/replay/urls.py	69.23% <0.00%> (+69.23%)	⬆️
perma_web/replay/views.py	91.42% <0.00%> (+91.42%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[rebeccacremona]

Getting closer...

Next battle: getting the playwright-driven browsers to trust the SSL certs, because the flags you can set to ignore untrusted SSL certs, while they work enough to get the page to load, do not work enough to get us into the secure context we need for all the playback tech to work:

Will setup makecert in the container, and will sub in those certs for the ones used by the test server; might contribute that feature upstream; might abandon the package; might switch to a proxy w/caddy as suggested by Jack. But, worth pointing out that we won't be able to just reuse the same certs for test runs as we do in development: these certs have to be made by a CA trusted by the container; dev certs need to be trusted by your laptop.

[rebeccacremona]

Looks like custom certs are supported; so maybe this will be quick. Though I probably shouldn't say that out loud 😛

[rebeccacremona]

Just realized this setup is super annoying: if you want to run the tests locally, with this as-is, you would have to re-run make_cert.sh every time you have a fresh web container (e.g., after any docker compose down or after any docker compose up from a new image/new build). That's a total pain.

I'll switch things so that the necessary cert moves are made when initializing the test suite instead.

UPDATE: done.

[bensteinberg]

I love this; I would love to add Firefox. Stay tuned, but let's land this now.