test: Add visual diff test to designkit #6707
Conversation
This adds a visual diff test to designkit. We generate screenshots of each section of the designkit on the branch in question using playwright and then again on the main branch. We then use playwright's built in image comparison facilities to generate a diff between the screenshots. If differences are found, both screenshots and the diff are output to a flat html file that displays the results and CI will report in github.
✅ Deploy Preview for determined-ui ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
There was a problem hiding this comment.
If there are no changes from the previous version this looks great to me. I liked it when you PRed it before. I didn't do a deep dive into the code but I can if you want me to. I could also ask somebody in devops if you want someone to look over your circleCI stuff.
There was a problem hiding this comment.
thanks for working. I have some questions
- Don't we wanna test it on different browsers as well as Chrome?
- Is this test available on local too?
- Why not using Playwright visual testing directly? Playwright offers rich image diff reports.
example:
Screen.Recording.2023-05-04.at.10.24.36.AM.mov
if these are already considered, i think its good to go.
|
We're not using playwright's image comparison stuff directly because we don't want this test to block. Playwright's workflow would require the user to notice the test failing and then generate new screenshots. While we could fake this by running a screenshot test on main to generate the expected image and then again on the branch to generate the diffs, we'd still need a way to define which components to look at. Doing it statically introduces the possibility of the components list drifting out of sync, and doing it dynamically means that we either don't get screenshots of added components or removed components. |
|
Noticed the latest version doesn't just display components with a diff. |
Description
This adds a visual diff test to designkit. We generate screenshots of each section of the designkit on the branch in question using playwright and then again on the main branch. We then use playwright's built in image comparison facilities to generate a diff between the screenshots. If differences are found, both screenshots and the diff are output to a flat html file that displays the results and CI will report in github.
Test Plan
CI Only -- no testing required
Commentary (optional)
we're using
undicifor our node fetch needs because it's the source of the experimental fetch library that landed in node 18, so when we upgrade, we should be able to just remove the import and library and everything will be the same.Checklist
docs/release-notes/.See Release Note for details.
Ticket