diff --git a/docs/src/api/class-locator.md b/docs/src/api/class-locator.md index e6654235502dd..f4a2fe6d7c7e3 100644 --- a/docs/src/api/class-locator.md +++ b/docs/src/api/class-locator.md @@ -155,7 +155,7 @@ Additional locator to match. - returns: <[string]> Captures the aria snapshot of the given element. -Read more about [accessibility snapshots](../aria-snapshot.md) and [`method: LocatorAssertions.toMatchAriaSnapshot`] for the corresponding assertion. +Read more about [aria snapshots](../aria-snapshots.md) and [`method: LocatorAssertions.toMatchAriaSnapshot`] for the corresponding assertion. **Usage** diff --git a/docs/src/api/class-locatorassertions.md b/docs/src/api/class-locatorassertions.md index f858e299a2428..5d06824b474d4 100644 --- a/docs/src/api/class-locatorassertions.md +++ b/docs/src/api/class-locatorassertions.md @@ -2109,7 +2109,7 @@ Expected options currently selected. * langs: - alias-java: matchesAriaSnapshot -Asserts that the target element matches the given [accessibility snapshot](../aria-snapshot.md). +Asserts that the target element matches the given [accessibility snapshot](../aria-snapshots.md). **Usage** diff --git a/docs/src/aria-snapshot.md b/docs/src/aria-snapshots.md similarity index 68% rename from docs/src/aria-snapshot.md rename to docs/src/aria-snapshots.md index 51f4155d26be0..d781907c0a3e3 100644 --- a/docs/src/aria-snapshot.md +++ b/docs/src/aria-snapshots.md @@ -1,19 +1,19 @@ --- -id: aria-snapshot -title: "Accessibility Snapshots" +id: aria-snapshots +title: "Aria snapshots" --- ## Overview -In Playwright, accessibility snapshots provide a YAML representation of the accessible elements on a page. +In Playwright, aria snapshots provide a YAML representation of the accessibility tree of a page. These snapshots can be stored and compared later to verify if the page structure remains consistent or meets defined expectations. -The YAML format describes the hierarchical structure of accessible elements on the page, detailing **roles**, -**attributes**, **values**, and **text content**. The structure follows a tree-like syntax, where each node represents -an accessible element, and indentation indicates nested elements. +The YAML format describes the hierarchical structure of accessible elements on the page, detailing **roles**, **attributes**, **values**, and **text content**. +The structure follows a tree-like syntax, where each node represents an accessible element, and indentation indicates +nested elements. -Following is a simple example of an accessibility snapshot for the playwright.dev homepage: +Following is a simple example of an aria snapshot for the playwright.dev homepage: ```yaml - banner: @@ -32,7 +32,7 @@ Each accessible element in the tree is represented as a YAML node: ``` - **role**: Specifies the ARIA or HTML role of the element (e.g., `heading`, `list`, `listitem`, `button`). -- **"name"**: Accessible name of the element. Quoted strings indicate exact values, or regular expression. +- **"name"**: Accessible name of the element. Quoted strings indicate exact values, `/patterns/` are used for regular expression. - **[attribute=value]**: Attributes and values, in square brackets, represent specific ARIA attributes, such as `checked`, `disabled`, `expanded`, `level`, `pressed`, or `selected`. @@ -40,11 +40,11 @@ These values are derived from ARIA attributes or calculated based on HTML semant structure of a page, use the [Chrome DevTools Accessibility Pane](https://developer.chrome.com/docs/devtools/accessibility/reference#pane). -## Snapshot Matching +## Snapshot matching The [`method: LocatorAssertions.toMatchAriaSnapshot`] assertion method in Playwright compares the accessible -structure of a page with a predefined accessibility snapshot template, helping validate the page's accessibility -state against testing requirements. +structure of the locator scope with a predefined aria snapshot template, helping validate the page's state against +testing requirements. For the following DOM: @@ -93,7 +93,7 @@ When matching, the snapshot template is compared to the current accessibility tr page's accessibility tree. -### Partial Matching +### Partial matching You can perform partial matches on nodes by omitting attributes or accessible names, enabling verification of specific parts of the accessibility tree without requiring exact matches. This flexibility is helpful for dynamic or irrelevant @@ -103,9 +103,9 @@ attributes. ``` +*aria snapshot* ```yaml -# accessibility snapshot - button ``` @@ -119,10 +119,9 @@ focusing solely on role and hierarchy. ```html - ``` -*accessibility tree for partial match* +*aria snapshot for partial match* ```yaml - checkbox @@ -142,17 +141,17 @@ Similarly, you can partially match children in lists or groups by omitting speci ``` -*accessibility tree for partial match* +*aria snapshot for partial match* ```yaml - list - listitem: Feature B ``` -Partial matches let you create flexible accessibility tests that verify essential page structure without enforcing +Partial matches let you create flexible snapshot tests that verify essential page structure without enforcing specific content or attributes. -### Matching with Regular Expressions +### Matching with regular expressions Regular expressions allow flexible matching for elements with dynamic or variable text. Accessible names and text can support regex patterns. @@ -161,19 +160,64 @@ support regex patterns.

Issues 12

``` -*accessibility tree with regular expression* +*aria snapshot with regular expression* ```yaml - heading /Issues \d+/ ``` +## Generating snapshots -## Generating Snapshots - -Creating accessibility snapshots in Playwright helps ensure and maintain your application’s structure. +Creating aria snapshots in Playwright helps ensure and maintain your application’s structure. You can generate snapshots in various ways depending on your testing setup and workflow. -### 1. Using the `Locator.ariaSnapshot` Method +### 1. Generating snapshots with the Playwright code generator + +If you’re using Playwright’s [Code Generator](./codegen.md), generating aria snapshots is streamlined with its +interactive interface: + +- **"Assert snapshot" Action**: In the code generator, you can use the "Assert snapshot" action to automatically create +a snapshot assertion for the selected elements. This is a quick way to capture the aria snapshot as part of your +recorded test flow. + +- **"Aria snapshot" Tab**: The "Aria snapshot" tab within the code generator interface visually represents the +aria snapshot for a selected locator, letting you explore, inspect, and verify element roles, attributes, and +accessible names to aid snapshot creation and review. + +### 2. Updating snapshots with `@playwright/test` and the `--update-snapshots` flag + +When using the Playwright test runner (`@playwright/test`), you can automatically update snapshots by running tests with +the `--update-snapshots` flag: + +```bash +npx playwright test --update-snapshots +``` + +This command regenerates snapshots for assertions, including aria snapshots, replacing outdated ones. It’s +useful when application structure changes require new snapshots as a baseline. Note that Playwright will wait for the +maximum expect timeout specified in the test runner configuration to ensure the +page is settled before taking the snapshot. It might be necessary to adjust the `--timeout` if the test hits the timeout +while generating snapshots. + +#### Empty template for snapshot generation + +Passing an empty string as the template in an assertion generates a snapshot on-the-fly: + +```js +await expect(locator).toMatchAriaSnapshot(''); +``` + +Note that Playwright will wait for the maximum expect timeout specified in the test runner configuration to ensure the +page is settled before taking the snapshot. It might be necessary to adjust the `--timeout` if the test hits the timeout +while generating snapshots. + +#### Snapshot patch files + +When updating snapshots, Playwright creates patch files that capture differences. These patch files can be reviewed, +applied, and committed to source control, allowing teams to track structural changes over time and ensure updates are +consistent with application requirements. + +### 3. Using the `Locator.ariaSnapshot` method The [`method: Locator.ariaSnapshot`] method allows you to programmatically create a YAML representation of accessible elements within a locator’s scope, especially helpful for generating snapshots dynamically during test execution. @@ -205,59 +249,12 @@ var snapshot = await page.Locator("body").AriaSnapshotAsync(); Console.WriteLine(snapshot); ``` -This command outputs the accessibility tree within the specified locator’s scope in YAML format, which you can validate +This command outputs the aria snapshot within the specified locator’s scope in YAML format, which you can validate or store as needed. -### 2. Generating Snapshots with the Playwright Code Generator - -If you’re using Playwright’s [Code Generator](./codegen.md), generating accessibility snapshots is streamlined with its -interactive interface: - -- **"Assert Snapshot" Action**: In the code generator, you can select elements and use the "Assert snapshot" action to -automatically create a snapshot assertion for those elements. This is a quick way to capture the accessibility structure -as part of your recorded test flow. - -- **"Accessibility" Tab**: The "Accessibility" tab within the code generator interface visually represents the -accessibility tree for a selected locator, letting you explore, inspect, and verify element roles, attributes, and -accessible names to aid snapshot creation and review. - -### 3. Updating Snapshots with `@playwright/test` and the `--update-snapshots` Flag -* langs: js - -When using the Playwright test runner (`@playwright/test`), you can automatically update snapshots by running tests with -the `--update-snapshots` flag: - -```bash -npx playwright test --update-snapshots -``` - -This command regenerates snapshots for assertions, including accessibility snapshots, replacing outdated ones. It’s -useful when application structure changes require new snapshots as a baseline. Note that Playwright will wait for the -maximum timeout specified in the test runner configuration to ensure the page is fully loaded before taking the -snapshot. It might be necessary to adjust the `--timeout` if the test hits the timeout -while generating snapshots. - -#### Empty Template for Snapshot Generation - -Passing an empty string as the template in an assertion generates a snapshot on-the-fly: - -```js -await expect(locator).toMatchAriaSnapshot(''); -``` - -Note that Playwright will wait for the maximum timeout specified in the test runner configuration to ensure the page is -fully loaded before taking the snapshot. It might be necessary to adjust the `--timeout` if the test hits the timeout -while generating snapshots. - -#### Snapshot Patch Files - -When updating snapshots, Playwright creates patch files that capture differences. These patch files can be reviewed, -approved, and committed to source control, allowing teams to track structural changes over time and ensure updates are -consistent with application requirements. - -## Accessibility Tree Examples +## Accessibility tree examples -### Headings with Level Attributes +### Headings with level attributes Headings can include a `level` attribute indicating their heading level. @@ -266,14 +263,14 @@ Headings can include a `level` attribute indicating their heading level.

Subtitle

``` -*accessibility tree* +*aria snapshot* ```yaml - heading "Title" [level=1] - heading "Subtitle" [level=2] ``` -### Text Nodes +### Text nodes Standalone or descriptive text elements appear as text nodes. @@ -281,21 +278,21 @@ Standalone or descriptive text elements appear as text nodes.
Sample accessible name
``` -*accessibility tree* +*aria snapshot* ```yaml - text: Sample accessible name ``` -### Inline Multiline Text +### Inline multiline text -Multiline text, such as paragraphs, is normalized in the accessibility tree. +Multiline text, such as paragraphs, is normalized in the aria snapshot. ```html

Line 1
Line 2

``` -*accessibility tree* +*aria snapshot* ```yaml - paragraph: Line 1 Line 2 @@ -309,7 +306,7 @@ Links display their text or composed content from pseudo-elements. Read more about Accessibility ``` -*accessibility tree* +*aria snapshot* ```yaml - link "Read more about Accessibility" @@ -323,13 +320,13 @@ Input elements of type `text` show their `value` attribute content. ``` -*accessibility tree* +*aria snapshot* ```yaml - textbox: Enter your name ``` -### Lists with Items +### Lists with items Ordered and unordered lists include their list items. @@ -340,7 +337,7 @@ Ordered and unordered lists include their list items. ``` -*accessibility tree* +*aria snapshot* ```yaml - list "Main Features": @@ -348,7 +345,7 @@ Ordered and unordered lists include their list items. - listitem: Feature 2 ``` -### Grouped Elements +### Grouped elements Groups capture nested elements, such as `
` elements with summary content. @@ -359,36 +356,36 @@ Groups capture nested elements, such as `
` elements with summary conten
``` -*accessibility tree* +*aria snapshot* ```yaml - group: Summary ``` -### Attributes and States +### Attributes and states Commonly used ARIA attributes, like `checked`, `disabled`, `expanded`, `level`, `pressed`, and `selected`, represent control states. -#### Checkbox with `checked` Attribute +#### Checkbox with `checked` attribute ```html ``` -*accessibility tree* +*aria snapshot* ```yaml -- checkbox [checked=true] +- checkbox [checked] ``` -#### Button with `pressed` Attribute +#### Button with `pressed` attribute ```html ``` -*accessibility tree* +*aria snapshot* ```yaml - button "Toggle" [pressed=true] diff --git a/packages/playwright-core/src/server/frames.ts b/packages/playwright-core/src/server/frames.ts index 129484a02702c..11aba2b1b208f 100644 --- a/packages/playwright-core/src/server/frames.ts +++ b/packages/playwright-core/src/server/frames.ts @@ -1828,5 +1828,7 @@ function renderUnexpectedValue(expression: string, received: any): string { return received ? 'empty' : 'not empty'; if (expression === 'to.be.focused') return received ? 'focused' : 'not focused'; + if (expression === 'to.match.aria') + return received ? received.raw : received; return received; } diff --git a/packages/playwright-core/types/types.d.ts b/packages/playwright-core/types/types.d.ts index 4ebfafa37d536..89ff7e84c9ce0 100644 --- a/packages/playwright-core/types/types.d.ts +++ b/packages/playwright-core/types/types.d.ts @@ -12425,7 +12425,7 @@ export interface Locator { and(locator: Locator): Locator; /** - * Captures the aria snapshot of the given element. Read more about [accessibility snapshots](https://playwright.dev/docs/aria-snapshot) and + * Captures the aria snapshot of the given element. Read more about [aria snapshots](https://playwright.dev/docs/aria-snapshots) and * [expect(locator).toMatchAriaSnapshot(expected[, options])](https://playwright.dev/docs/api/class-locatorassertions#locator-assertions-to-match-aria-snapshot) * for the corresponding assertion. * diff --git a/packages/playwright/src/runner/rebase.ts b/packages/playwright/src/runner/rebase.ts index 7558ea08000cc..7863f4d124516 100644 --- a/packages/playwright/src/runner/rebase.ts +++ b/packages/playwright/src/runner/rebase.ts @@ -23,6 +23,7 @@ import { generateUnifiedDiff } from 'playwright-core/lib/utils'; import { colors } from 'playwright-core/lib/utilsBundle'; import type { FullConfigInternal } from '../common/config'; import { filterProjects } from './projectUtils'; +import type { InternalReporter } from '../reporters/internalReporter'; const t: typeof T = types; type Location = { @@ -43,7 +44,7 @@ export function addSuggestedRebaseline(location: Location, suggestedRebaseline: suggestedRebaselines.set(location.file, { location, code: suggestedRebaseline }); } -export async function applySuggestedRebaselines(config: FullConfigInternal) { +export async function applySuggestedRebaselines(config: FullConfigInternal, reporter: InternalReporter) { if (config.config.updateSnapshots !== 'all' && config.config.updateSnapshots !== 'missing') return; if (!suggestedRebaselines.size) @@ -102,6 +103,5 @@ export async function applySuggestedRebaselines(config: FullConfigInternal) { await fs.promises.writeFile(patchFile, patches.join('\n')); const fileList = files.map(file => ' ' + colors.dim(file)).join('\n'); - // eslint-disable-next-line no-console - console.log(`New baselines created for:\n\n${fileList}\n\n ` + colors.cyan('git apply ' + path.relative(process.cwd(), patchFile)) + '\n'); + reporter.onStdErr(`\nNew baselines created for:\n\n${fileList}\n\n ` + colors.cyan('git apply ' + path.relative(process.cwd(), patchFile)) + '\n'); } diff --git a/packages/playwright/src/runner/runner.ts b/packages/playwright/src/runner/runner.ts index 966fb13e92292..94b4095e6cc5e 100644 --- a/packages/playwright/src/runner/runner.ts +++ b/packages/playwright/src/runner/runner.ts @@ -89,7 +89,7 @@ export class Runner { ]; const status = await runTasks(new TestRun(config, reporter), tasks, config.config.globalTimeout); - await applySuggestedRebaselines(config); + await applySuggestedRebaselines(config, reporter); // Calling process.exit() might truncate large stdout/stderr output. // See https://github.com/nodejs/node/issues/6456. diff --git a/packages/playwright/types/test.d.ts b/packages/playwright/types/test.d.ts index f17fba3502ac5..305ae67caff4c 100644 --- a/packages/playwright/types/test.d.ts +++ b/packages/playwright/types/test.d.ts @@ -8414,7 +8414,7 @@ interface LocatorAssertions { }): Promise; /** - * Asserts that the target element matches the given [accessibility snapshot](https://playwright.dev/docs/aria-snapshot). + * Asserts that the target element matches the given [accessibility snapshot](https://playwright.dev/docs/aria-snapshots). * * **Usage** * diff --git a/packages/recorder/src/recorder.tsx b/packages/recorder/src/recorder.tsx index 2e05b65f6e59d..2b606843635c3 100644 --- a/packages/recorder/src/recorder.tsx +++ b/packages/recorder/src/recorder.tsx @@ -137,6 +137,9 @@ export const Recorder: React.FC = ({ { window.dispatch({ event: 'setMode', params: { mode: mode === 'assertingValue' ? 'recording' : 'assertingValue' } }); }}> + { + window.dispatch({ event: 'setMode', params: { mode: mode === 'assertingSnapshot' ? 'recording' : 'assertingSnapshot' } }); + }}> { copy(source.text); @@ -179,7 +182,7 @@ export const Recorder: React.FC = ({ }, { id: 'aria', - title: 'Accessibility', + title: 'Aria snapshot', render: () => }, ]} diff --git a/packages/web/src/components/codeMirrorWrapper.css b/packages/web/src/components/codeMirrorWrapper.css index 6988ab6506061..b9293eea214ff 100644 --- a/packages/web/src/components/codeMirrorWrapper.css +++ b/packages/web/src/components/codeMirrorWrapper.css @@ -33,7 +33,8 @@ color: var(--vscode-debugTokenExpression-number); } -.CodeMirror span.cm-keyword { +.CodeMirror span.cm-keyword, +.CodeMirror span.cm-builtin { color: var(--vscode-debugTokenExpression-name); } diff --git a/tests/page/to-match-aria-snapshot.spec.ts b/tests/page/to-match-aria-snapshot.spec.ts index 7f3da42b9c0b0..5adc7e7da5f78 100644 --- a/tests/page/to-match-aria-snapshot.spec.ts +++ b/tests/page/to-match-aria-snapshot.spec.ts @@ -555,3 +555,12 @@ heading invalid `); } }); + +test('call log should contain actual snapshot', async ({ page }) => { + await page.setContent(`

todos

`); + const error = await expect(page.locator('body')).toMatchAriaSnapshot(` + - heading "wrong" + `, { timeout: 3000 }).catch(e => e); + + expect(stripAnsi(error.message)).toContain(`- unexpected value "- heading "todos" [level=1]"`); +}); diff --git a/tests/playwright-test/update-aria-snapshot.spec.ts b/tests/playwright-test/update-aria-snapshot.spec.ts index c9c9b0056e6b1..1e20bc1b3d4d9 100644 --- a/tests/playwright-test/update-aria-snapshot.spec.ts +++ b/tests/playwright-test/update-aria-snapshot.spec.ts @@ -15,7 +15,7 @@ */ import * as fs from 'fs'; -import { test, expect, playwrightCtConfigText } from './playwright-test-fixtures'; +import { test, expect, playwrightCtConfigText, stripAnsi } from './playwright-test-fixtures'; import { execSync } from 'child_process'; test.describe.configure({ mode: 'parallel' }); @@ -47,6 +47,13 @@ test('should update snapshot with the update-snapshots flag', async ({ runInline \`); }); +`); + + expect(stripAnsi(result.output).replace(/\\/g, '/')).toContain(`New baselines created for: + + a.spec.ts + + git apply test-results/rebaselines.patch `); execSync(`patch -p1 < ${patchPath}`, { cwd: testInfo.outputPath() }); @@ -66,6 +73,14 @@ test('should update missing snapshots', async ({ runInlineTest }, testInfo) => { }); expect(result.exitCode).toBe(0); + + expect(stripAnsi(result.output).replace(/\\/g, '/')).toContain(`New baselines created for: + + a.spec.ts + + git apply test-results/rebaselines.patch +`); + const patchPath = testInfo.outputPath('test-results/rebaselines.patch'); const data = fs.readFileSync(patchPath, 'utf-8'); expect(data).toBe(`--- a/a.spec.ts @@ -262,6 +277,15 @@ test('should update multiple files', async ({ runInlineTest }, testInfo) => { }); expect(result.exitCode).toBe(0); + + expect(stripAnsi(result.output).replace(/\\/g, '/')).toContain(`New baselines created for: + + src/button-1.test.tsx + src/button-2.test.tsx + + git apply test-results/rebaselines.patch +`); + const patchPath = testInfo.outputPath('test-results/rebaselines.patch'); const data = fs.readFileSync(patchPath, 'utf-8'); expect(data).toBe(`--- a/src/button-1.test.tsx