A11yUITests

A11yTests is an extension to XCTestCase that adds tests for common accessibility issues that can be run as part of an XCUI Test suite.

Tests can either be run separately or integrated into existing XCUI Tests.

Using These Tests

Good accessibility is not about ticking boxes and conforming to regulations and guidelines, but about how your app is experienced. You will only ever know if your app is actually accessible by letting real people use it. Consider these tests as hints for where you might be able to do better, and use them to detect regressions.

Failures for these tests should be seen as warnings for further investigation, not strict failures. As such i'd recommend always having continueAfterFailure = true set.

The library has two types of tests - assertions and snapshots. Assertion tests check each individual element for potential accessibility failures. Snapshot tests creates a snapshot of your app's accessibility tree and stores this as a reference for future tests. If something changes in the accessibility tree in a future test, the test will fail signifying you should validate the change. No assertions are made against the accessibility tree, only a check for any changes since the last snapshot state.

Failures have two categories: Warning and Failure. Failures are fails against WCAG or the HIG. Warnings may be acceptable, but require investigation.

add import A11yUITests to the top of your test file.

Assertion tests

Assertion tests check each individual element for potential accessibility failures.

Running Tests

Tests can be run individually or in suites.

Running All Tests on All Elements

func test_allTests() {
    XCUIApplication().launch()
    a11yCheckAllOnScreen()
}

Specifying Tests/Elements

To specify elements and tests use a11y(tests: [A11yTests], on elements: [XCUIElement]) passing an array of tests to run and an array of elements to run them on. To run all interactive element tests on all buttons:

func test_buttons() {
    let buttons = XCUIApplication().buttons.allElementsBoundByIndex
    a11y(tests: a11yTestSuiteInteractive, on: buttons)
}

To run a single test on a single element pass arrays with the test and element. To check if a button has a valid accessibility label:

func test_individualTest_individualButton() {
    let button = XCUIApplication().buttons["My Button"]
    a11y(tests: [.buttonLabel], on: [button])
}

Ignoring Elements

When running a11yCheckAllOnScreen() it is possible to ignore elements using their accessibility idenfiers by passing any identifiers you wish to ignore with the ignoringElementIdentifiers: [String] argument.

Test Suites

A11yUITests contains 4 pre-built test suites with tests suitable for different elements.

a11yTestSuiteAll Runs all tests.

a11yTestSuiteImages Runs tests suitable for images.

a11yTestSuiteInteractive runs tests suitable for interactive elements.

a11yTestSuiteLabels runs tests suitable for static text elements.

Alternatively you can create an array of A11yTests enum values for the tests you want to run.

Tests

Minimum Size

minimumSize or checks an element is at least 14px x 14px. Severity: Warning

Note: 14px is arbitrary.

Minimum Interactive Size

minimumInteractiveSize checks tappable elements are a minimum of 44px x 44px. This satisfies WCAG 2.1 Success Criteria 2.5.5 Target Size Level AAA Severity: Error

Note: Many of Apple's controls fail this requirement. For this reason, when running a suite of tests with minimumInteractiveSize only buttons and cells are checked. This may still result in some failures for UITabBarButtons for example. For full compliance, you should run a11yCheckValidSizeFor(interactiveElement: XCUIElement) on any element that your user might interact with, eg. sliders, steppers, switches, segmented controls. But you will need to make your own subclass as Apple's are not strictly adherent to WCAG.

Label Presence

labelPresence checks the element has an accessibility label that is a minimum of 2 characters long. Pass a minMeaningfulLength argument to a11yCheckValidLabelFor(element: XCUIElement, minMeaningfulLength: Int ) to change the minimum length. This counts towards WCAG 2.1 Guideline 1.1 Text Alternatives but does not guarantee compliance. Severity: Warning

Button Label

buttonLabel checks labels for interactive elements begin with a capital letter and don't contain a period or the word button. Checks the label is a minimum of 2 characters long. Pass a minMeaningfulLength argument to a11yCheckValidLabelFor(interactiveElement: XCUIElement, minMeaningfulLength: Int ) to change the minimum length. This follows Apple's guidance for writing accessibility labels. Severity: Error

Note: This test is not localised.

Image Label

imageLabel checks accessible images don't contain the words image, picture, graphic, or icon, and checks that the label isn't reusing the image filename. Checks the label is a minimum of 2 characters long. Pass a minMeaningfulLength argument to a11yCheckValidLabelFor(image: XCUIElement, minMeaningfulLength: Int ) to change the minimum length. This follows Apple's guidelines for writing accessibility labels. Care should be given when deciding whether to make images accessible to avoid creating unnecessary noise. Severity: Error

Note: This test is not localised.

Label Length

labelLength checks accessibility labels are <= 40 characters. This follows Apple's guidelines for writing accessibility labels. Ideally, labels should be as short as possible while retaining meaning. If you feel your element needs more context consider adding an accessibility hint. Severity: Warning

Header

header checks the screen has at least one text element with a header trait. Headers are used by VoiceOver users to orientate and quickly navigate content. This follows WCAG 2.1 Success Criterion 2.4.10 Severity: Error

Button Trait

buttonTrait checks that a button element has the Button or Link trait applied. This follows Apple's guide for using traits. Severity: Error

Image Trait

imageTrait checks that an image element has the Image trait applied. This follows Apple's guide for using traits. Severity: Error

Conflicting Traits

conflictingTraits checks elements don't have conflicting traits. Elements can't be both a button and a link, or static text and updates frequently

Disabled Elements

disabled checks that elements aren't disabled. Disabled elements can be confusing if it is not clear why the element is disabled. Ideally keep the element enabled and clearly message if your app is not ready to process the action. Severity: Warning

Duplicated Labels

duplicated checks all elements provided for duplication of accessibility labels. Duplicated accessibility labels can make your screen confusing to navigate with VoiceOver, and make Voice Control fail. Ideally you should avoid duplication if possible. Severity: Warning

Snapshot tests

Snapshot creates a JSON representation of your screen's accessibility tree on the first run. On subsequent runs this initial snapshot is taken as a reference. The test fails if there are any differences between the reference snapshot and the current snapshot. No assertions are made that the accessibility tree is correct or valid, you must make these checks yourself and generate a known-good reference snapshot to protect against future regressions.

Running tests

In your UI test call a11ySnapshot(). On first run the test will fail because no snapshot has been created for this test. A reference snapshot is generated. Grab the reference snapshot from the URL provided in the failure message or find it attached in the test's XCResult. Add this file to your UITest target ensuring the filename matches the filename property of the generated json file. Subsequent test runs will be compared against this snapshot, if you wish to generate a new snapshot, remove the reference from your UITest target, run the test, and a new reference will be generated.

Tests

Snapshot testing checks for changes in the following:

• Accessibility label
• Frame
• Enabled status
• Control type
• Accessibility traits

Example

To run the example project, clone the repo, and run pod install from the Example directory first.

A11yUITests_ExampleUITests.swift contains example tests that show a fail for each test above.

Requirements

iOS 11

Swift 5

Installation

Swift Package Manager

This library support Swift Package Manager. Ensure the package is added as a dependancy to your UITests target, not your app's target.

Cocoapods

A11yUITests is available through CocoaPods. To install add the pod to your target's test target in your podfile. eg

target 'My_Application' do
    target 'My_Application_UITests' do
    pod 'A11yUITests'
    end
end

Note

• This library accesses a private property in the iOS SDK, so care should be taken when adding it to your project to ensure you are not shipping this code. If you submit this code to app review you will likely receive a rejection from Apple.
• This library uses method swizzling of the value(forUndefinedKey:) method on NSObject to guard against potential crashes if Apple changes their private API in future. Any calls to this function will return nil after running any tests. This affects your test suite only, not your app.

Known Issues

If two elements of the same type have the same identifier (eg, two buttons both labeled 'Next') this will cause the tests to crash on some iOS versions. This was an issue on ios 13 and appears fixed as of iOS 15.

Author

Rob Whitaker, [email protected]
https://mobilea11y.com

License

A11yUITests is available under the MIT license. See the LICENSE file for more info.