Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test reliability #3891

Merged
merged 16 commits into from
May 14, 2021
Merged

Improve test reliability #3891

merged 16 commits into from
May 14, 2021

Conversation

compulim
Copy link
Contributor

@compulim compulim commented May 12, 2021

Changelog Entry

Fixed

  • Improved test reliability and adding snapshot to console of test harness in development mode, by @compulim, in PR #3891

Description

Now, we can show the image snapshot in console:

Console showing snapshot

Design

Specific Changes

  • Group multiple sendTab and sendShiftTab into a single call
  • pageObjects.scrollToTop
    • Before scrolling, wait for scroll to stabilize
      • If we scroll while the scroll is animating, our scroll may not land in precise position
    • After scrolling, wait for scroll to stabilize at the right scroll position
  • After rendering Web Chat with a static transcript, wait for all activities to show up
  • Add unit tests for marshal.js and unmarshal.js
  • Fix an issue in RPC that it is not marshalling when resolve/reject
  • Test harness running in development mode will take snapshot and draw it in the console log
  • Taking a snapshot will wait until the page is stabilized:
    • Wait until all images are completed (either loaded or errored)
    • Wait until 2 consecutive screenshots look exactly the same
  • Added a new option on sendMessageViaSendBox, waitForNumResponse, to wait for X number of responses from bot
    • pageObjects.sendMessageViaSendBox('Hello, World!', { waitForNumResponse: 1 }) will wait for 1 response from bot
  • Bump to [email protected] for a race condition fix
  • I have added tests and executed them locally
  • I have updated CHANGELOG.md
  • I have updated documentation

Review Checklist

This section is for contributors to review your work.

  • Accessibility reviewed (tab order, content readability, alt text, color contrast)
  • Browser and platform compatibilities reviewed
  • CSS styles reviewed (minimal rules, no z-index)
  • Documents reviewed (docs, samples, live demo)
  • Internationalization reviewed (strings, unit formatting)
  • package.json and package-lock.json reviewed
  • Security reviewed (no data URIs, check for nonce leak)
  • Tests reviewed (coverage, legitimacy)

@compulim compulim marked this pull request as ready for review May 12, 2021 23:01
@corinagum
Copy link
Contributor

FYI, I made some edits to your PR overview. LMK what you think :)

image

CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Contributor

@corinagum corinagum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, pending a few comments (and questions, haha)

@compulim
Copy link
Contributor Author

Saw this, thanks!

image

@compulim compulim merged commit 035ec1f into main May 14, 2021
@compulim compulim deleted the fix-test-reliability branch May 14, 2021 01:55
compulim added a commit that referenced this pull request Jun 22, 2021
* Improve test reliability

* Send tabs at once

* Wait for all activities

* scrollToTop should wait before and after scroll

* Remove console.log

* Stabilize screenshot before image snapshot

* Add entry

* Fix test break

* Fix tests

* Improve test reliability

* Fix expectation

* Improve test reliability

* Improve page object model

* Bump react-scroll-to-bottom

* Bump [email protected]

* Apply suggestions from code review

Co-authored-by: Corina <[email protected]>

Co-authored-by: Corina <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants