Testing with Selenium & Sauce labs #3321
Conversation
Will this help on Travis?
Idea from https://stackoverflow.com/questions/48236104/ssl-errors-using- sauce-labs-in-travis-ci-with-selenium-webriver-tests-django-pr
|
Updating to the latest Firefox re-reveals an issue I was seeing locally but hoped was interference with my main Firefox instance: it randomly seems to go back to the authentication page at times, as if it had lost the authentication cookie. Any ideas? This seemed to work when it was testing with an older version of Firefox on Linux, so we can push it back to an older version as a workaround if we can't find a better solution. Or maybe a new Firefox release will fix it. |
|
I've just realised that the tunnel to Sauce doesn't work for pull requests, to avoid exposing the secret Sauce token. I'll change this to run a headless browser on Travis. We could set up a Travis cron job to test with a range of different browsers on a regular basis without affecting pull requests. |
|
This is really cool! Could you say a bit more about some of the issues that you’re running into & how you’re debugging them? |
|
Thanks! It largely feels like trial and error, but to break down some specific issues:
|
|
This is epic, @takluyver!! Thank you! |
| visited.add(url) | ||
| browser.get(url) | ||
| wait_for_selector(browser, '.item_link') | ||
| assert browser.current_url == url |
There was a problem hiding this comment.
I've tried to translate the JS test I removed directly to Python to illustrate how the machinery works, but I don't think it's actually a very meaningful test. The next step will be to write some new tests using this; I'm planning to leave that for other PRs, because I think there's already enough to look at in this PR.
|
Thank you so much for going through this @takluyver, I'll help review. |
| @@ -40,6 +39,15 @@ before_install: | |||
| if [[ $GROUP == docs ]]; then | |||
| pip install -r docs/doc-requirements.txt | |||
| fi | |||
| - | | |||
| if [[ $GROUP == selenium ]]; then | |||
| pip install selenium | |||
There was a problem hiding this comment.
That makes it nice n' easy, this is great.
|
Is there someone else you'd like to look at this? Or shall we push the green button? |
|
Pushed! Thanks a lot for taking the initiative on this, @takluyver. Now we should be able to upgrade to React 16 and add the vdom and possibly JSON renderers to notebook 👍 |
|
Just to be clear, this puts the foundations in place for using Selenium, but we'll still need to convert the rest of the JS tests before the way is clear to using React. |
|
Understood. Replace "now" with "after we rewrite all the tests as Selenium tests" above 😋 |
|
@takluyver given how fresh this is I thought you might benefit from seeing how I set up protractor to run selenium tests for a notebook server based app: https://github.com/SimonBiggs/scriptedforms/tree/master/scriptedforms/tests-e2e Within the app I have a 'spinner' which defines when things are busy. Within the testing scripts I just wait for that 'spinner' to be gone to signify that the kernel has finished and the page has updated with the results from the kernel. It has worked quite well thus far. Granted, my lil ol package is nothing compared to the mammoth of the Jupyter Notebook project, but I thought you might like to know how my experience was. Cheers, |
|
To make it work do the following: Within yarn
yarn buildThen within pip install -e .Then within yarn
yarn run install
yarn run seleniumLeave that previous terminal running and then within yarn run e2e |
|
An example test is the following: |
This replaces the
js/treesection of the tests with Selenium tests run withpy.test, and sets up Travis to run it using browsers on Sauce Labs, which has a free open source plan. The tests can also be run locally with a suitable webdriver installed (geckodriver for Firefox, chromedriver for Chrome).My thinking is that this can form the basis for replacing more of the JS tests with Selenium tests. I think this is a nicer way of writing tests, it's more realistic because it drives a real browser rather than Qt webkit, and hopefully it won't suffer so much from random failures caused by async issues.
Sauce has some nice features - you can browse through screenshots at the different steps, to help diagnose problems:
This probably won't work on the jupyter repo at the moment, but it's working on my fork. We'll need to create a Jupyter account on Saucelabs, and update the credentials in the file.