Add Documentation how to login different users with parallel execution

[knoobie]

Migrating from Junit4 based Testbench to the Junit5 based Testbench, I noticied that all tests are now executed in parallel - resulting in multiple errors with tests that using the same browser to log-in different users and therefore deleting the old session of the user.

Things I tested so far, but all failed in some way:

• TestBench: Parameters.setTestsInParallel(1);
• Selenium: ChromeOptions:.addArguments("--incognito")
• Junit 5: @Execution(ExecutionMode.SAME_THREAD)
• Junit 5: @TestInstance(Lifecycle.PER_CLASS)
• Spring: @DirtiesContext(classMode = ClassMode.BEFORE_CLASS)

Working, but totally removing the possibility of paralle tests:

• Junit 5: @TestMethodOrder(OrderAnnotation.class)

A normal class structure looks like this:

// pseudo code

class View extending IntegrationTestBase extending BrowserTestBase {

  @BrowserTest
  void openViewAsCustomer() {
    login("customer");
    navigate("/dashboard", "Dashboard"); // check that the correct view is loaded

    // assert that stuff is visible
  }

  @BrowserTest
  void openViewAsUser() {
    login("user");
    navigate("/dashboard", "Dashboard"); // check that the correct view is loaded

    // assert that stuff is visible
  }

  @BrowserTest
  void openViewAsAdmin() {
    login("admin");
    navigate("/dashboard", "Dashboard"); // check that the correct view is loaded

    // assert that stuff is visible
  }
}
 

[mshabarov]

@MarcinVaadin do you remember how multithreading can be customised in TestBench, I recall we added a chapter about in for JUnit 5, but I couldn't quickly find it.

[MarcinVaadin]

https://vaadin.com/docs/latest/testing/end-to-end/advanced-concepts#running-tests-in-parallel

[mshabarov]

I also recall we put multithreading config like here https://github.com/vaadin/testbench/blob/a645fddcaf19ff161f4cf4bedfe0e081709a429c/vaadin-testbench-integration-tests-junit5/src/test/resources/junit-platform.properties , but this is not in the main branch for some reason.

[mshabarov]

Here it is https://github.com/vaadin/testbench/blob/main/vaadin-testbench-core-junit5/src/main/resources/junit-platform.properties

[knoobie]

That's a system property, which makes it really hard to configure this depending on the test class in action.

Like I've created multiple classes to test different views that could theoretically all run in parallel without problem - so adding this system property would instantly remove the parallel execution of all those classes even tho I would also want to disable it for a specific class.

Additionally this question adds a bit more to the mix, because it's also requesting documenting about the proper testing of different users with (the default enabled) multi threading.

[Artur-]

resulting in multiple errors with tests that using the same browser to log-in different users and therefore deleting the old session of the user.

Is this a matter of you manually deleting sessions in code? The browser instances running in parallel should not share anything

[knoobie]

Are you sure? In my observation one month ago I had the feeling that chrome does share some things between the browser instances / tabs / however you call it.

We have multiple tests that include different user-logins / logouts and all kind of events. Once those run in parallel, random tests failure occure from "this grid does not contain items (probably spring security returning another user for the current tab and therefore no items)" or multiple exceptions of this kind:

org.openqa.selenium.NoSuchSessionException: invalid session id
Build info: version: '4.10.0', revision: 'c14d967899'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-1160.92.1.el7.x86_64', java.version: '17.0.7'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [654bb52741eafb91c4d8543ff76938d5, executeScript {script=if (window.Vaadin && window.Vaadin.Flow && window.Vaadin.Flow.clients) {  var clients = window.Vaadin.Flow.clients;  for (var client in clients) {    if (clients[client].isActive()) {      return false;    }  }  return true;} else if (window.Vaadin && window.Vaadin.Flow && window.Vaadin.Flow.devServerIsNotLoaded) {  return false;} else {  return true;}, args=[]}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 113.0.5672.126, chrome: {chromedriverVersion: 113.0.5672.63 (0e1a4471d5ae..., userDataDir: /tmp/.com.google.Chrome.6l8Y51}, goog:chromeOptions: {debuggerAddress: localhost:38749}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:cdp: ws://localhost:38749/devtoo..., se:cdpVersion: 113.0.5672.126, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: 654bb52741eafb91c4d8543ff76938d5
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:199)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:132)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:51)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:191)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:196)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:171)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:531)
	at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:476)
	at com.vaadin.testbench.commands.TestBenchCommandExecutor.waitForVaadin(TestBenchCommandExecutor.java:119)
	at com.vaadin.testbench.DriverInvocationHandler.waitForVaadinIfNecessary(DriverInvocationHandler.java:65)
	at com.vaadin.testbench.DriverInvocationHandler.invoke(DriverInvocationHandler.java:33)
	at com.vaadin.testbench.TestBenchDriverProxy_$$_jvstb40_0.getScreenshotAs(TestBenchDriverProxy_$$_jvstb40_0.java)
	at com.vaadin.testbench.ScreenshotOnFailureExtension.testFailed(ScreenshotOnFailureExtension.java:123)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$nodeFinished$15(TestMethodTestDescriptor.java:307)
	at org.junit.jupiter.engine.descriptor.MethodBasedTestDescriptor.lambda$invokeTestWatchers$3(MethodBasedTestDescriptor.java:130)
	at org.junit.platform.commons.util.CollectionUtils.forEachInReverseOrder(CollectionUtils.java:217)
	at org.junit.jupiter.engine.descriptor.MethodBasedTestDescriptor.invokeTestWatchers(MethodBasedTestDescriptor.java:144)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.nodeFinished(TestMethodTestDescriptor.java:298)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.nodeFinished(TestMethodTestDescriptor.java:69)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.reportCompletion(NodeTestTask.java:188)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:100)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:202)

or

org.openqa.selenium.WebDriverException: 
unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
  (Session info: headless chrome=113.0.5672.126)
Build info: version: '4.10.0', revision: 'c14d967899'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-1160.92.1.el7.x86_64', java.version: '17.0.7'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [d26a723e7a533f23ed601731c3cf537f, executeScript {script=if (window.Vaadin && window.Vaadin.Flow && window.Vaadin.Flow.clients) {  var clients = window.Vaadin.Flow.clients;  for (var client in clients) {    if (clients[client].isActive()) {      return false;    }  }  return true;} else if (window.Vaadin && window.Vaadin.Flow && window.Vaadin.Flow.devServerIsNotLoaded) {  return false;} else {  return true;}, args=[]}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 113.0.5672.126, chrome: {chromedriverVersion: 113.0.5672.63 (0e1a4471d5ae..., userDataDir: /tmp/.com.google.Chrome.YLlxh0}, goog:chromeOptions: {debuggerAddress: localhost:39169}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:cdp: ws://localhost:39169/devtoo..., se:cdpVersion: 113.0.5672.126, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: d26a723e7a533f23ed601731c3cf537f
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:199)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:132)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:51)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:191)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:196)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:171)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:531)
	at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:476)
	at com.vaadin.testbench.commands.TestBenchCommandExecutor.waitForVaadin(TestBenchCommandExecutor.java:119)

All problems went away once I extended AbstractBrowserDriverTestBase instead of BrowserTestBasewhich adds the concurrent execution mode.