Issues starting Firefox in CI

[webmaster128]

Since moving the Firefox tests from Mac to Linux, we sometimes have issues starting Firefox. With Karma debugging enabled, this looks like

02 02 2019 20:22:08.813:INFO [karma-server]: Karma v3.1.4 server started at http://0.0.0.0:9878/
02 02 2019 20:22:08.813:INFO [launcher]: Launching browsers Firefox with concurrency unlimited
02 02 2019 20:22:08.825:INFO [launcher]: Starting browser Firefox
02 02 2019 20:22:08.826:DEBUG [temp-dir]: Creating temp dir at /tmp/karma-58370055
02 02 2019 20:22:08.827:DEBUG [launcher]: firefox http://localhost:9878/?id=58370055 -profile /tmp/karma-58370055 -no-remote
02 02 2019 20:22:09.256:DEBUG [launcher]: Process Firefox exited with code null
02 02 2019 20:22:09.256:ERROR [launcher]: Cannot start Firefox
	
02 02 2019 20:22:09.257:ERROR [launcher]: Firefox stdout: 
02 02 2019 20:22:09.257:ERROR [launcher]: Firefox stderr: 
02 02 2019 20:22:09.257:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-58370055
02 02 2019 20:22:09.261:INFO [launcher]: Trying to start Firefox again (1/2).
02 02 2019 20:22:09.262:DEBUG [launcher]: Restarting Firefox
02 02 2019 20:22:09.262:DEBUG [temp-dir]: Creating temp dir at /tmp/karma-58370055
02 02 2019 20:22:09.263:DEBUG [launcher]: firefox http://localhost:9878/?id=58370055 -profile /tmp/karma-58370055 -no-remote
02 02 2019 20:22:09.568:DEBUG [launcher]: Process Firefox exited with code null
02 02 2019 20:22:09.568:ERROR [launcher]: Cannot start Firefox
	
02 02 2019 20:22:09.568:ERROR [launcher]: Firefox stdout: 
02 02 2019 20:22:09.568:ERROR [launcher]: Firefox stderr: 
02 02 2019 20:22:09.568:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-58370055
02 02 2019 20:22:09.570:INFO [launcher]: Trying to start Firefox again (2/2).
02 02 2019 20:22:09.571:DEBUG [launcher]: Restarting Firefox
02 02 2019 20:22:09.571:DEBUG [temp-dir]: Creating temp dir at /tmp/karma-58370055
02 02 2019 20:22:09.571:DEBUG [launcher]: firefox http://localhost:9878/?id=58370055 -profile /tmp/karma-58370055 -no-remote
02 02 2019 20:22:09.852:DEBUG [launcher]: Process Firefox exited with code null
02 02 2019 20:22:09.853:ERROR [launcher]: Cannot start Firefox
	
02 02 2019 20:22:09.854:ERROR [launcher]: Firefox stdout: 
02 02 2019 20:22:09.854:ERROR [launcher]: Firefox stderr: 
02 02 2019 20:22:09.854:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-58370055
02 02 2019 20:22:09.858:ERROR [launcher]: Firefox failed 2 times (cannot start). Giving up.
02 02 2019 20:22:09.859:DEBUG [karma-server]: Run complete, exiting.
02 02 2019 20:22:09.861:DEBUG [launcher]: Disconnecting all browsers

The part

Process Firefox exited with code null

means the process terminated due to receipt of a signal. This PR to karma should log the signal in the future. Let's see. Nothing we can do for now.

[webmaster128]

Issue still exists

08 05 2019 15:16:58.638:DEBUG [config]: Loading config /home/travis/build/iov-one/iov-core/packages/iov-crypto/karma.conf.js
08 05 2019 15:16:58.645:DEBUG [karma-server]: Final config [{"LOG_DISABLE":"1","LOG_ERROR":"2","LOG_WARN":"3","LOG_INFO":"4","LOG_DEBUG":"5","frameworks":"6","protocol":"7","port":9876,"listenAddress":"8","hostname":"9","httpsServerConfig":"10","basePath":"11","files":"12","browserConsoleLogOptions":"13","customContextFile":null,"customDebugFile":null,"customClientContextFile":null,"exclude":"14","logLevel":"5","colors":true,"autoWatch":false,"autoWatchBatchDelay":250,"restartOnFileChange":false,"usePolling":true,"reporters":"15","singleRun":true,"browsers":"16","captureTimeout":60000,"proxies":"17","proxyValidateSSL":true,"preprocessors":"18","urlRoot":"19","reportSlowerThan":0,"loggers":"20","transports":"21","forceJSONP":false,"plugins":"22","client":"23","defaultClient":"23","browserDisconnectTimeout":2000,"browserDisconnectTolerance":0,"browserNoActivityTimeout":90000,"processKillTimeout":2000,"concurrency":null,"failOnEmptyTestSuite":true,"retryLimit":2,"detached":false,"crossOriginAttribute":true,"browserSocketTimeout":20000,"cmd":"24","configFile":"25"},"OFF","ERROR","WARN","INFO","DEBUG",["26"],"http:","0.0.0.0","localhost",{},"/home/travis/build/iov-one/iov-core/packages/iov-crypto",["27"],{"level":"28","format":"29","terminal":true},["25"],["30","31"],["32"],{},{},"/",["33"],["34","35"],["36"],{"args":"37","useIframe":true,"runInParent":false,"captureConsole":true,"clearContext":true,"jasmine":"38"},"start","/home/travis/build/iov-one/iov-core/packages/iov-crypto/karma.conf.js","jasmine",{"pattern":"39","served":true,"included":true,"watched":true,"nocache":false,"weight":"40"},"debug","%b %T: %m","progress","kjhtml","Firefox",{"type":"41","layout":"42"},"polling","websocket","karma-*",[],{"random":false,"timeoutInterval":15000},"/home/travis/build/iov-one/iov-core/packages/iov-crypto/dist/web/tests.js",[1,0,0,0,0,0],"console",{"type":"43","pattern":"44"},"pattern","%[%d{DATE}:%p [%c]: %]%m"]
08 05 2019 15:16:58.647:DEBUG [plugin]: Loading karma-* from /home/travis/build/iov-one/iov-core/node_modules
08 05 2019 15:16:58.648:DEBUG [plugin]: Loading plugin /home/travis/build/iov-one/iov-core/node_modules/karma-chrome-launcher.
08 05 2019 15:16:58.665:DEBUG [plugin]: Loading plugin /home/travis/build/iov-one/iov-core/node_modules/karma-edge-launcher.
08 05 2019 15:16:58.666:DEBUG [plugin]: Loading plugin /home/travis/build/iov-one/iov-core/node_modules/karma-firefox-launcher.
08 05 2019 15:16:58.668:DEBUG [plugin]: Loading plugin /home/travis/build/iov-one/iov-core/node_modules/karma-jasmine.
08 05 2019 15:16:58.669:DEBUG [plugin]: Loading plugin /home/travis/build/iov-one/iov-core/node_modules/karma-jasmine-html-reporter.
08 05 2019 15:16:58.669:DEBUG [plugin]: Loading plugin /home/travis/build/iov-one/iov-core/node_modules/karma-safari-launcher.
08 05 2019 15:16:58.684:DEBUG [web-server]: Instantiating middleware
08 05 2019 15:16:58.695:DEBUG [reporter]: Trying to load reporter: kjhtml
08 05 2019 15:16:58.696:DEBUG [reporter]: Trying to load color-version of reporter: kjhtml (kjhtml_color)
08 05 2019 15:16:58.697:DEBUG [reporter]: Couldn't load color-version.
08 05 2019 15:16:58.777:INFO [karma-server]: Karma v3.1.4 server started at http://0.0.0.0:9877/
08 05 2019 15:16:58.777:INFO [launcher]: Launching browsers Firefox with concurrency unlimited
08 05 2019 15:16:58.784:INFO [launcher]: Starting browser Firefox
08 05 2019 15:16:58.785:DEBUG [temp-dir]: Creating temp dir at /tmp/karma-63735238
08 05 2019 15:16:58.786:DEBUG [launcher]: firefox http://localhost:9877/?id=63735238 -profile /tmp/karma-63735238 -no-remote
08 05 2019 15:17:58.829:WARN [launcher]: Firefox have not captured in 60000 ms, killing.
08 05 2019 15:18:00.832:WARN [launcher]: Firefox was not killed in 2000 ms, sending SIGKILL.
08 05 2019 15:18:02.832:WARN [launcher]: Firefox was not killed by SIGKILL in 2000 ms, continuing.
08 05 2019 15:18:02.833:DEBUG [launcher]: Process Firefox exited with code -1
08 05 2019 15:18:02.833:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-63735238
08 05 2019 15:18:02.837:DEBUG [launcher]: Firefox failed (timeout). Not restarting.
08 05 2019 15:18:02.837:DEBUG [karma-server]: Run complete, exiting.
08 05 2019 15:18:02.837:DEBUG [launcher]: Disconnecting all browsers
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
lerna ERR! yarn run test-firefox stderr:
error Command failed with exit code 1.
lerna ERR! yarn run test-firefox exited 1 in '@iov/crypto'

https://travis-ci.com/iov-one/iov-core/jobs/198597727

This time we have a non-null exit code.

[webmaster128]

Here we have a different issue that appeared after Firefox ran successfully for other tests:

08 05 2019 15:25:02.518:INFO [karma-server]: Karma v3.1.4 server started at http://0.0.0.0:9877/
08 05 2019 15:25:02.522:INFO [launcher]: Launching browsers Firefox with concurrency unlimited
08 05 2019 15:25:02.530:INFO [launcher]: Starting browser Firefox
08 05 2019 15:25:02.576:ERROR [launcher]: Cannot start Firefox
	XPCOMGlueLoad error for file /usr/local/firefox-63.0.1/libxul.so:
/usr/local/firefox-63.0.1/libxul.so: cannot open shared object file: No such file or directory
Couldn't load XPCOM.
08 05 2019 15:25:02.576:ERROR [launcher]: Firefox stdout: 
08 05 2019 15:25:02.576:ERROR [launcher]: Firefox stderr: XPCOMGlueLoad error for file /usr/local/firefox-63.0.1/libxul.so:
/usr/local/firefox-63.0.1/libxul.so: cannot open shared object file: No such file or directory
Couldn't load XPCOM.
08 05 2019 15:25:02.580:INFO [launcher]: Trying to start Firefox again (1/2).
08 05 2019 15:26:02.590:WARN [launcher]: Firefox have not captured in 60000 ms, killing.
08 05 2019 15:26:04.593:WARN [launcher]: Firefox was not killed in 2000 ms, sending SIGKILL.
08 05 2019 15:26:06.596:WARN [launcher]: Firefox was not killed by SIGKILL in 2000 ms, continuing.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
lerna ERR! yarn run test-firefox stderr:
error Command failed with exit code 1.
lerna ERR! yarn run test-firefox exited 1 in '@iov/tendermint-rpc'

https://travis-ci.com/iov-one/iov-core/jobs/198597856