//src/test/java/com/google/devtools/build/lib:remote-tests is very flaky

[aehlig]

The test //src/test/java/com/google/devtools/build/lib:remote-tests is flaky to an extend that it showed up several times as a spurious failure in our CI system, despite this CI system running with --flaky_test_attempts=3.

A quick check doing 100 runs of the test on my corp desktop showed 16 failures, but the flakyness seems to be worse of the CI machines.

[laszlocsomor]

Example flake in "Bazel Remote Execution" on CI (Bazel with downstream projects #298; see log file):

1) multipleBlobsUploadShouldWork(com.google.devtools.build.lib.remote.ByteStreamUploaderTest)
java.lang.Exception: test timed out after 20000 milliseconds
	at java.lang.Thread.sleep(Native Method)
	at com.google.devtools.build.lib.remote.ByteStreamUploaderTest.blockUntilInternalStateConsistent(ByteStreamUploaderTest.java:697)
	at com.google.devtools.build.lib.remote.ByteStreamUploaderTest.multipleBlobsUploadShouldWork(ByteStreamUploaderTest.java:292)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

[ulfjack]

1. singleBlobUploadShouldWork(com.google.devtools.build.lib.remote.ByteStreamUploaderTest)
   java.lang.Exception: test timed out after 10000 milliseconds
   at java.lang.Thread.sleep(Native Method)
   at com.google.devtools.build.lib.remote.ByteStreamUploaderTest.blockUntilInternalStateConsistent(ByteStreamUploaderTest.java:697)
   at com.google.devtools.build.lib.remote.ByteStreamUploaderTest.singleBlobUploadShouldWork(ByteStreamUploaderTest.java:201)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

[fweikert]

Culprit is ebe1ec8
Doubling the timeout doesn't help. We can either increase the timeout even more, roll back the CL or disable the test.