diff --git a/.scala-steward.conf b/.scala-steward.conf index b2233ebc077..02e2704e901 100644 --- a/.scala-steward.conf +++ b/.scala-steward.conf @@ -25,8 +25,7 @@ # # Default: @asap # -#pullRequests.frequency = "0 0 ? * 3" # every thursday on midnight -pullRequests.frequency = "@asap" +pullRequests.frequency = "0 0 1 1,4,7,10 ?" # Run at 00:00 on the 1st day of Jan,Apr,Jul,Oct (whatever day that is) # Only these dependencies which match the given patterns are updated. # @@ -51,7 +50,7 @@ pullRequests.frequency = "@asap" # If set, Scala Steward will only attempt to create or update `n` PRs. # Useful if running frequently and/or CI build are costly # Default: None -updates.limit = 5 +# updates.limit = 5 # By default, Scala Steward does not update scala version since its tricky, error-prone # and results in bad PRs and/or failed builds diff --git a/.travis.yml b/.travis.yml index 1b8bdeba15e..f464339003b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -32,6 +32,9 @@ env: - >- BUILD_TYPE=centaurBcs BUILD_MYSQL=5.7 + - >- + BUILD_TYPE=centaurDummy + BUILD_MYSQL=5.7 - >- BUILD_TYPE=centaurEngineUpgradeLocal BUILD_MYSQL=5.7 diff --git a/CHANGELOG.md b/CHANGELOG.md index 470fb6b8238..2754c152cf4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,41 @@ # Cromwell Change Log +## 55 Release Notes + +### Apple Silicon support statement + +Users with access to the new Mac hardware should review [important information provided here](https://cromwell.readthedocs.io/en/stable/Releases). + +### Bug Fixes + +* Fixed a bug that prevented `read_json()` from working with arrays and primitives. The function now works as expected for all valid JSON data inputs. +More information on JSON Type to WDL Type conversion can be found [here](https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#mixed-read_jsonstringfile). + +* Now retries HTTP 408 responses as well as HTTP 429 responses during DOS/DRS resolution requests. + +* Fixed a bug that prevented the call caching diff endpoint from working with scatters in workflows with archived metadata. + +### New Features + +#### Reference disk support on PAPI v2 + +Cromwell now offers support for the use of reference disks on the PAPI v2 backend as an alternative to localizing +reference inputs. More details [here](https://cromwell.readthedocs.io/en/develop/backends/Google#reference-disk-support). + +#### Docker image cache support on PAPI v2 lifesciences beta + +Cromwell now offers support for the use of Docker image caches on the PAPI v2 lifesciences beta backend. More details [here](https://cromwell.readthedocs.io/en/develop/backends/Google#docker-image-cache-support). + +#### Preemptible Recovery via Checkpointing + +* Cromwell can now help tasks recover from preemption by allowing them to specify a 'checkpoint' file which will be restored +to the worker VM on the next attempt if the task is interrupted. More details [here](https://cromwell.readthedocs.io/en/develop/optimizations/CheckpointFiles) + ## 54 Release Notes ### Bug Fixes -* Fixed a bug where `write_json()` failed for `Array[_]` inputs. It should now work for `Boolean`, `String`, `Integer`, `Float`, +* Fixed a bug that prevented `write_json()` from working with arrays and primitives. The function now works as expected for `Boolean`, `String`, `Integer`, `Float`, `Pair[_, _]`, `Object`, `Map[_, _]` and `Array[_]` (including array of objects) type inputs. More information on WDL Type to JSON Type conversion can be found [here](https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#mixed-read_jsonstringfile). diff --git a/README.md b/README.md index dea34692231..936a484889b 100644 --- a/README.md +++ b/README.md @@ -3,20 +3,43 @@ ## Welcome to Cromwell -Cromwell is a Workflow Management System geared towards scientific workflows. Cromwell is open sourced under the [BSD 3-Clause license](LICENSE.txt). +Cromwell is an open-source Workflow Management System for bioinformatics. Licensing is [BSD 3-Clause](LICENSE.txt). -The Cromwell documentation has a new home, [click here to check it out](https://cromwell.readthedocs.io/en/stable)! +The [Cromwell documentation has a dedicated site](https://cromwell.readthedocs.io/en/stable). -First time to Cromwell? Get started with [Tutorials](https://cromwell.readthedocs.io/en/stable/tutorials/FiveMinuteIntro/)! +First time to Cromwell? Get started with [Tutorials](https://cromwell.readthedocs.io/en/stable/tutorials/FiveMinuteIntro/). + +### Community Thinking about contributing to Cromwell? Get started by reading our [Contributor Guide](CONTRIBUTING.md). Cromwell has a growing ecosystem of community-backed projects to make your experience even better! Check out our [Ecosystem](https://cromwell.readthedocs.io/en/stable/Ecosystem/) page to learn more. -### Issue tracking is now on JIRA +Talk to us: +- [Join the Cromwell Slack workspace](https://join.slack.com/t/cromwellhq/shared_invite/zt-dxmmrtye-JHxwKE53rfKE_ZWdOHIB4g) to discuss the Cromwell workflow engine. +- [Join the OpenWDL Slack workspace](https://join.slack.com/t/openwdl/shared_invite/zt-ctmj4mhf-cFBNxIiZYs6SY9HgM9UAVw) to discuss the evolution of the WDL language itself. + - More information about WDL is available in [that project's repository](https://github.com/openwdl/wdl). + +### Capabilities and roadmap + +A majority of Cromwell users today run their workflows in [Terra](https://app.terra.bio/), a fully-managed cloud-native bioinformatics computing platform. See [here](https://support.terra.bio/hc/en-us/articles/360036379771-Get-started-running-workflows) for a quick-start guide. + +Users with specialized needs who wish to install and maintain their own Cromwell instances can [download](https://github.com/broadinstitute/cromwell/releases) a JAR or Docker image. The development team accepts reproducible bug reports from self-managed instances, but cannot feasibly provide direct support. + +[Cromwell's backends](https://cromwell.readthedocs.io/en/stable/backends/Backends/) receive development resources proportional to customer demand. The team is actively developing for Google Cloud and AWS. Maintenance of other backends is primarily community-based. + +Cromwell [supports](https://cromwell.readthedocs.io/en/stable/LanguageSupport/) the WDL and CWL workflow languages. The Cromwell team is actively developing WDL, while maintenance for CWL is primarily community-based. + +### Issue tracking in JIRA + + + +Need to file an issue? Head over to [our JIRA](https://broadworkbench.atlassian.net/jira/software/c/projects/CROM/issues). You must create a free profile to view or create. -Need to file an issue? Head over to [our JIRA](https://broadworkbench.atlassian.net/projects/BA/issues). You can sign in with any Google account. +[Issues in Github](https://github.com/broadinstitute/cromwell/issues) remain available for discussion among community members but are not actively monitored by the development team. -As of May 2019, we are in the process of migrating all issues from Github to JIRA. At a later date to be announced, submitting new Github issues will be disabled. +![Cromwell JIRA](docs/img/cromwell_jira.png) ![Jamie, the Cromwell pig](docs/jamie_the_cromwell_pig.png) diff --git a/backend/src/main/scala/cromwell/backend/dummy/DummyAsyncExecutionActor.scala b/backend/src/main/scala/cromwell/backend/dummy/DummyAsyncExecutionActor.scala new file mode 100644 index 00000000000..5c491c5a741 --- /dev/null +++ b/backend/src/main/scala/cromwell/backend/dummy/DummyAsyncExecutionActor.scala @@ -0,0 +1,127 @@ +package cromwell.backend.dummy + +import java.time.OffsetDateTime +import java.util.UUID + +import cats.data.NonEmptyList +import cats.data.Validated.{Invalid, Valid} +import cats.implicits._ +import common.exception.AggregatedMessageException +import common.validation.ErrorOr.ErrorOr +import cromwell.backend.BackendJobLifecycleActor +import cromwell.backend.async.{ExecutionHandle, FailedNonRetryableExecutionHandle, PendingExecutionHandle, SuccessfulExecutionHandle} +import cromwell.backend.standard.{StandardAsyncExecutionActor, StandardAsyncExecutionActorParams, StandardAsyncJob} +import cromwell.core.CallOutputs +import cromwell.core.retry.SimpleExponentialBackoff +import cromwell.services.instrumentation.CromwellInstrumentation +import wom.expression.NoIoFunctionSet +import wom.graph.GraphNodePort.{ExpressionBasedOutputPort, OutputPort} +import wom.values.WomValue + +import scala.concurrent.Future +import scala.concurrent.duration._ + +class DummyAsyncExecutionActor(override val standardParams: StandardAsyncExecutionActorParams) + extends BackendJobLifecycleActor + with StandardAsyncExecutionActor + with CromwellInstrumentation { + + /** The type of the run info when a job is started. */ + override type StandardAsyncRunInfo = String + /** The type of the run status returned during each poll. */ + override type StandardAsyncRunState = String + + /** Should return true if the status contained in `thiz` is equivalent to `that`, delta any other data that might be carried around + * in the state type. + */ + override def statusEquivalentTo(thiz: String)(that: String): Boolean = thiz == that + + /** + * Returns true when a job is complete, either successfully or unsuccessfully. + * + * @param runStatus The run status. + * @return True if the job has completed. + */ + override def isTerminal(runStatus: String): Boolean = runStatus == "DummyDone" + + override def dockerImageUsed: Option[String] = None + + override def pollBackOff: SimpleExponentialBackoff = SimpleExponentialBackoff(initialInterval = 1.second, maxInterval = 300.seconds, multiplier = 1.1) + + override def executeOrRecoverBackOff: SimpleExponentialBackoff = SimpleExponentialBackoff(initialInterval = 1.second, maxInterval = 300.seconds, multiplier = 1.1) + + override val logJobIds: Boolean = false + + val singletonActor = standardParams.backendSingletonActorOption.getOrElse( + throw new RuntimeException("Dummy Backend actor cannot exist without its singleton actor")) + + var finishTime: Option[OffsetDateTime] = None + + override def executeAsync(): Future[ExecutionHandle] = { + finishTime = Option(OffsetDateTime.now().plusMinutes(3)) + increment(NonEmptyList("jobs", List("dummy", "executing", "starting"))) + singletonActor ! DummySingletonActor.PlusOne + Future.successful( + PendingExecutionHandle[StandardAsyncJob, StandardAsyncRunInfo, StandardAsyncRunState]( + jobDescriptor = jobDescriptor, + pendingJob = StandardAsyncJob(UUID.randomUUID().toString), + runInfo = Option("pending"), + previousState = None + ) + ) + } + + override def pollStatusAsync(handle: StandardAsyncPendingExecutionHandle): Future[String] = { + finishTime match { + case Some(ft) if (ft.isBefore(OffsetDateTime.now)) => Future.successful("done") + case Some(_) => Future.successful("running") + case None => Future.failed(new Exception("Dummy backend polling for status before finishTime is established(!!?)")) + } + + } + + override def handlePollSuccess(oldHandle: StandardAsyncPendingExecutionHandle, state: String): Future[ExecutionHandle] = { + + if (state == "done") { + + increment(NonEmptyList("jobs", List("dummy", "executing", "done"))) + singletonActor ! DummySingletonActor.MinusOne + + val outputsValidation: ErrorOr[Map[OutputPort, WomValue]] = jobDescriptor.taskCall.outputPorts.toList.traverse { + case expressionBasedOutputPort: ExpressionBasedOutputPort => + expressionBasedOutputPort.expression.evaluateValue(Map.empty, NoIoFunctionSet).map(expressionBasedOutputPort -> _) + case other => s"Unknown output port type for Dummy backend output evaluator: ${other.getClass.getSimpleName}".invalidNel + }.map(_.toMap) + + outputsValidation match { + case Valid(outputs) => + Future.successful(SuccessfulExecutionHandle( + outputs = CallOutputs(outputs.toMap), + returnCode = 0, + jobDetritusFiles = Map.empty, + executionEvents = Seq.empty, + resultsClonedFrom = None + )) + case Invalid(errors) => + Future.successful(FailedNonRetryableExecutionHandle( + throwable = AggregatedMessageException("Evaluate outputs from dummy job", errors.toList), + returnCode = None, + kvPairsToSave = None + )) + } + } + else if (state == "running") { + Future.successful( + PendingExecutionHandle[StandardAsyncJob, StandardAsyncRunInfo, StandardAsyncRunState]( + jobDescriptor = jobDescriptor, + pendingJob = StandardAsyncJob(UUID.randomUUID().toString), + runInfo = Option("pending"), + previousState = Option(state) + ) + ) + } + else { + Future.failed(new Exception(s"Unexpected Dummy state in handlePollSuccess: $state")) + } + } +} diff --git a/backend/src/main/scala/cromwell/backend/dummy/DummyInitializationActor.scala b/backend/src/main/scala/cromwell/backend/dummy/DummyInitializationActor.scala new file mode 100644 index 00000000000..b0962298b18 --- /dev/null +++ b/backend/src/main/scala/cromwell/backend/dummy/DummyInitializationActor.scala @@ -0,0 +1,29 @@ +package cromwell.backend.dummy + +import cats.syntax.validated._ +import common.validation.ErrorOr.ErrorOr +import cromwell.backend.standard.{StandardInitializationActor, StandardInitializationActorParams, StandardValidatedRuntimeAttributesBuilder} +import cromwell.backend.validation.RuntimeAttributesValidation +import wom.expression.WomExpression +import wom.types.{WomStringType, WomType} +import wom.values.{WomString, WomValue} + +class DummyInitializationActor(pipelinesParams: StandardInitializationActorParams) + extends StandardInitializationActor(pipelinesParams) { + + override protected lazy val runtimeAttributeValidators: Map[String, Option[WomExpression] => Boolean] = Map("backend" -> { _ => true } ) + + // Specific validator for "backend" to let me specify it in test cases (to avoid accidentally submitting the workflow to real backends!) + val backendAttributeValidation: RuntimeAttributesValidation[String] = new RuntimeAttributesValidation[String] { + override def key: String = "backend" + + override def coercion: Traversable[WomType] = Vector(WomStringType) + + override protected def validateValue: PartialFunction[WomValue, ErrorOr[String]] = { + case WomString("Dummy") => "Dummy".validNel + case other => s"Unexpected dummy backend value: $other".invalidNel + } + } + + override def runtimeAttributesBuilder: StandardValidatedRuntimeAttributesBuilder = super.runtimeAttributesBuilder.withValidation(backendAttributeValidation) +} diff --git a/backend/src/main/scala/cromwell/backend/dummy/DummyLifecycleActorFactory.scala b/backend/src/main/scala/cromwell/backend/dummy/DummyLifecycleActorFactory.scala new file mode 100644 index 00000000000..ee959513a36 --- /dev/null +++ b/backend/src/main/scala/cromwell/backend/dummy/DummyLifecycleActorFactory.scala @@ -0,0 +1,30 @@ +package cromwell.backend.dummy + +import akka.actor.{ActorRef, Props} +import cromwell.backend.BackendConfigurationDescriptor +import cromwell.backend.standard.callcaching.{StandardCacheHitCopyingActor, StandardFileHashingActor} +import cromwell.backend.standard.{StandardAsyncExecutionActor, StandardInitializationActor, StandardLifecycleActorFactory} + +class DummyLifecycleActorFactory(override val name: String, override val configurationDescriptor: BackendConfigurationDescriptor) extends StandardLifecycleActorFactory { + + /** + * @return the key to use for storing and looking up the job id. + */ + override def jobIdKey: String = "__dummy_operation_id" + + /** + * @return the asynchronous executor class. + */ + override def asyncExecutionActorClass: Class[_ <: StandardAsyncExecutionActor] = classOf[DummyAsyncExecutionActor] + + // Don't cache-hit copy + override lazy val cacheHitCopyingActorClassOption: Option[Class[_ <: StandardCacheHitCopyingActor]] = None + + // Don't hash files + override lazy val fileHashingActorClassOption: Option[Class[_ <: StandardFileHashingActor]] = None + + override def backendSingletonActorProps(serviceRegistryActor: ActorRef): Option[Props] = Option(Props(new DummySingletonActor())) + + override lazy val initializationActorClass: Class[_ <: StandardInitializationActor] = classOf[DummyInitializationActor] + +} diff --git a/backend/src/main/scala/cromwell/backend/dummy/DummySingletonActor.scala b/backend/src/main/scala/cromwell/backend/dummy/DummySingletonActor.scala new file mode 100644 index 00000000000..35c689ae2b4 --- /dev/null +++ b/backend/src/main/scala/cromwell/backend/dummy/DummySingletonActor.scala @@ -0,0 +1,63 @@ +package cromwell.backend.dummy + +import java.io.File +import java.time.OffsetDateTime +import java.util.UUID + +import akka.actor.Actor +import com.typesafe.scalalogging.StrictLogging +import cromwell.backend.dummy.DummySingletonActor._ + +import scala.concurrent.ExecutionContext +import scala.concurrent.duration._ + +final class DummySingletonActor() extends Actor with StrictLogging { + + implicit val ec: ExecutionContext = context.dispatcher + var count: Int = 0 + + var countHistory: Vector[(OffsetDateTime, Int)] = Vector.empty + + override def receive: Receive = { + case PlusOne => count = count + 1 + case MinusOne => count = count - 1 + case PrintCount => + if(countHistory.lastOption.exists(_._2 != count)) { + countHistory = countHistory :+ (OffsetDateTime.now() -> count) + logger.info("The current count is now: " + count) + if (count == 0) { + outputCountHistory() + countHistory = Vector.empty + } + } else { + countHistory = countHistory :+ (OffsetDateTime.now() -> count) + } + + } + + private def outputCountHistory() = { + import java.io.BufferedWriter + import java.io.FileOutputStream + import java.io.OutputStreamWriter + val fout = new File(s"timestamps-${UUID.randomUUID().toString}.tsv") + val fos = new FileOutputStream(fout) + + val bw = new BufferedWriter(new OutputStreamWriter(fos)) + + for ((timestamp, count) <- countHistory) { + bw.write(s"$timestamp\t$count") + bw.newLine() + } + bw.flush() + bw.close() + } + + context.system.scheduler.schedule(10.seconds, 1.second) { self ! PrintCount } +} + +object DummySingletonActor { + case object PlusOne + case object MinusOne + case object PrintCount +} + diff --git a/backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala b/backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala index b498e4cc025..61dab8f3b2c 100644 --- a/backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala +++ b/backend/src/main/scala/cromwell/backend/standard/StandardAsyncExecutionActor.scala @@ -131,6 +131,8 @@ trait StandardAsyncExecutionActor default = s"""$$(mkdir -p "${runtimeEnvironment.tempPath}" && echo "${runtimeEnvironment.tempPath}")""" ) + val logJobIds: Boolean = true + /** Used to convert cloud paths into local paths. */ protected def preProcessWomFile(womFile: WomFile): WomFile = womFile @@ -1105,7 +1107,7 @@ trait StandardAsyncExecutionActor configurationDescriptor.slowJobWarningAfter foreach { duration => self ! WarnAboutSlownessAfter(handle.pendingJob.jobId, duration) } tellKvJobId(handle.pendingJob) map { _ => - jobLogger.info(s"job id: ${handle.pendingJob.jobId}") + if (logJobIds) jobLogger.info(s"job id: ${handle.pendingJob.jobId}") tellMetadata(Map(CallMetadataKeys.JobId -> handle.pendingJob.jobId)) /* NOTE: Because of the async nature of the Scala Futures, there is a point in time where we have submitted this or diff --git a/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala b/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala index 5f20e60318d..791f2a3905a 100644 --- a/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala +++ b/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardCacheHitCopyingActor.scala @@ -128,11 +128,11 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit override lazy val configurationDescriptor: BackendConfigurationDescriptor = standardParams.configurationDescriptor protected val commandBuilder: IoCommandBuilder = DefaultIoCommandBuilder - lazy val cacheCopyJobPaths = jobPaths.forCallCacheCopyAttempts + lazy val cacheCopyJobPaths: JobPaths = jobPaths.forCallCacheCopyAttempts lazy val destinationCallRootPath: Path = cacheCopyJobPaths.callRoot def destinationJobDetritusPaths: Map[String, Path] = cacheCopyJobPaths.detritusPaths - lazy val ioActor = standardParams.ioActor + lazy val ioActor: ActorRef = standardParams.ioActor startWith(Idle, None) @@ -174,10 +174,29 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit detritusAndOutputsIoCommands foreach sendIoCommand // Add potential additional commands to the list - val additionalCommands = additionalIoCommands(sourceCallRootPath, simpletons, destinationCallOutputs, jobDetritus, destinationDetritus) - val allCommands = List(detritusAndOutputsIoCommands) ++ additionalCommands - - goto(WaitingForIoResponses) using Option(StandardCacheHitCopyingActorData(allCommands, destinationCallOutputs, destinationDetritus, cacheHit, returnCode)) + val additionalCommandsTry = + additionalIoCommands( + sourceCallRootPath = sourceCallRootPath, + originalSimpletons = simpletons, + newOutputs = destinationCallOutputs, + originalDetritus = jobDetritus, + newDetritus = destinationDetritus, + ) + additionalCommandsTry match { + case Success(additionalCommands) => + val allCommands = List(detritusAndOutputsIoCommands) ++ additionalCommands + goto(WaitingForIoResponses) using + Option(StandardCacheHitCopyingActorData( + commandsToWaitFor = allCommands, + newJobOutputs = destinationCallOutputs, + newDetritus = destinationDetritus, + cacheHit = cacheHit, + returnCode = returnCode, + )) + // Something went wrong in generating duplication commands. + // We consider this a loggable error because we don't expect this to happen: + case Failure(failure) => failAndStop(CopyAttemptError(failure)) + } case _ => succeedAndStop(returnCode, destinationCallOutputs, destinationDetritus) } @@ -270,7 +289,7 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit andThen } - def succeedAndStop(returnCode: Option[Int], copiedJobOutputs: CallOutputs, detritusMap: DetritusMap) = { + def succeedAndStop(returnCode: Option[Int], copiedJobOutputs: CallOutputs, detritusMap: DetritusMap): State = { import cromwell.services.metadata.MetadataService.implicits.MetadataAutoPutter serviceRegistryActor.putMetadata(jobDescriptor.workflowDescriptor.id, Option(jobDescriptor.key), startMetadataKeyValues) context.parent ! JobSucceededResponse(jobDescriptor.key, returnCode, copiedJobOutputs, Option(detritusMap), Seq.empty, None, resultGenerationMode = CallCached) @@ -300,7 +319,7 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit } } - def abort() = { + def abort(): State = { log.warning("{}: Abort not supported during cache hit copying", jobTag) context.parent ! JobAbortedResponse(jobDescriptor.key) context stop self @@ -329,7 +348,8 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit val destinationSimpleton = WomValueSimpleton(key, WomSingleFile(destinationPath.pathAsString)) - List(destinationSimpleton) -> Set(commandBuilder.copyCommand(sourcePath, destinationPath, overwrite = true)) + // PROD-444: Keep It Short and Simple: Throw on the first error and let the outer Try catch-and-re-wrap + List(destinationSimpleton) -> Set(commandBuilder.copyCommand(sourcePath, destinationPath).get) case nonFileSimpleton => (List(nonFileSimpleton), Set.empty[IoCommand[_]]) }) @@ -339,7 +359,7 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit /** * Returns the file (and ONLY the file detritus) intersection between the cache hit and this call. */ - protected final def detritusFileKeys(sourceJobDetritusFiles: Map[String, String]) = { + protected final def detritusFileKeys(sourceJobDetritusFiles: Map[String, String]): Set[String] = { val sourceKeys = sourceJobDetritusFiles.keySet val destinationKeys = destinationJobDetritusPaths.keySet sourceKeys.intersect(destinationKeys).filterNot(_ == JobPaths.CallRootPathKey) @@ -360,7 +380,8 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit val newDetrituses = detrituses + (detritus -> destinationPath) - (newDetrituses, commands + commandBuilder.copyCommand(sourcePath, destinationPath, overwrite = true)) + // PROD-444: Keep It Short and Simple: Throw on the first error and let the outer Try catch-and-re-wrap + (newDetrituses, commands + commandBuilder.copyCommand(sourcePath, destinationPath).get) }) (destinationDetritus + (JobPaths.CallRootPathKey -> destinationCallRootPath), ioCommands) @@ -374,7 +395,7 @@ abstract class StandardCacheHitCopyingActor(val standardParams: StandardCacheHit originalSimpletons: Seq[WomValueSimpleton], newOutputs: CallOutputs, originalDetritus: Map[String, String], - newDetritus: Map[String, Path]): List[Set[IoCommand[_]]] = List.empty + newDetritus: Map[String, Path]): Try[List[Set[IoCommand[_]]]] = Success(Nil) override protected def onTimeout(message: Any, to: ActorRef): Unit = { val exceptionMessage = message match { diff --git a/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardFileHashingActor.scala b/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardFileHashingActor.scala index 252da5fbf52..78fdc078cb9 100644 --- a/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardFileHashingActor.scala +++ b/backend/src/main/scala/cromwell/backend/standard/callcaching/StandardFileHashingActor.scala @@ -47,7 +47,7 @@ case class DefaultStandardFileHashingActorParams case class FileHashContext(hashKey: HashKey, file: String) class DefaultStandardFileHashingActor(standardParams: StandardFileHashingActorParams) extends StandardFileHashingActor(standardParams) { - override val ioCommandBuilder = DefaultIoCommandBuilder + override val ioCommandBuilder: IoCommandBuilder = DefaultIoCommandBuilder } object StandardFileHashingActor { @@ -67,7 +67,7 @@ abstract class StandardFileHashingActor(standardParams: StandardFileHashingActor with IoClientHelper with StandardCachingActorHelper with Timers { - override lazy val ioActor = standardParams.ioActor + override lazy val ioActor: ActorRef = standardParams.ioActor override lazy val jobDescriptor: BackendJobDescriptor = standardParams.jobDescriptor override lazy val backendInitializationDataOption: Option[BackendInitializationData] = standardParams.backendInitializationDataOption override lazy val serviceRegistryActor: ActorRef = standardParams.serviceRegistryActor @@ -91,7 +91,10 @@ abstract class StandardFileHashingActor(standardParams: StandardFileHashingActor context.parent ! FileHashResponse(HashResult(fileHashRequest.hashKey, HashValue(result))) case (fileHashRequest: FileHashContext, IoSuccess(_, other)) => - context.parent ! HashingFailedMessage(fileHashRequest.file, new Exception(s"Hash function supposedly succeeded but responded with '${other}' instead of a string hash")) + context.parent ! HashingFailedMessage( + fileHashRequest.file, + new Exception(s"Hash function supposedly succeeded but responded with '$other' instead of a string hash"), + ) // Hash Failure case (fileHashRequest: FileHashContext, IoFailAck(_, failure: Throwable)) => @@ -101,12 +104,12 @@ abstract class StandardFileHashingActor(standardParams: StandardFileHashingActor log.warning(s"Async File hashing actor received unexpected message: $other") } - def asyncHashing(fileRequest: SingleFileHashRequest, replyTo: ActorRef) = { + def asyncHashing(fileRequest: SingleFileHashRequest, replyTo: ActorRef): Unit = { val fileAsString = fileRequest.file.value - val ioHashCommandTry = Try { - val gcsPath = getPath(fileAsString).get - ioCommandBuilder.hashCommand(gcsPath) - } + val ioHashCommandTry = for { + gcsPath <- getPath(fileAsString) + command <- ioCommandBuilder.hashCommand(gcsPath) + } yield command lazy val fileHashContext = FileHashContext(fileRequest.hashKey, fileRequest.file.value) ioHashCommandTry match { diff --git a/backend/src/test/scala/cromwell/backend/BackendWorkflowInitializationActorSpec.scala b/backend/src/test/scala/cromwell/backend/BackendWorkflowInitializationActorSpec.scala index a8c0c24cdf4..1fc1f0c6b29 100644 --- a/backend/src/test/scala/cromwell/backend/BackendWorkflowInitializationActorSpec.scala +++ b/backend/src/test/scala/cromwell/backend/BackendWorkflowInitializationActorSpec.scala @@ -4,7 +4,7 @@ import _root_.wdl.draft2.model.WdlExpression import _root_.wdl.draft2.model.types._ import akka.actor.ActorRef import akka.testkit.TestActorRef -import com.typesafe.config.ConfigFactory +import com.typesafe.config.{Config, ConfigFactory} import cromwell.backend.validation.{ContinueOnReturnCodeFlag, ContinueOnReturnCodeSet, ContinueOnReturnCodeValidation} import cromwell.core.{TestKitSuite, WorkflowOptions} import org.scalatest.flatspec.AnyFlatSpecLike @@ -20,7 +20,7 @@ import scala.concurrent.Future import scala.util.Try -class BackendWorkflowInitializationActorSpec extends TestKitSuite("BackendWorkflowInitializationActorSpec") +class BackendWorkflowInitializationActorSpec extends TestKitSuite with AnyFlatSpecLike with Matchers with TableDrivenPropertyChecks { behavior of "BackendWorkflowInitializationActorSpec" @@ -33,11 +33,11 @@ class BackendWorkflowInitializationActorSpec extends TestKitSuite("BackendWorkfl TestPredicateBackendWorkflowInitializationActor = testPredicateBackendWorkflowInitializationActorRef.underlyingActor - val testContinueOnReturnCode: (Option[WomValue]) => Boolean = { + val testContinueOnReturnCode: Option[WomValue] => Boolean = { testPredicateBackendWorkflowInitializationActor.continueOnReturnCodePredicate(valueRequired = false) } - val optionalConfig = Option(TestConfig.optionalRuntimeConfig) + val optionalConfig: Option[Config] = Option(TestConfig.optionalRuntimeConfig) it should "continueOnReturnCodePredicate" in { testContinueOnReturnCode(None) should be(true) @@ -191,7 +191,7 @@ class TestPredicateBackendWorkflowInitializationActor extends BackendWorkflowIni override def calls: Set[CommandCallNode] = throw new UnsupportedOperationException("calls") - override protected def runtimeAttributeValidators: Map[String, (Option[WomExpression]) => Boolean] = + override protected def runtimeAttributeValidators: Map[String, Option[WomExpression] => Boolean] = throw new UnsupportedOperationException("runtimeAttributeValidators") override protected def coerceDefaultRuntimeAttributes(options: WorkflowOptions): Try[Map[String, WomValue]] = diff --git a/backend/src/test/scala/cromwell/backend/standard/callcaching/RootWorkflowHashCacheActorSpec.scala b/backend/src/test/scala/cromwell/backend/standard/callcaching/RootWorkflowHashCacheActorSpec.scala index 377d012ed11..f546dd63630 100644 --- a/backend/src/test/scala/cromwell/backend/standard/callcaching/RootWorkflowHashCacheActorSpec.scala +++ b/backend/src/test/scala/cromwell/backend/standard/callcaching/RootWorkflowHashCacheActorSpec.scala @@ -13,17 +13,20 @@ import org.scalatest.flatspec.AnyFlatSpecLike import scala.concurrent.duration._ -class RootWorkflowHashCacheActorSpec extends TestKitSuite("RootWorkflowHashCacheActorSpec") with ImplicitSender +class RootWorkflowHashCacheActorSpec extends TestKitSuite with ImplicitSender with AnyFlatSpecLike { private val fakeWorkflowId = WorkflowId.randomId() private val fakeFileName = "fakeFileName" it should "properly handle the situation when response from IoActor received after the timeout" in { - val ioActorProbe = TestProbe() - val rootWorkflowFileHashCacheActor = system.actorOf(Props(new RootWorkflowFileHashCacheActor(ioActorProbe.ref, fakeWorkflowId) { - override lazy val defaultIoTimeout = 1.second - })) + val ioActorProbe = TestProbe("ioActorProbe-without-timer") + val rootWorkflowFileHashCacheActor = system.actorOf( + props = Props(new RootWorkflowFileHashCacheActor(ioActorProbe.ref, fakeWorkflowId) { + override lazy val defaultIoTimeout: FiniteDuration = 1.second + }), + name = "rootWorkflowFileHashCacheActor-without-timer", + ) val ioHashCommandWithContext = IoHashCommandWithContext(DefaultIoHashCommand(DefaultPathBuilder.build("").get), FileHashContext(HashKey(checkForHitOrMiss = false, List.empty), fakeFileName)) rootWorkflowFileHashCacheActor ! ioHashCommandWithContext @@ -37,11 +40,14 @@ class RootWorkflowHashCacheActorSpec extends TestKitSuite("RootWorkflowHashCache } it should "properly handle the situation when timeout occurs when response from IoActor has already been received, but timer has not yet been disabled" in { - val ioActorProbe = TestProbe() - val rootWorkflowFileHashCacheActor = system.actorOf(Props(new RootWorkflowFileHashCacheActor(ioActorProbe.ref, fakeWorkflowId) { - // Effectively disabling automatic timeout firing here. We'll send RequestTimeout ourselves - override lazy val defaultIoTimeout = 1.hour - })) + val ioActorProbe = TestProbe("ioActorProbe-with-timer") + val rootWorkflowFileHashCacheActor = system.actorOf( + Props(new RootWorkflowFileHashCacheActor(ioActorProbe.ref, fakeWorkflowId) { + // Effectively disabling automatic timeout firing here. We'll send RequestTimeout ourselves + override lazy val defaultIoTimeout: FiniteDuration = 1.hour + }), + "rootWorkflowFileHashCacheActor-with-timer", + ) val ioHashCommandWithContext = IoHashCommandWithContext(DefaultIoHashCommand(DefaultPathBuilder.build("").get), FileHashContext(HashKey(checkForHitOrMiss = false, List.empty), fakeFileName)) rootWorkflowFileHashCacheActor ! ioHashCommandWithContext diff --git a/backend/src/test/scala/cromwell/backend/standard/callcaching/StandardFileHashingActorSpec.scala b/backend/src/test/scala/cromwell/backend/standard/callcaching/StandardFileHashingActorSpec.scala index 4d0280a946a..51767801cea 100644 --- a/backend/src/test/scala/cromwell/backend/standard/callcaching/StandardFileHashingActorSpec.scala +++ b/backend/src/test/scala/cromwell/backend/standard/callcaching/StandardFileHashingActorSpec.scala @@ -6,26 +6,29 @@ import cromwell.backend.standard.callcaching.StandardFileHashingActor.SingleFile import cromwell.backend.{BackendConfigurationDescriptor, BackendInitializationData, BackendJobDescriptor} import cromwell.core.TestKitSuite import cromwell.core.callcaching.HashingFailedMessage -import cromwell.core.io.{IoCommandBuilder, IoHashCommand, PartialIoCommandBuilder} +import cromwell.core.io.{IoCommand, IoCommandBuilder, IoHashCommand, IoSuccess, PartialIoCommandBuilder} import cromwell.core.path.{DefaultPathBuilder, Path} import org.scalatest.flatspec.AnyFlatSpecLike import org.scalatest.matchers.should.Matchers +import org.specs2.mock.Mockito import wom.values.WomSingleFile import scala.concurrent.TimeoutException import scala.concurrent.duration._ -import scala.util.Try +import scala.util.control.NoStackTrace +import scala.util.{Failure, Try} -class StandardFileHashingActorSpec extends TestKitSuite("StandardFileHashingActorSpec") with ImplicitSender - with AnyFlatSpecLike with Matchers { +class StandardFileHashingActorSpec extends TestKitSuite with ImplicitSender + with AnyFlatSpecLike with Matchers with Mockito { behavior of "StandardFileHashingActor" - it should "return a failure to the parent when getPath throws an exception" in { - val parentProbe = TestProbe() + it should "return a failure to the parent when getPath returns an exception" in { + val parentProbe = TestProbe("parentProbe") val params = StandardFileHashingActorSpec.defaultParams() val props = Props(new StandardFileHashingActor(params) { - override def getPath(str: String): Try[Path] = throw new RuntimeException("I am expected during tests") + override def getPath(str: String): Try[Path] = + Failure(new RuntimeException("I am expected during tests") with NoStackTrace) }) val standardFileHashingActorRef = TestActorRef(props, parentProbe.ref) val request = SingleFileHashRequest(null, null, WomSingleFile("/expected/failure/path"), None) @@ -41,10 +44,10 @@ class StandardFileHashingActorSpec extends TestKitSuite("StandardFileHashingActo } it should "return a failure to the parent when hashCommand throws an exception" in { - val parentProbe = TestProbe() + val parentProbe = TestProbe("parentProbe") val params = StandardFileHashingActorSpec.defaultParams() val props = Props(new StandardFileHashingActor(params) { - override val ioCommandBuilder = IoCommandBuilder( + override val ioCommandBuilder: IoCommandBuilder = IoCommandBuilder( new PartialIoCommandBuilder { override def hashCommand = throw new RuntimeException("I am expected during tests") } @@ -65,11 +68,11 @@ class StandardFileHashingActorSpec extends TestKitSuite("StandardFileHashingActo } it should "send a timeout to the ioActor the command doesn't hash" in { - val parentProbe = TestProbe() - val ioActorProbe = TestProbe() + val parentProbe = TestProbe("parentProbe") + val ioActorProbe = TestProbe("ioActorProbe") val params = StandardFileHashingActorSpec.ioActorParams(ioActorProbe.ref) val props = Props(new StandardFileHashingActor(params) { - override lazy val defaultIoTimeout = 1.second.dilated + override lazy val defaultIoTimeout: FiniteDuration = 1.second.dilated override def getPath(str: String): Try[Path] = Try(DefaultPathBuilder.get(str)) }) @@ -91,6 +94,32 @@ class StandardFileHashingActorSpec extends TestKitSuite("StandardFileHashingActo } } + it should "handle string hash responses" in { + val parentProbe = TestProbe("testParentHashString") + val params = StandardFileHashingActorSpec.ioActorParams(ActorRef.noSender) + val props = Props(new StandardFileHashingActor(params) { + override lazy val defaultIoTimeout: FiniteDuration = 1.second.dilated + + override def getPath(str: String): Try[Path] = Try(DefaultPathBuilder.get(str)) + }) + val standardFileHashingActorRef = parentProbe.childActorOf(props, "testStandardFileHashingActorHashString") + + val fileHashContext = mock[FileHashContext].smart + fileHashContext.file returns "/expected/failure/path" + val command = mock[IoCommand[Int]].smart + val message: (FileHashContext, IoSuccess[Int]) = (fileHashContext, IoSuccess(command, 1357)) + + standardFileHashingActorRef ! message + + parentProbe.expectMsgPF(10.seconds.dilated) { + case failed: HashingFailedMessage => + failed.reason should be(a[Exception]) + failed.reason.getMessage should + be("Hash function supposedly succeeded but responded with '1357' instead of a string hash") + case unexpected => fail(s"received unexpected message $unexpected") + } + } + } object StandardFileHashingActorSpec { @@ -113,15 +142,15 @@ object StandardFileHashingActorSpec { withBackendInitializationDataOption: => Option[BackendInitializationData] ): StandardFileHashingActorParams = new StandardFileHashingActorParams { - override def jobDescriptor = withJobDescriptor + override def jobDescriptor: BackendJobDescriptor = withJobDescriptor - override def configurationDescriptor = withConfigurationDescriptor + override def configurationDescriptor: BackendConfigurationDescriptor = withConfigurationDescriptor - override def ioActor = withIoActor + override def ioActor: ActorRef = withIoActor - override def serviceRegistryActor = withServiceRegistryActor + override def serviceRegistryActor: ActorRef = withServiceRegistryActor - override def backendInitializationDataOption = withBackendInitializationDataOption + override def backendInitializationDataOption: Option[BackendInitializationData] = withBackendInitializationDataOption override def fileHashCachingActor: Option[ActorRef] = None } diff --git a/centaur/src/main/resources/standardTestCases/attempt_to_call_size_function_on_bucket.test b/centaur/src/main/resources/standardTestCases/attempt_to_call_size_function_on_bucket.test index 3fd5f4c8dbf..4c444548580 100644 --- a/centaur/src/main/resources/standardTestCases/attempt_to_call_size_function_on_bucket.test +++ b/centaur/src/main/resources/standardTestCases/attempt_to_call_size_function_on_bucket.test @@ -7,5 +7,6 @@ files { } metadata { - "failures.0.causedBy.0.causedBy.0.message": "Failed to evaluate input 'in_file_size' (reason 1 of 1): [Attempted 1 time(s)] - Exception: Error processing IO response in onSuccessCallback: null" + "failures.0.causedBy.0.causedBy.0.message": + "Failed to evaluate input 'in_file_size' (reason 1 of 1): GcsPath 'gs://hsd_salmon_index/' is bucket only and does not specify an object blob." } diff --git a/centaur/src/main/resources/standardTestCases/bucket_name_with_no_trailing_slash.test b/centaur/src/main/resources/standardTestCases/bucket_name_with_no_trailing_slash.test index f6d6cb73f0d..a7de7263245 100644 --- a/centaur/src/main/resources/standardTestCases/bucket_name_with_no_trailing_slash.test +++ b/centaur/src/main/resources/standardTestCases/bucket_name_with_no_trailing_slash.test @@ -5,11 +5,11 @@ backends: [Papiv2] files { workflow: attempt_to_localize_bucket_as_file/attempt_to_localize_bucket_as_file.wdl inputs: attempt_to_localize_bucket_as_file/bucket_name_with_no_trailing_slash.json +} metadata { "calls.localizer_workflow.localizer_task.callCaching.allowResultReuse": false "calls.localizer_workflow.localizer_task.callCaching.effectiveCallCachingMode": "CallCachingOff" - "calls.localizer_workflow.localizer_task.callCaching.hashFailures.0.message": "Hash function supposedly succeeded but responded with 'null' instead of a string hash" + "calls.localizer_workflow.localizer_task.callCaching.hashFailures.0.message": "GcsPath 'gs://gcp-public-data-landsat/' is bucket only and does not specify an object blob." "calls.localizer_workflow.localizer_task.callCaching.hit": false } -} diff --git a/centaur/src/main/resources/standardTestCases/bucket_name_with_trailing_slash.test b/centaur/src/main/resources/standardTestCases/bucket_name_with_trailing_slash.test index d640f6d847a..255efdafb15 100644 --- a/centaur/src/main/resources/standardTestCases/bucket_name_with_trailing_slash.test +++ b/centaur/src/main/resources/standardTestCases/bucket_name_with_trailing_slash.test @@ -10,6 +10,6 @@ files { metadata { "calls.localizer_workflow.localizer_task.callCaching.allowResultReuse": false "calls.localizer_workflow.localizer_task.callCaching.effectiveCallCachingMode": "CallCachingOff" - "calls.localizer_workflow.localizer_task.callCaching.hashFailures.0.message": "Hash function supposedly succeeded but responded with 'null' instead of a string hash" + "calls.localizer_workflow.localizer_task.callCaching.hashFailures.0.message": "GcsPath 'gs://gcp-public-data-landsat/' is bucket only and does not specify an object blob." "calls.localizer_workflow.localizer_task.callCaching.hit": false } diff --git a/centaur/src/main/resources/standardTestCases/checkpointing.test b/centaur/src/main/resources/standardTestCases/checkpointing.test new file mode 100644 index 00000000000..84ab392f9d6 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/checkpointing.test @@ -0,0 +1,13 @@ +name: checkpointing +testFormat: workflowsuccess +backends: [Papiv2] + +files { + workflow: checkpointing/checkpointing.wdl +} + +metadata { + workflowName: checkpointing + status: Succeeded + "outputs.checkpointing.preempted": "GOTPREEMPTED" +} diff --git a/centaur/src/main/resources/standardTestCases/checkpointing/checkpointing.wdl b/centaur/src/main/resources/standardTestCases/checkpointing/checkpointing.wdl new file mode 100644 index 00000000000..710dd40f2e2 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/checkpointing/checkpointing.wdl @@ -0,0 +1,69 @@ +version 1.0 + +workflow checkpointing { + call count { input: count_to = 100 } + output { + String preempted = count.preempted + } +} + +task count { + input { + Int count_to + } + + meta { + volatile: true + } + + command <<< + # Read from the my_checkpoint file if there's content there: + FROM_CKPT=$(cat my_checkpoint | tail -n1 | awk '{ print $1 }') + FROM_CKPT=${FROM_CKPT:-1} + + # We don't want any single VM run the entire count, so work out the max counter value for this attempt: + MAX="$(($FROM_CKPT + 66))" + + INSTANCE_NAME=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/name" -H "Metadata-Flavor: Google") + echo "Discovered instance: $INSTANCE_NAME" + + # Run the counter: + echo '--' >> my_checkpoint + for i in $(seq $FROM_CKPT ~{count_to}) + do + echo $i + echo $i ${INSTANCE_NAME} $(date) >> my_checkpoint + + # If we're over our max, simulate "preempting" the VM by killing it: + if [ "${i}" -gt "${MAX}" ] + then + fully_qualified_zone=$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/zone) + zone=$(basename "$fully_qualified_zone") + gcloud compute instances delete ${INSTANCE_NAME} --zone=$zone -q + fi + + sleep 1 + done + + # Prove that we got preempted at least once: + FIRST_INSTANCE=$(cat my_checkpoint | head -n1 | awk '{ print $2 }') + LAST_INSTANCE=$(cat my_checkpoint | tail -n1 | awk '{ print $2 }') + if [ "${FIRST_INSTANCE}" != "LAST_INSTANCE" ] + then + echo "GOTPREEMPTED" > preempted.txt + else + echo "NEVERPREEMPTED" > preempted.txt + fi + >>> + + runtime { + docker: "gcr.io/google.com/cloudsdktool/cloud-sdk:slim" + preemptible: 3 + checkpointFile: "my_checkpoint" + } + + output { + File checkpoint_log = "my_checkpoint" + String preempted = read_string("preempted.txt") + } +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_false.options.json b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_false.options.json new file mode 100644 index 00000000000..0f53721ded6 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_false.options.json @@ -0,0 +1,3 @@ +{ + "use_docker_image_cache": false +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_false.wdl b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_false.wdl new file mode 100644 index 00000000000..49abfdb53e3 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_false.wdl @@ -0,0 +1,31 @@ +version 1.0 + +task check_if_docker_image_cache_disk_mounted { + String google_docker_cache_disk_name = "google-docker-cache" + command { + instance_metadata_disks=$(curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/?recursive=true" -H "Metadata-Flavor: Google") + if [[ "$instance_metadata_disks" == *~{google_docker_cache_disk_name}* ]]; then + echo "true" + else + echo "false" + fi + } + runtime { + docker: "gcr.io/gcp-runtimes/ubuntu_16_0_4:latest" + backend: "Papiv2-Docker-Image-Cache" + useDockerImageCache: false + } + meta { + volatile: true + } + output { + Boolean mounted_docker_image_cache_disk = read_boolean(stdout()) + } +} + +workflow docker_image_cache_false_test { + call check_if_docker_image_cache_disk_mounted + output { + Boolean is_docker_image_cache_disk_mounted = check_if_docker_image_cache_disk_mounted.mounted_docker_image_cache_disk + } +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_true.options.json b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_true.options.json new file mode 100644 index 00000000000..8305ee4aed4 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_true.options.json @@ -0,0 +1,3 @@ +{ + "use_docker_image_cache": true +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_true.wdl b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_true.wdl new file mode 100644 index 00000000000..07e3c50c046 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_true.wdl @@ -0,0 +1,31 @@ +version 1.0 + +task check_if_docker_image_cache_disk_mounted { + String google_docker_cache_disk_name = "google-docker-cache" + command { + instance_metadata_disks=$(curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/?recursive=true" -H "Metadata-Flavor: Google") + if [[ "$instance_metadata_disks" == *~{google_docker_cache_disk_name}* ]]; then + echo "true" + else + echo "false" + fi + } + runtime { + docker: "gcr.io/gcp-runtimes/ubuntu_16_0_4:latest" + backend: "Papiv2-Docker-Image-Cache" + useDockerImageCache: true + } + meta { + volatile: true + } + output { + Boolean mounted_docker_image_cache_disk = read_boolean(stdout()) + } +} + +workflow docker_image_cache_true_test { + call check_if_docker_image_cache_disk_mounted + output { + Boolean is_docker_image_cache_disk_mounted = check_if_docker_image_cache_disk_mounted.mounted_docker_image_cache_disk + } +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_unspecified.options.json b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_unspecified.options.json new file mode 100644 index 00000000000..0967ef424bc --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_unspecified.options.json @@ -0,0 +1 @@ +{} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_unspecified.wdl b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_unspecified.wdl new file mode 100644 index 00000000000..c879e6f0267 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache/docker_image_cache_unspecified.wdl @@ -0,0 +1,30 @@ +version 1.0 + +task check_if_docker_image_cache_disk_mounted { + String google_docker_cache_disk_name = "google-docker-cache" + command { + instance_metadata_disks=$(curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/?recursive=true" -H "Metadata-Flavor: Google") + if [[ "$instance_metadata_disks" == *~{google_docker_cache_disk_name}* ]]; then + echo "true" + else + echo "false" + fi + } + runtime { + docker: "gcr.io/gcp-runtimes/ubuntu_16_0_4:latest" + backend: "Papiv2-Docker-Image-Cache" + } + meta { + volatile: true + } + output { + Boolean mounted_docker_image_cache_disk = read_boolean(stdout()) + } +} + +workflow docker_image_cache_unspecified_test { + call check_if_docker_image_cache_disk_mounted + output { + Boolean is_docker_image_cache_disk_mounted = check_if_docker_image_cache_disk_mounted.mounted_docker_image_cache_disk + } +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_false_false.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_false_false.test new file mode 100644 index 00000000000..62dae357766 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_false_false.test @@ -0,0 +1,14 @@ +name: docker_image_cache_false_false +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_false.wdl + options: docker_image_cache/docker_image_cache_false.options.json +} + +metadata { + workflowName: docker_image_cache_false_test + status: Succeeded + "outputs.docker_image_cache_false_test.is_docker_image_cache_disk_mounted": false +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_false_true.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_false_true.test new file mode 100644 index 00000000000..08b21394281 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_false_true.test @@ -0,0 +1,14 @@ +name: docker_image_cache_false_true +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_false.wdl + options: docker_image_cache/docker_image_cache_true.options.json +} + +metadata { + workflowName: docker_image_cache_false_test + status: Succeeded + "outputs.docker_image_cache_false_test.is_docker_image_cache_disk_mounted": false +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_false_unspecified.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_false_unspecified.test new file mode 100644 index 00000000000..43d8d775ea9 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_false_unspecified.test @@ -0,0 +1,14 @@ +name: docker_image_cache_false_unspecified +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_false.wdl + options: docker_image_cache/docker_image_cache_unspecified.options.json +} + +metadata { + workflowName: docker_image_cache_false_test + status: Succeeded + "outputs.docker_image_cache_false_test.is_docker_image_cache_disk_mounted": false +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_true_false.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_true_false.test new file mode 100644 index 00000000000..233d53b3ea0 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_true_false.test @@ -0,0 +1,14 @@ +name: docker_image_cache_true_false +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_true.wdl + options: docker_image_cache/docker_image_cache_false.options.json +} + +metadata { + workflowName: docker_image_cache_true_test + status: Succeeded + "outputs.docker_image_cache_true_test.is_docker_image_cache_disk_mounted": true +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_true_true.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_true_true.test new file mode 100644 index 00000000000..ffb5bc86e27 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_true_true.test @@ -0,0 +1,14 @@ +name: docker_image_cache_true_true +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_true.wdl + options: docker_image_cache/docker_image_cache_true.options.json +} + +metadata { + workflowName: docker_image_cache_true_test + status: Succeeded + "outputs.docker_image_cache_true_test.is_docker_image_cache_disk_mounted": true +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_true_unspecified.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_true_unspecified.test new file mode 100644 index 00000000000..953ffda3340 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_true_unspecified.test @@ -0,0 +1,14 @@ +name: docker_image_cache_true_unspecified +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_true.wdl + options: docker_image_cache/docker_image_cache_unspecified.options.json +} + +metadata { + workflowName: docker_image_cache_true_test + status: Succeeded + "outputs.docker_image_cache_true_test.is_docker_image_cache_disk_mounted": true +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_false.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_false.test new file mode 100644 index 00000000000..a948841e394 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_false.test @@ -0,0 +1,14 @@ +name: docker_image_cache_unspecified_false +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_unspecified.wdl + options: docker_image_cache/docker_image_cache_false.options.json +} + +metadata { + workflowName: docker_image_cache_unspecified_test + status: Succeeded + "outputs.docker_image_cache_unspecified_test.is_docker_image_cache_disk_mounted": false +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_true.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_true.test new file mode 100644 index 00000000000..4625876612f --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_true.test @@ -0,0 +1,14 @@ +name: docker_image_cache_unspecified_true +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_unspecified.wdl + options: docker_image_cache/docker_image_cache_true.options.json +} + +metadata { + workflowName: docker_image_cache_unspecified_test + status: Succeeded + "outputs.docker_image_cache_unspecified_test.is_docker_image_cache_disk_mounted": true +} diff --git a/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_unspecified.test b/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_unspecified.test new file mode 100644 index 00000000000..279ce60f160 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/docker_image_cache_unspecified_unspecified.test @@ -0,0 +1,14 @@ +name: docker_image_cache_unspecified_unspecified +testFormat: workflowsuccess +backends: [Papiv2-Docker-Image-Cache] + +files { + workflow: docker_image_cache/docker_image_cache_unspecified.wdl + options: docker_image_cache/docker_image_cache_unspecified.options.json +} + +metadata { + workflowName: docker_image_cache_unspecified_test + status: Succeeded + "outputs.docker_image_cache_unspecified_test.is_docker_image_cache_disk_mounted": false +} diff --git a/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.inputs b/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.inputs index a2211bb140f..d5738ac4d17 100644 --- a/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.inputs +++ b/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.inputs @@ -1,5 +1,27 @@ { - # For the below JDR uuid, Martha does not return a service account + # Martha does not return service accounts for JDR paths. Therefore they shouldn't need to be localized using the + # Cromwell custom DOS/DRS localizer. + # + # However, the first file1 was generated before JDR stared saving file names at the end of the gsUri. + # + # Thus the JDR generated gsUri for file1 is: + # 'gs://broad-jade-dev-data-bucket/ca8edd48-e954-4c20-b911-b017fedffb67/585f3f19-985f-43b0-ab6a-79fa4c8310fc' + # With a fileName of: + # 'hello_jade.json' + # + # While the JDR generated gsUri for file2 is: + # 'gs://broad-jade-dev-data-bucket/e1941fb9-6537-4e1a-b70d-34352a3a7817/ad783b60-aeba-4055-8f7b-194880f37259/hello_jade_2.json' + # With a fileName of: + # 'hello_jade_2.json' + # + # The effect of this is that file1 must always be downloaded by the Cromwell custom DOS/DRS localizer which will be + # sure to download the file to a path ending in true file name hello_jade.json. + # + # Meanwhile the gsUri for file2 may be passed directly to specialized command line programs that can read the file + # name and extension from the end of the GCS path. + # "drs_usa_jdr.file1": - "drs://jade.datarepo-dev.broadinstitute.org/v1_f90f5d7f-c507-4e56-abfc-b965a66023fb_585f3f19-985f-43b0-ab6a-79fa4c8310fc" + "drs://jade.datarepo-dev.broadinstitute.org/v1_f90f5d7f-c507-4e56-abfc-b965a66023fb_585f3f19-985f-43b0-ab6a-79fa4c8310fc", + "drs_usa_jdr.file2": + "drs://jade.datarepo-dev.broadinstitute.org/v1_001afc86-da4c-4739-85be-26ca98d2693f_ad783b60-aeba-4055-8f7b-194880f37259" } diff --git a/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.wdl b/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.wdl index b73b3f7fba1..ba2a17f292d 100644 --- a/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.wdl +++ b/centaur/src/main/resources/standardTestCases/drs_tests/drs_usa_jdr.wdl @@ -3,46 +3,61 @@ version 1.0 workflow drs_usa_jdr { input { File file1 + File file2 } call localize_jdr_drs_with_usa { input: - file1 = file1 + file1 = file1, + file2 = file2 } call skip_localize_jdr_drs_with_usa { input: - file1 = file1 + file1 = file1, + file2 = file2 } call read_drs_with_usa { input: - file1 = file1 + file1 = file1, + file2 = file2 } output { String path1 = localize_jdr_drs_with_usa.path1 + String path2 = localize_jdr_drs_with_usa.path2 String hash1 = localize_jdr_drs_with_usa.hash1 + String hash2 = localize_jdr_drs_with_usa.hash2 Float size1 = localize_jdr_drs_with_usa.size1 + Float size2 = localize_jdr_drs_with_usa.size2 String cloud1 = skip_localize_jdr_drs_with_usa.path1 + String cloud2 = skip_localize_jdr_drs_with_usa.path2 Map[String, String] map1 = read_drs_with_usa.map1 + Map[String, String] map2 = read_drs_with_usa.map2 } } task localize_jdr_drs_with_usa { input { File file1 + File file2 } command <<< echo ~{file1} > path1 + echo ~{file2} > path2 md5sum ~{file1} | cut -c1-32 > hash1 + md5sum ~{file2} | cut -c1-32 > hash2 >>> output { String path1 = read_string("path1") + String path2 = read_string("path2") String hash1 = read_string("hash1") + String hash2 = read_string("hash2") Float size1 = size(file1) + Float size2 = size(file2) } runtime { @@ -54,18 +69,22 @@ task localize_jdr_drs_with_usa { task skip_localize_jdr_drs_with_usa { input { File file1 + File file2 } parameter_meta { file1: { localization_optional: true } + file2: { localization_optional: true } } command <<< echo ~{file1} > path1 + echo ~{file2} > path2 >>> output { String path1 = read_string("path1") + String path2 = read_string("path2") } runtime { @@ -77,6 +96,7 @@ task skip_localize_jdr_drs_with_usa { task read_drs_with_usa { input { File file1 + File file2 } command <<< @@ -85,6 +105,7 @@ task read_drs_with_usa { output { Map[String, String] map1 = read_json(file1) + Map[String, String] map2 = read_json(file2) } runtime { diff --git a/centaur/src/main/resources/standardTestCases/drs_usa_hca.test b/centaur/src/main/resources/standardTestCases/drs_usa_hca.test index 6d4eaa8a3e4..3d9ddf26c6d 100644 --- a/centaur/src/main/resources/standardTestCases/drs_usa_hca.test +++ b/centaur/src/main/resources/standardTestCases/drs_usa_hca.test @@ -1,3 +1,4 @@ +ignore: true # See https://broadworkbench.atlassian.net/browse/BW-467 name: drs_usa_hca testFormat: WorkflowSuccess backends: ["papi-v2-usa"] @@ -35,14 +36,15 @@ metadata { "outputs.drs_usa_hca.size3" = 15419927 "outputs.drs_usa_hca.size4" = 1.849607E+7 "outputs.drs_usa_hca.size5" = 2138914697 + # These files must be localized by the DRS localizer, so their paths are NOT cloud paths "outputs.drs_usa_hca.cloud1" = - "gs://org-hca-dss-checkout-prod/blobs/160fde1559f7154b03a6f645b4c7ff0eb2af37241e2cab3961e7780ead93860a.b0fcf2baaadb4aa6545804998867eff29330762a.d18ef9b8fd14ac922588baeec4853c0d.0ba92b16" + "/cromwell_root/drs.data.humancellatlas.org/4cf48dbf-cf09-452e-bb5b-fd016af0c747_version_2019-09-14T024754.281908Z/eb32bfc6-e7be-4093-8959-b8bf27f2404f.zarr!.zattrs" "outputs.drs_usa_hca.cloud2" = - "gs://org-hca-dss-checkout-prod/blobs/2972c63e8230550e198d43df4e2fc3065e2a7b0b8cd683686d9da192ca3b6a00.921ffb13f1642ecdb7e2705d23cb1fdd55730fc6.a75d7e4302ce4685470eb2da97945ce1.3fb63ee7" + "/cromwell_root/drs.data.humancellatlas.org/fe6729a0-9d24-4034-a0e8-720830044af2_version_2019-09-14T075241.272244Z/empty_drops_result.csv" "outputs.drs_usa_hca.cloud3" = - "gs://org-hca-dss-checkout-prod/blobs/cf84e70f7775555a4a3dce44c988be56eee91df57ac5e2cffd0afea7670ad444.e24a504147ec249d9cf092f4d625fdc20ea7d9e5.4a44a0f9ca3a801332070e3a5e3e8e7d.a4608468" + "/cromwell_root/drs.data.humancellatlas.org/82544678-fb65-4da4-aaf8-3a2506b64993_version_2019-09-14T080840.009338Z/empty_drops_result.csv" "outputs.drs_usa_hca.cloud4" = - "gs://org-hca-dss-checkout-prod/blobs/562d452106477d94f0da48826d0b73c2c1bb0123554a6a0f832728baa62f4fd1.4b0bc495ccca61ede14708c60ee3827973004f0e.d4716b00e39809505e45d7a4605a6fb8.6b4e23c0" + "/cromwell_root/drs.data.humancellatlas.org/d205fbd5-0b2d-4ec9-a898-ee08712e9c18_version_2019-09-14T024945.734484Z/empty_drops_result.csv" "outputs.drs_usa_hca.cloud5" = - "gs://org-hca-dss-checkout-prod/blobs/979ff8bf675f4a3d56bb83e80e64641656bbfd1b7bfc805023445cf53fdd5450.05b544255eccdacdfe433569a0b28826cb3e18c1.ad7a88538e8a9c00b6d904c7eb00318c-32.f549d920" + "/cromwell_root/drs.data.humancellatlas.org/39ae3998-72a4-41ba-b00f-526bfecb8ca2_version_2019-09-13T091626.837580Z/Retina_Wong_scRNA_Sample1_I1.fastq.gz" } diff --git a/centaur/src/main/resources/standardTestCases/drs_usa_jdr.test b/centaur/src/main/resources/standardTestCases/drs_usa_jdr.test index 093d5f63e3f..c6d523f3937 100644 --- a/centaur/src/main/resources/standardTestCases/drs_usa_jdr.test +++ b/centaur/src/main/resources/standardTestCases/drs_usa_jdr.test @@ -17,9 +17,19 @@ metadata { "outputs.drs_usa_jdr.path1" = "/cromwell_root/jade.datarepo-dev.broadinstitute.org/v1_f90f5d7f-c507-4e56-abfc-b965a66023fb_585f3f19-985f-43b0-ab6a-79fa4c8310fc/hello_jade.json" + "outputs.drs_usa_jdr.path2" = + "/cromwell_root/jade.datarepo-dev.broadinstitute.org/v1_001afc86-da4c-4739-85be-26ca98d2693f_ad783b60-aeba-4055-8f7b-194880f37259/hello_jade_2.json" "outputs.drs_usa_jdr.hash1" = "faf12e94c25bef7df62e4a5eb62573f5" + "outputs.drs_usa_jdr.hash2" = "19e1b021628130fda04c79ee9a056b67" "outputs.drs_usa_jdr.size1" = 18.0 + "outputs.drs_usa_jdr.size2" = 38.0 + # This JDR file has a gsUri that doesn't end in /fileName so it must be downloaded with the DRS localizer "outputs.drs_usa_jdr.cloud1" = - "gs://broad-jade-dev-data-bucket/ca8edd48-e954-4c20-b911-b017fedffb67/585f3f19-985f-43b0-ab6a-79fa4c8310fc" + "/cromwell_root/jade.datarepo-dev.broadinstitute.org/v1_f90f5d7f-c507-4e56-abfc-b965a66023fb_585f3f19-985f-43b0-ab6a-79fa4c8310fc/hello_jade.json" + # This JDR file has a gsUri that can skip localization + "outputs.drs_usa_jdr.cloud2" = + "gs://broad-jade-dev-data-bucket/e1941fb9-6537-4e1a-b70d-34352a3a7817/ad783b60-aeba-4055-8f7b-194880f37259/hello_jade_2.json" "outputs.drs_usa_jdr.map1.hello" = "jade" + "outputs.drs_usa_jdr.map2.hello" = "jade" + "outputs.drs_usa_jdr.map2.attempt" = "2" } diff --git a/centaur/src/main/resources/standardTestCases/drs_usa_jdr_preresolve.test b/centaur/src/main/resources/standardTestCases/drs_usa_jdr_preresolve.test new file mode 100644 index 00000000000..4d6cfd57936 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/drs_usa_jdr_preresolve.test @@ -0,0 +1,35 @@ +name: drs_usa_jdr_preresolve +testFormat: WorkflowSuccess +backends: ["papi-v2-usa"] +skipDescribeEndpointValidation: true + +files { + workflow: drs_tests/drs_usa_jdr.wdl + options-dir: "Error: BA-6546 The environment variable CROMWELL_BUILD_RESOURCES_DIRECTORY must be set/export pointing to a valid path such as '${YOUR_CROMWELL_DIR}/target/ci/resources'" + options-dir: ${?CROMWELL_BUILD_RESOURCES_DIRECTORY} + options: ${files.options-dir}/papi_v2_usa_preresolve.options.json + inputs: drs_tests/drs_usa_jdr.inputs +} + +metadata { + workflowName: drs_usa_jdr + status: Succeeded + + "outputs.drs_usa_jdr.path1" = + "/cromwell_root/jade.datarepo-dev.broadinstitute.org/v1_f90f5d7f-c507-4e56-abfc-b965a66023fb_585f3f19-985f-43b0-ab6a-79fa4c8310fc/hello_jade.json" + "outputs.drs_usa_jdr.path2" = + "/cromwell_root/broad-jade-dev-data-bucket/e1941fb9-6537-4e1a-b70d-34352a3a7817/ad783b60-aeba-4055-8f7b-194880f37259/hello_jade_2.json" + "outputs.drs_usa_jdr.hash1" = "faf12e94c25bef7df62e4a5eb62573f5" + "outputs.drs_usa_jdr.hash2" = "19e1b021628130fda04c79ee9a056b67" + "outputs.drs_usa_jdr.size1" = 18.0 + "outputs.drs_usa_jdr.size2" = 38.0 + # This JDR file has a gsUri that doesn't end in /fileName so it must be downloaded with the DRS localizer + "outputs.drs_usa_jdr.cloud1" = + "/cromwell_root/jade.datarepo-dev.broadinstitute.org/v1_f90f5d7f-c507-4e56-abfc-b965a66023fb_585f3f19-985f-43b0-ab6a-79fa4c8310fc/hello_jade.json" + # This JDR file has a gsUri that can skip localization + "outputs.drs_usa_jdr.cloud2" = + "gs://broad-jade-dev-data-bucket/e1941fb9-6537-4e1a-b70d-34352a3a7817/ad783b60-aeba-4055-8f7b-194880f37259/hello_jade_2.json" + "outputs.drs_usa_jdr.map1.hello" = "jade" + "outputs.drs_usa_jdr.map2.hello" = "jade" + "outputs.drs_usa_jdr.map2.attempt" = "2" +} diff --git a/centaur/src/main/resources/standardTestCases/dummy_scatter.test b/centaur/src/main/resources/standardTestCases/dummy_scatter.test new file mode 100644 index 00000000000..5b61412a013 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/dummy_scatter.test @@ -0,0 +1,14 @@ +name: dummy_scatter +testFormat: workflowsuccessandtimedoutputs +backends: [Dummy] +tags: [Dummy] + +files { + workflow: dummy_scatter/dummy_scatter.wdl +} + +metadata { + "outputs.dummy_scatter.results_count": 35000 +} + +maximumTime = 12 minutes diff --git a/centaur/src/main/resources/standardTestCases/dummy_scatter/dummy_scatter.wdl b/centaur/src/main/resources/standardTestCases/dummy_scatter/dummy_scatter.wdl new file mode 100644 index 00000000000..78bc8c05d7d --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/dummy_scatter/dummy_scatter.wdl @@ -0,0 +1,24 @@ +version 1.0 + +workflow dummy_scatter { + scatter (x in range(35000)) { + call dummy_scattered_task + } + output { + Int results_count = length(dummy_scattered_task.string_out) + } +} + +task dummy_scattered_task { + command { + echo hello + } + output { + String string_out = "hello" + } + runtime { + # This is technically unnecessary given the test setup, but I want to double check this isn't accidentally sent + # to a _real_, money-spending backend: + backend: "Dummy" + } +} diff --git a/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification.test b/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification.test new file mode 100644 index 00000000000..7d6e8c93949 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification.test @@ -0,0 +1,12 @@ +name: invalid_use_reference_disks_specification +testFormat: workflowfailure +backends: [ Papiv2 ] + +files { + workflow: invalid_use_reference_disks_specification/invalid_use_reference_disks_specification.wdl + options: invalid_use_reference_disks_specification/invalid_use_reference_disks_specification.options.json +} + +metadata { + "failures.0.causedBy.0.message": "'use_reference_disks' is specified in workflow options but value is not of expected Boolean type: Unsupported JsValue as JsBoolean: \"I like turtles\"" +} diff --git a/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification/invalid_use_reference_disks_specification.options.json b/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification/invalid_use_reference_disks_specification.options.json new file mode 100644 index 00000000000..cb4f05ccb69 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification/invalid_use_reference_disks_specification.options.json @@ -0,0 +1,3 @@ +{ + "use_reference_disks": "I like turtles" +} diff --git a/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification/invalid_use_reference_disks_specification.wdl b/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification/invalid_use_reference_disks_specification.wdl new file mode 100644 index 00000000000..c2247991072 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/invalid_use_reference_disks_specification/invalid_use_reference_disks_specification.wdl @@ -0,0 +1,9 @@ +task hello { + command { + echo "Hello world" + } +} + +workflow invalid_use_reference_disks_specification { + call hello +} diff --git a/centaur/src/main/resources/standardTestCases/read_write_json/read_write_json_roundtrip.wdl b/centaur/src/main/resources/standardTestCases/read_write_json/read_write_json_roundtrip.wdl new file mode 100644 index 00000000000..aee44a13699 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/read_write_json/read_write_json_roundtrip.wdl @@ -0,0 +1,29 @@ +version 1.0 + +workflow read_write_json_roundtrip { + input { + Array[Int] indices = [0, 1, 2] + Map[String, String] map_input = {"genre": "comedy", "movie": "mr. bean"} + Pair[Int, String] pair_input = (1, "abc") + Array[Map[String, String]] map_array = [{"artist": "maroon 5", "song": "memories"}] + } + + scatter (i in indices) { + Object object_array = object {index: i, name: "mr_bean"} + Object object_with_array = object {index: i, samples: ["s1", "s2"]} + } + + output{ + Boolean boolean_out = read_json(write_json(false)) + Int int_out = read_json(write_json(1234)) + Float float_out = read_json(write_json(123.456)) + String string_out = read_json(write_json("Mr. Bean")) + Array[Int] array_out = read_json(write_json(indices)) + Object map_out = read_json(write_json(map_input)) + Object pair_out = read_json(write_json(pair_input)) + Object object_out = read_json(write_json(object_array[0])) + Array[Object] array_object_out = read_json(write_json(object_array)) + Array[Object] array_object_with_array_out = read_json(write_json(object_with_array)) + Array[Map[String, String]] object_out_as_map = read_json(write_json(map_array)) + } +} diff --git a/centaur/src/main/resources/standardTestCases/read_write_json/read_write_json_roundtrip_develop.wdl b/centaur/src/main/resources/standardTestCases/read_write_json/read_write_json_roundtrip_develop.wdl new file mode 100644 index 00000000000..bab8556fa83 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/read_write_json/read_write_json_roundtrip_develop.wdl @@ -0,0 +1,44 @@ +version development + +struct Random { + Int index + String name +} + +struct Another_Random { + Int index + Array[String] samples +} + +struct Pair_Object { + String left + String right +} + +workflow read_write_json_roundtrip_develop { + input { + Array[Int] indices = [0, 1, 2] + Map[String, String] map_input = {"genre": "comedy", "movie": "mr. bean"} + Pair[Int, String] pair_input = (1, "abc") + Array[Map[String, String]] map_array = [{"artist": "maroon 5", "song": "memories"}] + } + + scatter (i in indices) { + Random object_array = object {index: i, name: "mr_bean"} + Another_Random object_with_array = object {index: i, samples: ["s1", "s2"]} + } + + output{ + Boolean boolean_out = read_json(write_json(false)) + Int int_out = read_json(write_json(1234)) + Float float_out = read_json(write_json(123.456)) + String string_out = read_json(write_json("Mr. Bean")) + Array[Int] array_out = read_json(write_json(indices)) + Map[String, String] map_out = read_json(write_json(map_input)) + Pair_Object pair_out = read_json(write_json(pair_input)) + Random object_out = read_json(write_json(object_array[0])) + Array[Random] array_object_out = read_json(write_json(object_array)) + Array[Another_Random] array_object_with_array_out = read_json(write_json(object_with_array)) + Array[Map[String, String]] object_out_as_map = read_json(write_json(map_array)) + } +} diff --git a/centaur/src/main/resources/standardTestCases/read_write_json_roundtrip.test b/centaur/src/main/resources/standardTestCases/read_write_json_roundtrip.test new file mode 100644 index 00000000000..a947f79de82 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/read_write_json_roundtrip.test @@ -0,0 +1,34 @@ +name: read_write_json_roundtrip +testFormat: workflowsuccess + +files { + workflow: read_write_json/read_write_json_roundtrip.wdl +} + +metadata { + workflowName: read_write_json_roundtrip + status: Succeeded + "outputs.read_write_json_roundtrip.int_out" = 1234 + "outputs.read_write_json_roundtrip.float_out" = 123.456 + "outputs.read_write_json_roundtrip.boolean_out" = false + "outputs.read_write_json_roundtrip.string_out" = "Mr. Bean" + "outputs.read_write_json_roundtrip.pair_out.left" = 1 + "outputs.read_write_json_roundtrip.pair_out.right" = "abc" + "outputs.read_write_json_roundtrip.array_object_with_array_out.0.index" = 0 + "outputs.read_write_json_roundtrip.array_object_with_array_out.0.samples.0" = "s1" + "outputs.read_write_json_roundtrip.array_object_with_array_out.0.samples.1" = "s2" + "outputs.read_write_json_roundtrip.array_object_with_array_out.1.index" = 1 + "outputs.read_write_json_roundtrip.array_out.0" = 0 + "outputs.read_write_json_roundtrip.array_out.1" = 1 + "outputs.read_write_json_roundtrip.array_out.2" = 2 + "outputs.read_write_json_roundtrip.map_out.genre" = "comedy" + "outputs.read_write_json_roundtrip.map_out.movie" = "mr. bean" + "outputs.read_write_json_roundtrip.array_object_out.0.index" = 0 + "outputs.read_write_json_roundtrip.array_object_out.0.name" = "mr_bean" + "outputs.read_write_json_roundtrip.array_object_out.2.index" = 2 + "outputs.read_write_json_roundtrip.array_object_out.2.name" = "mr_bean" + "outputs.read_write_json_roundtrip.object_out.index" = 0 + "outputs.read_write_json_roundtrip.object_out.name" = "mr_bean" + "outputs.read_write_json_roundtrip.object_out_as_map.0.artist" = "maroon 5" + "outputs.read_write_json_roundtrip.object_out_as_map.0.song" = "memories" +} diff --git a/centaur/src/main/resources/standardTestCases/read_write_json_roundtrip_develop.test b/centaur/src/main/resources/standardTestCases/read_write_json_roundtrip_develop.test new file mode 100644 index 00000000000..e6360ede893 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/read_write_json_roundtrip_develop.test @@ -0,0 +1,34 @@ +name: read_write_json_roundtrip_develop +testFormat: workflowsuccess + +files { + workflow: read_write_json/read_write_json_roundtrip_develop.wdl +} + +metadata { + workflowName: read_write_json_roundtrip_develop + status: Succeeded + "outputs.read_write_json_roundtrip_develop.int_out" = 1234 + "outputs.read_write_json_roundtrip_develop.float_out" = 123.456 + "outputs.read_write_json_roundtrip_develop.boolean_out" = false + "outputs.read_write_json_roundtrip_develop.string_out" = "Mr. Bean" + "outputs.read_write_json_roundtrip_develop.pair_out.left" = 1 + "outputs.read_write_json_roundtrip_develop.pair_out.right" = "abc" + "outputs.read_write_json_roundtrip_develop.array_object_with_array_out.0.index" = 0 + "outputs.read_write_json_roundtrip_develop.array_object_with_array_out.0.samples.0" = "s1" + "outputs.read_write_json_roundtrip_develop.array_object_with_array_out.0.samples.1" = "s2" + "outputs.read_write_json_roundtrip_develop.array_object_with_array_out.1.index" = 1 + "outputs.read_write_json_roundtrip_develop.array_out.0" = 0 + "outputs.read_write_json_roundtrip_develop.array_out.1" = 1 + "outputs.read_write_json_roundtrip_develop.array_out.2" = 2 + "outputs.read_write_json_roundtrip_develop.map_out.genre" = "comedy" + "outputs.read_write_json_roundtrip_develop.map_out.movie" = "mr. bean" + "outputs.read_write_json_roundtrip_develop.array_object_out.0.index" = 0 + "outputs.read_write_json_roundtrip_develop.array_object_out.0.name" = "mr_bean" + "outputs.read_write_json_roundtrip_develop.array_object_out.2.index" = 2 + "outputs.read_write_json_roundtrip_develop.array_object_out.2.name" = "mr_bean" + "outputs.read_write_json_roundtrip_develop.object_out.index" = 0 + "outputs.read_write_json_roundtrip_develop.object_out.name" = "mr_bean" + "outputs.read_write_json_roundtrip_develop.object_out_as_map.0.artist" = "maroon 5" + "outputs.read_write_json_roundtrip_develop.object_out_as_map.0.song" = "memories" +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_false.options.json b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_false.options.json new file mode 100644 index 00000000000..fab0f88b724 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_false.options.json @@ -0,0 +1,4 @@ +{ + "read_from_cache": false, + "use_reference_disks": false +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_true.options.json b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_true.options.json new file mode 100644 index 00000000000..4430dfddaaa --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_true.options.json @@ -0,0 +1,4 @@ +{ + "read_from_cache": false, + "use_reference_disks": true +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_unspecified.options.json b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_unspecified.options.json new file mode 100644 index 00000000000..1840c33cd39 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk/reference_disk_test_unspecified.options.json @@ -0,0 +1,3 @@ +{ + "read_from_cache": false +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_false_options.test b/centaur/src/main/resources/standardTestCases/reference_disk_false_options.test new file mode 100644 index 00000000000..b75ef58dabc --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk_false_options.test @@ -0,0 +1,15 @@ +name: reference_disk_test_false_option +testFormat: workflowsuccess +backends: [Papiv2-Reference-Disk-Localization] + +files { + workflow: reference_disk/reference_disk_test.wdl + inputs: reference_disk/reference_disk_test.inputs + options: reference_disk/reference_disk_test_false.options.json +} + +metadata { + workflowName: wf_reference_disk_test + status: Succeeded + "outputs.wf_reference_disk_test.is_input_file_a_symlink": false +} diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_test.test b/centaur/src/main/resources/standardTestCases/reference_disk_true_options.test similarity index 76% rename from centaur/src/main/resources/standardTestCases/reference_disk_test.test rename to centaur/src/main/resources/standardTestCases/reference_disk_true_options.test index 47c37734bd2..dab9c516164 100644 --- a/centaur/src/main/resources/standardTestCases/reference_disk_test.test +++ b/centaur/src/main/resources/standardTestCases/reference_disk_true_options.test @@ -1,10 +1,11 @@ -name: reference_disk_test +name: reference_disk_test_true_option testFormat: workflowsuccess backends: [Papiv2-Reference-Disk-Localization] files { workflow: reference_disk/reference_disk_test.wdl inputs: reference_disk/reference_disk_test.inputs + options: reference_disk/reference_disk_test_true.options.json } metadata { diff --git a/centaur/src/main/resources/standardTestCases/reference_disk_unspecified_options.test b/centaur/src/main/resources/standardTestCases/reference_disk_unspecified_options.test new file mode 100644 index 00000000000..441db8748f2 --- /dev/null +++ b/centaur/src/main/resources/standardTestCases/reference_disk_unspecified_options.test @@ -0,0 +1,15 @@ +name: reference_disk_test_unspecified_option +testFormat: workflowsuccess +backends: [Papiv2-Reference-Disk-Localization] + +files { + workflow: reference_disk/reference_disk_test.wdl + inputs: reference_disk/reference_disk_test.inputs + options: reference_disk/reference_disk_test_unspecified.options.json +} + +metadata { + workflowName: wf_reference_disk_test + status: Succeeded + "outputs.wf_reference_disk_test.is_input_file_a_symlink": false +} diff --git a/centaur/src/main/resources/standardTestCases/write_json.test b/centaur/src/main/resources/standardTestCases/write_json.test deleted file mode 100644 index 0e2289d754b..00000000000 --- a/centaur/src/main/resources/standardTestCases/write_json.test +++ /dev/null @@ -1,20 +0,0 @@ -name: write_json -testFormat: workflowsuccess - -files { - workflow: write_json/write_json.wdl -} - -metadata { - workflowName: write_json - status: Succeeded - "outputs.write_json.boolean_out" = false - "outputs.write_json.string_out" = "\"Mr. Bean\"" - "outputs.write_json.int_out" = 1234 - "outputs.write_json.float_out" = 123.456 - "outputs.write_json.array_out" = "[0,1,2]" - "outputs.write_json.map_out" = "{\"genre\":\"comedy\",\"movie\":\"mr. bean\"}" - "outputs.write_json.pair_out" = "{\"left\":1,\"right\":\"abc\"}" - "outputs.write_json.object_out" = "{\"index\":0,\"name\":\"mr_bean\"}" - "outputs.write_json.array_object_out" = "[{\"index\":0,\"name\":\"mr_bean\"},{\"index\":1,\"name\":\"mr_bean\"},{\"index\":2,\"name\":\"mr_bean\"}]" -} diff --git a/centaur/src/main/resources/standardTestCases/write_json/write_json.wdl b/centaur/src/main/resources/standardTestCases/write_json/write_json.wdl deleted file mode 100644 index 49619535772..00000000000 --- a/centaur/src/main/resources/standardTestCases/write_json/write_json.wdl +++ /dev/null @@ -1,39 +0,0 @@ -version 1.0 - -workflow write_json { - input { - Array[Int] indices = [0, 1, 2] - Map[String, String] map_input = {"genre": "comedy", "movie": "mr. bean"} - Pair[Int, String] pair_input = (1, "abc") - } - scatter (i in indices) { - call create_single_object { input: i=i } - } - - output{ - Boolean boolean_out = read_boolean(write_json(false)) - Int int_out = read_int(write_json(1234)) - Float float_out = read_float(write_json(123.456)) - String string_out = read_string(write_json("Mr. Bean")) - String array_out = read_string(write_json(indices)) - String map_out = read_string(write_json(map_input)) - String pair_out = read_string(write_json(pair_input)) - String object_out = read_string(write_json(create_single_object.out[0])) - String array_object_out = read_string(write_json(create_single_object.out)) - } -} - -task create_single_object { - input { - Int i - } - command { - echo "Creating single object" - } - output { - Object out = object {index: i, name: "mr_bean"} - } - runtime { - docker: "ubuntu:latest" - } -} diff --git a/centaur/src/main/resources/standardTestCases/write_json/write_json_version_develop.wdl b/centaur/src/main/resources/standardTestCases/write_json/write_json_version_develop.wdl deleted file mode 100644 index d2156bb0f4b..00000000000 --- a/centaur/src/main/resources/standardTestCases/write_json/write_json_version_develop.wdl +++ /dev/null @@ -1,44 +0,0 @@ -version development - -struct Random { - Int index - String name -} - -workflow write_json_version_develop { - input { - Array[Int] indices = [0, 1, 2] - Map[String, String] map_input = {"genre": "comedy", "movie": "mr. bean"} - Pair[Int, String] pair_input = (1, "abc") - } - scatter (i in indices) { - call create_single_object { input: i=i } - } - - output{ - Boolean boolean_out = read_boolean(write_json(false)) - Int int_out = read_int(write_json(1234)) - Float float_out = read_float(write_json(123.456)) - String string_out = read_string(write_json("Mr. Bean")) - String array_out = read_string(write_json(indices)) - String map_out = read_string(write_json(map_input)) - String pair_out = read_string(write_json(pair_input)) - String object_out = read_string(write_json(create_single_object.out[0])) - String array_object_out = read_string(write_json(create_single_object.out)) - } -} - -task create_single_object { - input { - Int i - } - command { - echo "Creating single object" - } - output { - Random out = object {index: i, name: "mr_bean"} - } - runtime { - docker: "ubuntu:latest" - } -} diff --git a/centaur/src/main/resources/standardTestCases/write_json_version_develop.test b/centaur/src/main/resources/standardTestCases/write_json_version_develop.test deleted file mode 100644 index 33ff38d7bce..00000000000 --- a/centaur/src/main/resources/standardTestCases/write_json_version_develop.test +++ /dev/null @@ -1,20 +0,0 @@ -name: write_json_version_develop -testFormat: workflowsuccess - -files { - workflow: write_json/write_json_version_develop.wdl -} - -metadata { - workflowName: write_json_version_develop - status: Succeeded - "outputs.write_json_version_develop.boolean_out" = false - "outputs.write_json_version_develop.string_out" = "\"Mr. Bean\"" - "outputs.write_json_version_develop.int_out" = 1234 - "outputs.write_json_version_develop.float_out" = 123.456 - "outputs.write_json_version_develop.array_out" = "[0,1,2]" - "outputs.write_json_version_develop.map_out" = "{\"genre\":\"comedy\",\"movie\":\"mr. bean\"}" - "outputs.write_json_version_develop.pair_out" = "{\"left\":1,\"right\":\"abc\"}" - "outputs.write_json_version_develop.object_out" = "{\"index\":0,\"name\":\"mr_bean\"}" - "outputs.write_json_version_develop.array_object_out" = "[{\"index\":0,\"name\":\"mr_bean\"},{\"index\":1,\"name\":\"mr_bean\"},{\"index\":2,\"name\":\"mr_bean\"}]" -} diff --git a/centaur/src/main/scala/centaur/test/Test.scala b/centaur/src/main/scala/centaur/test/Test.scala index 3919482b2d9..5d95a346533 100644 --- a/centaur/src/main/scala/centaur/test/Test.scala +++ b/centaur/src/main/scala/centaur/test/Test.scala @@ -170,6 +170,27 @@ object Operations extends StrictLogging { } yield () } + def checkTimingRequirement(timeRequirement: Option[FiniteDuration]): Test[FiniteDuration] = new Test[FiniteDuration] { + override def run: IO[FiniteDuration] = timeRequirement match { + case Some(duration) => IO.pure(duration) + case None => IO.raiseError(new Exception("Duration value for 'maximumTime' required but not supplied in test config")) + } + } + + def checkFastEnough(before: Long, after: Long, allowance: FiniteDuration): Test[Unit] = new Test[Unit] { + override def run: IO[Unit] = { + if (after - before < allowance.toSeconds) IO.pure(()) + else IO.raiseError(new Exception(s"Test took too long. Allowance was $allowance. Actual time: ${after - before}")) + } + } + + def timingVerificationNotSupported(timingRequirement: Option[FiniteDuration]): Test[Unit] = new Test[Unit] { + override def run: IO[Unit] = if (timingRequirement.isDefined) { + IO.raiseError(new Exception("Maximum workflow time verification is not supported in this test mode")) + } else IO.pure(()) + + } + def checkDescription(workflow: Workflow, validityExpectation: Option[Boolean], retries: Int = 3): Test[Unit] = { new Test[Unit] { @@ -593,7 +614,8 @@ object Operations extends StrictLogging { val message = s"In actual outputs but not in expected and other outputs not allowed: ${inActualButNotInExpected.mkString(", ")}" IO.raiseError(CentaurTestException(message, workflow, submittedWorkflow)) } else if (inExpectedButNotInActual.nonEmpty) { - val message = s"In expected outputs but not in actual: ${inExpectedButNotInActual.mkString(", ")}" + val message = s"In actual outputs but not in expected: ${inExpectedButNotInActual.mkString(", ")}" + System.lineSeparator + + s"In expected outputs but not in actual: ${inExpectedButNotInActual.mkString(", ")}" IO.raiseError(CentaurTestException(message, workflow, submittedWorkflow)) } else { IO.unit diff --git a/centaur/src/main/scala/centaur/test/formulas/TestFormulas.scala b/centaur/src/main/scala/centaur/test/formulas/TestFormulas.scala index fbf84b104ff..a646e2c3b0a 100644 --- a/centaur/src/main/scala/centaur/test/formulas/TestFormulas.scala +++ b/centaur/src/main/scala/centaur/test/formulas/TestFormulas.scala @@ -1,5 +1,7 @@ package centaur.test.formulas +import java.time.OffsetDateTime + import cats.syntax.flatMap._ import cats.syntax.functor._ import centaur.api.CentaurCromwellClient @@ -39,8 +41,19 @@ object TestFormulas extends StrictLogging { private def runSuccessfulWorkflow(workflow: Workflow): Test[SubmittedWorkflow] = runWorkflowUntilTerminalStatus(workflow, Succeeded) private def runFailingWorkflow(workflow: Workflow): Test[SubmittedWorkflow] = runWorkflowUntilTerminalStatus(workflow, Failed) + def runSuccessfulWorkflowAndVerifyTimeAndOutputs(workflowDefinition: Workflow): Test[SubmitResponse] = for { + _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + timeAllowance <- checkTimingRequirement(workflowDefinition.maximumAllowedTime) + beforeTimestamp = OffsetDateTime.now().toInstant.getEpochSecond + submittedWorkflow <- runSuccessfulWorkflow(workflowDefinition) + afterTimestamp = OffsetDateTime.now().toInstant.getEpochSecond + _ <- fetchAndValidateOutputs(submittedWorkflow, workflowDefinition, "ROOT NOT SUPPORTED IN TIMING/OUTPUT ONLY TESTS", validateArchived = Option(false)) + _ <- checkFastEnough(beforeTimestamp, afterTimestamp, timeAllowance) + } yield SubmitResponse(submittedWorkflow) + def runSuccessfulWorkflowAndVerifyMetadata(workflowDefinition: Workflow)(implicit cromwellTracker: Option[CromwellTracker]): Test[SubmitResponse] = for { _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) submittedWorkflow <- runSuccessfulWorkflow(workflowDefinition) labelsLikelyBeforeArchival = CentaurCromwellClient.labels(submittedWorkflow) unarchivedNonSubworkflowMetadata <- fetchAndValidateNonSubworkflowMetadata(submittedWorkflow, workflowDefinition, validateArchived = Option(false)) @@ -84,6 +97,7 @@ object TestFormulas extends StrictLogging { def runFailingWorkflowAndVerifyMetadata(workflowDefinition: Workflow)(implicit cromwellTracker: Option[CromwellTracker]): Test[SubmitResponse] = for { _ <- checkDescription(workflowDefinition, validityExpectation = None) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) submittedWorkflow <- runFailingWorkflow(workflowDefinition) labelsLikelyBeforeArchival = CentaurCromwellClient.labels(submittedWorkflow) unarchivedNonSubworkflowMetadata <- fetchAndValidateNonSubworkflowMetadata(submittedWorkflow, workflowDefinition, validateArchived = Option(false)) @@ -112,6 +126,7 @@ object TestFormulas extends StrictLogging { def runWorkflowTwiceExpectingCaching(workflowDefinition: Workflow)(implicit cromwellTracker: Option[CromwellTracker]): Test[SubmitResponse] = { for { _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) firstWF <- runSuccessfulWorkflow(workflowDefinition) secondWf <- runSuccessfulWorkflow(workflowDefinition.secondRun) _ <- printHashDifferential(firstWF, secondWf) @@ -126,6 +141,7 @@ object TestFormulas extends StrictLogging { def runWorkflowThriceExpectingCaching(workflowDefinition: Workflow)(implicit cromwellTracker: Option[CromwellTracker]): Test[SubmitResponse] = { for { _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) firstWf <- runSuccessfulWorkflow(workflowDefinition) secondWf <- runSuccessfulWorkflow(workflowDefinition.secondRun) metadataTwo <- fetchAndValidateNonSubworkflowMetadata(secondWf, workflowDefinition, Option(firstWf.id.id)) @@ -142,6 +158,7 @@ object TestFormulas extends StrictLogging { def runWorkflowTwiceExpectingNoCaching(workflowDefinition: Workflow)(implicit cromwellTracker: Option[CromwellTracker]): Test[SubmitResponse] = { for { _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) _ <- runSuccessfulWorkflow(workflowDefinition) // Build caches testWf <- runSuccessfulWorkflow(workflowDefinition.secondRun) metadata <- fetchAndValidateNonSubworkflowMetadata(testWf, workflowDefinition) @@ -155,6 +172,7 @@ object TestFormulas extends StrictLogging { def runFailingWorkflowTwiceExpectingNoCaching(workflowDefinition: Workflow)(implicit cromwellTracker: Option[CromwellTracker]): Test[SubmitResponse] = { for { _ <- checkDescription(workflowDefinition, validityExpectation = None) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) _ <- runFailingWorkflow(workflowDefinition) // Build caches testWf <- runFailingWorkflow(workflowDefinition) metadata <- fetchAndValidateNonSubworkflowMetadata(testWf, workflowDefinition) @@ -174,6 +192,7 @@ object TestFormulas extends StrictLogging { case ManagedCromwellServer(_, postRestart, withRestart) if withRestart => for { _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) submittedWorkflow <- submitWorkflow(workflowDefinition) jobId <- pollUntilCallIsRunning(workflowDefinition, submittedWorkflow, callMarker.callKey) _ = CromwellManager.stopCromwell(s"Scheduled restart from ${workflowDefinition.testName}") @@ -199,6 +218,7 @@ object TestFormulas extends StrictLogging { def instantAbort(workflowDefinition: Workflow)(implicit cromwellTracker: Option[CromwellTracker]): Test[SubmitResponse] = for { _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) submittedWorkflow <- submitWorkflow(workflowDefinition) _ <- abortWorkflow(submittedWorkflow) _ <- expectSomeProgress(submittedWorkflow, workflowDefinition, Set(Running, Aborting, Aborted), workflowProgressTimeout) @@ -219,6 +239,7 @@ object TestFormulas extends StrictLogging { for { _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) submittedWorkflow <- submitWorkflow(workflowDefinition) jobId <- pollUntilCallIsRunning(workflowDefinition, submittedWorkflow, callMarker.callKey) // The Cromwell call status could be running but the backend job might not have started yet, give it some time @@ -248,6 +269,7 @@ object TestFormulas extends StrictLogging { def submitInvalidWorkflow(workflow: Workflow, expectedSubmitResponse: SubmitHttpResponse): Test[SubmitResponse] = { for { _ <- checkDescription(workflow, validityExpectation = None) + _ <- timingVerificationNotSupported(workflow.maximumAllowedTime) actualSubmitResponse <- Operations.submitInvalidWorkflow(workflow) _ <- validateSubmitFailure(workflow, expectedSubmitResponse, actualSubmitResponse) } yield actualSubmitResponse @@ -258,6 +280,7 @@ object TestFormulas extends StrictLogging { case ManagedCromwellServer(_, postRestart, withRestart) if withRestart => for { _ <- checkDescription(workflowDefinition, validityExpectation = Option(true)) + _ <- timingVerificationNotSupported(workflowDefinition.maximumAllowedTime) first <- submitWorkflow(workflowDefinition) _ <- pollUntilCallIsRunning(workflowDefinition, first, callMarker.callKey) _ = CromwellManager.stopCromwell(s"Scheduled restart from ${workflowDefinition.testName}") diff --git a/centaur/src/main/scala/centaur/test/standard/CentaurTestCase.scala b/centaur/src/main/scala/centaur/test/standard/CentaurTestCase.scala index 268802975ea..82cd33cbcbc 100644 --- a/centaur/src/main/scala/centaur/test/standard/CentaurTestCase.scala +++ b/centaur/src/main/scala/centaur/test/standard/CentaurTestCase.scala @@ -23,6 +23,7 @@ case class CentaurTestCase(workflow: Workflow, def testFunction: Test[SubmitResponse] = this.testFormat match { case WorkflowSuccessTest => TestFormulas.runSuccessfulWorkflowAndVerifyMetadata(workflow) + case WorkflowSuccessAndTimedOutputsTest => TestFormulas.runSuccessfulWorkflowAndVerifyTimeAndOutputs(workflow) case WorkflowFailureTest => TestFormulas.runFailingWorkflowAndVerifyMetadata(workflow) case RunTwiceExpectingCallCachingTest => TestFormulas.runWorkflowTwiceExpectingCaching(workflow) case RunThriceExpectingCallCachingTest => TestFormulas.runWorkflowThriceExpectingCaching(workflow) diff --git a/centaur/src/main/scala/centaur/test/standard/CentaurTestFormat.scala b/centaur/src/main/scala/centaur/test/standard/CentaurTestFormat.scala index abf8fdba2d2..8bc9a5fbac1 100644 --- a/centaur/src/main/scala/centaur/test/standard/CentaurTestFormat.scala +++ b/centaur/src/main/scala/centaur/test/standard/CentaurTestFormat.scala @@ -12,6 +12,7 @@ sealed abstract class CentaurTestFormat(val name: String) { def testSpecString: String = this match { case WorkflowSuccessTest => "successfully run" + case WorkflowSuccessAndTimedOutputsTest => "successfully run" case WorkflowFailureTest => "fail during execution" case RunTwiceExpectingCallCachingTest => "call cache the second run of" case RunThriceExpectingCallCachingTest => "call cache the third run of" @@ -43,6 +44,7 @@ object CentaurTestFormat { sealed trait WithCallMarker { this: CentaurTestFormat => val build: CallMarker => CentaurTestFormat } case object WorkflowSuccessTest extends CentaurTestFormat("WorkflowSuccess") + case object WorkflowSuccessAndTimedOutputsTest extends CentaurTestFormat("WorkflowSuccessAndTimedOutputs") case object WorkflowFailureTest extends CentaurTestFormat("WorkflowFailure") case object RunTwiceExpectingCallCachingTest extends CentaurTestFormat("RunTwiceExpectingCallCaching") case object RunThriceExpectingCallCachingTest extends CentaurTestFormat(name = "RunThriceExpectingCallCaching") @@ -104,6 +106,7 @@ object CentaurTestFormat { List( WorkflowSuccessTest, + WorkflowSuccessAndTimedOutputsTest, WorkflowFailureTest, RunTwiceExpectingCallCachingTest, RunThriceExpectingCallCachingTest, diff --git a/centaur/src/main/scala/centaur/test/workflow/Workflow.scala b/centaur/src/main/scala/centaur/test/workflow/Workflow.scala index dc117cbe3f1..953f6fdef9a 100644 --- a/centaur/src/main/scala/centaur/test/workflow/Workflow.scala +++ b/centaur/src/main/scala/centaur/test/workflow/Workflow.scala @@ -2,7 +2,6 @@ package centaur.test.workflow import java.nio.file.Path -import common.validation.Validation._ import better.files._ import cats.data.Validated._ import cats.syntax.apply._ @@ -10,10 +9,12 @@ import cats.syntax.validated._ import centaur.test.metadata.WorkflowFlatMetadata import com.typesafe.config.{Config, ConfigFactory} import common.validation.ErrorOr.ErrorOr +import common.validation.Validation._ import configs.Result import configs.syntax._ import cromwell.api.model.{WorkflowDescribeRequest, WorkflowSingleSubmission} +import scala.concurrent.duration.FiniteDuration import scala.util.{Failure, Success, Try} final case class Workflow private(testName: String, @@ -24,7 +25,8 @@ final case class Workflow private(testName: String, backends: BackendsRequirement, retryTestFailures: Boolean, allowOtherOutputs: Boolean, - skipDescribeEndpointValidation: Boolean) { + skipDescribeEndpointValidation: Boolean, + maximumAllowedTime: Option[FiniteDuration]) { def toWorkflowSubmission: WorkflowSingleSubmission = WorkflowSingleSubmission( workflowSource = data.workflowContent, workflowUrl = data.workflowUrl, @@ -91,8 +93,10 @@ object Workflow { val validateDescription: Boolean = conf.get[Boolean]("skipDescribeEndpointValidation").valueOrElse(false) + val maximumTime: Option[FiniteDuration] = conf.get[Option[FiniteDuration]]("maximumTime").value + (files, directoryContentCheckValidation, metadata, retryTestFailuresErrorOr) mapN { - (f, d, m, retryTestFailures) => Workflow(n, f, m, absentMetadata, d, backendsRequirement, retryTestFailures, allowOtherOutputs, validateDescription) + (f, d, m, retryTestFailures) => Workflow(n, f, m, absentMetadata, d, backendsRequirement, retryTestFailures, allowOtherOutputs, validateDescription, maximumTime) } case Result.Failure(_) => invalidNel(s"No test 'name' for: $configFile") diff --git a/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunner.scala b/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunner.scala index 4b4ce33c7e2..2b02bb77308 100644 --- a/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunner.scala +++ b/centaurCwlRunner/src/main/scala/centaur/cwl/CentaurCwlRunner.scala @@ -64,7 +64,7 @@ object CentaurCwlRunner extends StrictLogging { private lazy val versionString = s"$centaurCwlRunnerVersion ${centaurCwlRunnerRunMode.description}" private def showUsage(): ExitCode.Value = { - parser.showUsage() + System.err.println(parser.usage) ExitCode.Failure } @@ -175,7 +175,8 @@ object CentaurCwlRunner extends StrictLogging { backends, retryTestFailures = false, allowOtherOutputs = true, - skipDescribeEndpointValidation = true + skipDescribeEndpointValidation = true, + maximumAllowedTime = None ) val testCase = CentaurTestCase(workflow, testFormat, testOptions, submitResponseOption)(cromwellTracker = None) diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsPathResolver.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsPathResolver.scala index 22ee5c1c0cb..88959096e8d 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsPathResolver.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/DrsPathResolver.scala @@ -78,6 +78,7 @@ object MarthaField extends Enumeration { val Size: MarthaField.Value = Value("size") val TimeCreated: MarthaField.Value = Value("timeCreated") val TimeUpdated: MarthaField.Value = Value("timeUpdated") + val BondProvider: MarthaField.Value = Value("bondProvider") val GoogleServiceAccount: MarthaField.Value = Value("googleServiceAccount") val Hashes: MarthaField.Value = Value("hashes") val FileName: MarthaField.Value = Value("fileName") @@ -94,17 +95,20 @@ final case class SADataObject(data: Json) * @param timeCreated The creation time of the object at gsUri * @param timeUpdated The last update time of the object at gsUri * @param gsUri Where the object bytes are stored, possibly using a generated path name such as "gs://bucket/12/345" - * @param googleServiceAccount The service account to access the gsUri contents + * @param bondProvider The bond provider returning the googleServiceAccount + * @param googleServiceAccount The service account to access the gsUri contents created via bondProvider * @param fileName A possible different file name for the object at gsUri, ex: "gsutil cp gs://bucket/12/345 my.vcf" * @param hashes Hashes for the contents stored at gsUri */ -final case class MarthaResponse(size: Option[Long], - timeCreated: Option[String], - timeUpdated: Option[String], - gsUri: Option[String], - googleServiceAccount: Option[SADataObject], - fileName: Option[String], - hashes: Option[Map[String, String]]) +final case class MarthaResponse(size: Option[Long] = None, + timeCreated: Option[String] = None, + timeUpdated: Option[String] = None, + gsUri: Option[String] = None, + bondProvider: Option[String] = None, + googleServiceAccount: Option[SADataObject] = None, + fileName: Option[String] = None, + hashes: Option[Map[String, String]] = None, + ) // Adapted from https://github.com/broadinstitute/martha/blob/f31933a3a11e20d30698ec4b4dc1e0abbb31a8bc/common/helpers.js#L210-L218 final case class MarthaFailureResponse(response: MarthaFailureResponsePayload) diff --git a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategy.scala b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategy.scala index 597bdee8654..2881fc60b05 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategy.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/main/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategy.scala @@ -29,7 +29,7 @@ class MarthaHttpRequestRetryStrategy(drsConfig: DrsConfig) /** Returns true if HttpResponse should be retried after getRetryInterval. */ override def retryRequest(response: HttpResponse, executionCount: Int, context: HttpContext): Boolean = { response.getStatusLine.getStatusCode match { - case 429 => retryRequestTransient(executionCount) + case code if code == 408 || code == 429 => retryRequestTransient(executionCount) case code if 500 <= code && code <= 599 => retryRequest(executionCount) case _ => false } diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsCloudNioFileProviderSpec.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsCloudNioFileProviderSpec.scala index ae5491f7491..c256f0baecd 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsCloudNioFileProviderSpec.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/DrsCloudNioFileProviderSpec.scala @@ -77,17 +77,7 @@ class DrsCloudNioFileProviderSpec extends AnyFlatSpecLike with CromwellTimeoutSp override def resolveDrsThroughMartha(drsPath: String, fields: NonEmptyList[MarthaField.Value], ): IO[MarthaResponse] = { - IO( - MarthaResponse( - size = None, - timeCreated = None, - timeUpdated = None, - gsUri = Option("gs://bucket/object/path"), - googleServiceAccount = None, - fileName = None, - hashes = None, - ) - ) + IO(MarthaResponse(gsUri = Option("gs://bucket/object/path"))) } } @@ -120,9 +110,6 @@ class DrsCloudNioFileProviderSpec extends AnyFlatSpecLike with CromwellTimeoutSp size = Option(789L), timeCreated = Option(OffsetDateTime.ofInstant(instantCreated, ZoneOffset.UTC).toString), timeUpdated = Option(OffsetDateTime.ofInstant(instantUpdated, ZoneOffset.UTC).toString), - gsUri = None, - googleServiceAccount = None, - fileName = None, hashes = Option(Map("rot13" -> "gg0217869")), ) ) diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategySpec.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategySpec.scala index e3449053a08..dbe85425264 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategySpec.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MarthaHttpRequestRetryStrategySpec.scala @@ -33,11 +33,13 @@ class MarthaHttpRequestRetryStrategySpec extends AnyFlatSpec with Matchers with retryStrategy.retryRequest(http500Response, 5, httpContext) should be(false) } - it should "retry 500 errors even after a number of 429 errors" in { + it should "retry 500 errors even after a number of 408/429 errors" in { val drsConfig = MockDrsPaths.mockDrsConfig.copy(numRetries = 3) val retryStrategy = new MarthaHttpRequestRetryStrategy(drsConfig) val http500Response = mock[CloseableHttpResponse].smart http500Response.getStatusLine returns new BasicStatusLine(HttpVersion.HTTP_1_1, 500, "Testing 500") + val http408Response = mock[CloseableHttpResponse].smart + http408Response.getStatusLine returns new BasicStatusLine(HttpVersion.HTTP_1_1, 408, "Testing 408") val http429Response = mock[CloseableHttpResponse].smart http429Response.getStatusLine returns new BasicStatusLine(HttpVersion.HTTP_1_1, 429, "Testing 429") val httpContext = mock[HttpContext].smart @@ -46,14 +48,14 @@ class MarthaHttpRequestRetryStrategySpec extends AnyFlatSpec with Matchers with retryStrategy.retryRequest(http500Response, 1, httpContext) should be(true) // one retry retryStrategy.retryRequest(http500Response, 2, httpContext) should be(true) - // a couple 429s - retryStrategy.retryRequest(http429Response, 3, httpContext) should be(true) + // a 408 and a 429 + retryStrategy.retryRequest(http408Response, 3, httpContext) should be(true) retryStrategy.retryRequest(http429Response, 4, httpContext) should be(true) // two more 500s should still retry retryStrategy.retryRequest(http500Response, 5, httpContext) should be(true) retryStrategy.retryRequest(http500Response, 6, httpContext) should be(true) - // can still retry 429 - retryStrategy.retryRequest(http429Response, 7, httpContext) should be(true) + // can still retry a 408 and a 429 + retryStrategy.retryRequest(http408Response, 7, httpContext) should be(true) retryStrategy.retryRequest(http429Response, 8, httpContext) should be(true) // but no more retries for 500 retryStrategy.retryRequest(http500Response, 9, httpContext) should be(false) diff --git a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockEngineDrsPathResolver.scala b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockEngineDrsPathResolver.scala index 919552ee494..45410656592 100644 --- a/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockEngineDrsPathResolver.scala +++ b/cloud-nio/cloud-nio-impl-drs/src/test/scala/cloud/nio/impl/drs/MockEngineDrsPathResolver.scala @@ -32,8 +32,6 @@ class MockEngineDrsPathResolver(drsConfig: DrsConfig = MockDrsPaths.mockDrsConfi timeCreated = Option("2020-04-27T15:56:09.696Z"), timeUpdated = Option("2020-04-27T15:56:09.696Z"), gsUri = Option(s"gs://${MockDrsPaths.drsRelativePath}"), - googleServiceAccount = None, - fileName = None, hashes = Option(hashesObj) ) diff --git a/core/src/main/resources/reference.conf b/core/src/main/resources/reference.conf index 711b75c4950..bc15c71d099 100644 --- a/core/src/main/resources/reference.conf +++ b/core/src/main/resources/reference.conf @@ -295,6 +295,10 @@ filesystems { config { martha { url = "https://martha-url-here" + # When true (and when possible) any DrsPath will be internally converted to another compatible Path such as a + # GcsPath. This enables other optimizations such as using batch requests for GCS for hashes, using the + # existing GCS downloader in Papi, etc. + preresolve = false } access-token-acceptable-ttl = 1 minute } diff --git a/core/src/main/scala/cromwell/core/WorkflowOptions.scala b/core/src/main/scala/cromwell/core/WorkflowOptions.scala index 6e7aa9f4712..acd1dab8ad4 100644 --- a/core/src/main/scala/cromwell/core/WorkflowOptions.scala +++ b/core/src/main/scala/cromwell/core/WorkflowOptions.scala @@ -60,6 +60,7 @@ object WorkflowOptions { // Misc. case object DefaultRuntimeOptions extends WorkflowOption("default_runtime_attributes") case object WorkflowFailureMode extends WorkflowOption("workflow_failure_mode") + case object UseReferenceDisks extends WorkflowOption("use_reference_disks") private lazy val WorkflowOptionsConf = ConfigFactory.load.getConfig("workflow-options") private lazy val EncryptedFields: Seq[String] = WorkflowOptionsConf.getStringList("encrypted-fields").asScala diff --git a/core/src/main/scala/cromwell/core/io/AsyncIo.scala b/core/src/main/scala/cromwell/core/io/AsyncIo.scala index f8637f484b4..557586cb283 100644 --- a/core/src/main/scala/cromwell/core/io/AsyncIo.scala +++ b/core/src/main/scala/cromwell/core/io/AsyncIo.scala @@ -10,20 +10,28 @@ import scala.concurrent.Future import scala.concurrent.duration._ import net.ceedubs.ficus.Ficus._ +import scala.util.{Failure, Success, Try} + object AsyncIo { private val ioTimeouts = ConfigFactory.load().as[Config]("system.io.timeout") - val defaultTimeout = ioTimeouts.as[FiniteDuration]("default") - val copyTimeout = ioTimeouts.as[FiniteDuration]("copy") + val defaultTimeout: FiniteDuration = ioTimeouts.as[FiniteDuration]("default") + val copyTimeout: FiniteDuration = ioTimeouts.as[FiniteDuration]("copy") } /** * Provides Futurized methods for I/O actions processed through the IoActor */ class AsyncIo(ioEndpoint: ActorRef, ioCommandBuilder: IoCommandBuilder) { - private def asyncCommand[A](command: IoCommand[A], timeout: FiniteDuration = AsyncIo.defaultTimeout) = { - val commandWithPromise = IoCommandWithPromise(command, timeout) - ioEndpoint ! commandWithPromise - commandWithPromise.promise.future + private def asyncCommand[A](commandTry: Try[IoCommand[A]], + timeout: FiniteDuration = AsyncIo.defaultTimeout): Future[A] = { + commandTry match { + case Failure(throwable) => + Future.failed(throwable) + case Success(command) => + val commandWithPromise = IoCommandWithPromise(command, timeout) + ioEndpoint ! commandWithPromise + commandWithPromise.promise.future + } } /** @@ -62,8 +70,8 @@ class AsyncIo(ioEndpoint: ActorRef, ioCommandBuilder: IoCommandBuilder) { asyncCommand(ioCommandBuilder.isDirectoryCommand(path)) } - def copyAsync(src: Path, dest: Path, overwrite: Boolean = true): Future[Unit] = { + def copyAsync(src: Path, dest: Path): Future[Unit] = { // Allow for a much larger timeout for copies, as large files can take a while (even on gcs, if they are in different locations...) - asyncCommand(ioCommandBuilder.copyCommand(src, dest, overwrite), AsyncIo.copyTimeout) + asyncCommand(ioCommandBuilder.copyCommand(src, dest), AsyncIo.copyTimeout) } } diff --git a/core/src/main/scala/cromwell/core/io/DefaultIoCommand.scala b/core/src/main/scala/cromwell/core/io/DefaultIoCommand.scala index 291abef71e5..bb5e5eec973 100644 --- a/core/src/main/scala/cromwell/core/io/DefaultIoCommand.scala +++ b/core/src/main/scala/cromwell/core/io/DefaultIoCommand.scala @@ -7,8 +7,8 @@ import cromwell.core.path.Path object DefaultIoCommand { case class DefaultIoCopyCommand(override val source: Path, override val destination: Path, - override val overwrite: Boolean) extends IoCopyCommand(source, destination, overwrite) { - override def commandDescription: String = s"DefaultIoCopyCommand source '$source' destination '$destination' overwrite '$overwrite'" + ) extends IoCopyCommand(source, destination) { + override def commandDescription: String = s"DefaultIoCopyCommand source '$source' destination '$destination'" } case class DefaultIoContentAsStringCommand(override val file: Path, override val options: IoReadOptions) extends IoContentAsStringCommand(file, options) { diff --git a/core/src/main/scala/cromwell/core/io/IoCommand.scala b/core/src/main/scala/cromwell/core/io/IoCommand.scala index cc9c3cf6dfa..303504a135a 100644 --- a/core/src/main/scala/cromwell/core/io/IoCommand.scala +++ b/core/src/main/scala/cromwell/core/io/IoCommand.scala @@ -5,20 +5,21 @@ import java.util.UUID import better.files.File.OpenOptions import com.google.api.client.util.ExponentialBackOff +import common.util.Backoff import common.util.StringUtil.EnhancedToStringable import cromwell.core.io.IoContentAsStringCommand.IoReadOptions import cromwell.core.path.Path import cromwell.core.retry.SimpleExponentialBackoff import org.slf4j.LoggerFactory -import scala.concurrent.duration.{FiniteDuration, _} +import scala.concurrent.duration._ import scala.language.postfixOps object IoCommand { private val logger = LoggerFactory.getLogger(IoCommand.getClass) val IOCommandWarnLimit: FiniteDuration = 5 minutes - def defaultGoogleBackoff = new ExponentialBackOff.Builder() + def defaultGoogleBackoff: ExponentialBackOff = new ExponentialBackOff.Builder() .setInitialIntervalMillis((1 second).toMillis.toInt) .setMaxIntervalMillis((5 minutes).toMillis.toInt) .setMultiplier(3L) @@ -26,7 +27,7 @@ object IoCommand { .setMaxElapsedTimeMillis((10 minutes).toMillis.toInt) .build() - def defaultBackoff = SimpleExponentialBackoff(defaultGoogleBackoff) + def defaultBackoff: Backoff = SimpleExponentialBackoff(defaultGoogleBackoff) type RetryCommand[T] = (FiniteDuration, IoCommand[T]) } @@ -93,8 +94,8 @@ trait SingleFileIoCommand[T] extends IoCommand[T] { * Copy source -> destination * Will create the destination directory if it doesn't exist. */ -abstract class IoCopyCommand(val source: Path, val destination: Path, val overwrite: Boolean) extends IoCommand[Unit] { - override def toString = s"copy ${source.pathAsString} to ${destination.pathAsString} with overwrite = $overwrite" +abstract class IoCopyCommand(val source: Path, val destination: Path) extends IoCommand[Unit] { + override def toString = s"copy ${source.pathAsString} to ${destination.pathAsString}" override lazy val name = "copy" } diff --git a/core/src/main/scala/cromwell/core/io/IoCommandBuilder.scala b/core/src/main/scala/cromwell/core/io/IoCommandBuilder.scala index 23c17eb9a5f..4265bd43fb5 100644 --- a/core/src/main/scala/cromwell/core/io/IoCommandBuilder.scala +++ b/core/src/main/scala/cromwell/core/io/IoCommandBuilder.scala @@ -5,20 +5,24 @@ import cromwell.core.io.IoContentAsStringCommand.IoReadOptions import cromwell.core.path.BetterFileMethods.OpenOptions import cromwell.core.path.Path +import scala.util.Try + /** * Can be used to customize IoCommands for the desired I/O operations */ +//noinspection MutatorLikeMethodIsParameterless abstract class PartialIoCommandBuilder { - def contentAsStringCommand: PartialFunction[(Path, Option[Int], Boolean), IoContentAsStringCommand] = PartialFunction.empty - def writeCommand: PartialFunction[(Path, String, OpenOptions, Boolean), IoWriteCommand] = PartialFunction.empty - def sizeCommand: PartialFunction[Path, IoSizeCommand] = PartialFunction.empty - def deleteCommand: PartialFunction[(Path, Boolean), IoDeleteCommand] = PartialFunction.empty - def copyCommand: PartialFunction[(Path, Path, Boolean), IoCopyCommand] = PartialFunction.empty - def hashCommand: PartialFunction[Path, IoHashCommand] = PartialFunction.empty - def touchCommand: PartialFunction[Path, IoTouchCommand] = PartialFunction.empty - def existsCommand: PartialFunction[Path, IoExistsCommand] = PartialFunction.empty - def isDirectoryCommand: PartialFunction[Path, IoIsDirectoryCommand] = PartialFunction.empty - def readLinesCommand: PartialFunction[Path, IoReadLinesCommand] = PartialFunction.empty + def contentAsStringCommand: PartialFunction[(Path, Option[Int], Boolean), Try[IoContentAsStringCommand]] = + PartialFunction.empty + def writeCommand: PartialFunction[(Path, String, OpenOptions, Boolean), Try[IoWriteCommand]] = PartialFunction.empty + def sizeCommand: PartialFunction[Path, Try[IoSizeCommand]] = PartialFunction.empty + def deleteCommand: PartialFunction[(Path, Boolean), Try[IoDeleteCommand]] = PartialFunction.empty + def copyCommand: PartialFunction[(Path, Path), Try[IoCopyCommand]] = PartialFunction.empty + def hashCommand: PartialFunction[Path, Try[IoHashCommand]] = PartialFunction.empty + def touchCommand: PartialFunction[Path, Try[IoTouchCommand]] = PartialFunction.empty + def existsCommand: PartialFunction[Path, Try[IoExistsCommand]] = PartialFunction.empty + def isDirectoryCommand: PartialFunction[Path, Try[IoIsDirectoryCommand]] = PartialFunction.empty + def readLinesCommand: PartialFunction[Path, Try[IoReadLinesCommand]] = PartialFunction.empty } object IoCommandBuilder { @@ -43,51 +47,56 @@ object IoCommandBuilder { */ class IoCommandBuilder(partialBuilders: List[PartialIoCommandBuilder] = List.empty) { // Find the first partialBuilder for which the partial function is defined, or use the default - private def buildOrDefault[A, B](builder: PartialIoCommandBuilder => PartialFunction[A, B], + private def buildOrDefault[A, B](builder: PartialIoCommandBuilder => PartialFunction[A, Try[B]], params: A, - default: => B) = { + default: => B): Try[B] = { partialBuilders.toStream.map(builder(_).lift(params)).collectFirst({ case Some(command) => command - }).getOrElse(default) + }).getOrElse(Try(default)) } - def contentAsStringCommand(path: Path, maxBytes: Option[Int], failOnOverflow: Boolean): IoContentAsStringCommand = { + def contentAsStringCommand(path: Path, + maxBytes: Option[Int], + failOnOverflow: Boolean): Try[IoContentAsStringCommand] = { buildOrDefault(_.contentAsStringCommand, (path, maxBytes, failOnOverflow), DefaultIoContentAsStringCommand(path, IoReadOptions(maxBytes, failOnOverflow))) } - def writeCommand(path: Path, content: String, options: OpenOptions, compressPayload: Boolean = false): IoWriteCommand = { + def writeCommand(path: Path, + content: String, + options: OpenOptions, + compressPayload: Boolean = false): Try[IoWriteCommand] = { buildOrDefault(_.writeCommand, (path, content, options, compressPayload), DefaultIoWriteCommand(path, content, options, compressPayload)) } - def sizeCommand(path: Path): IoSizeCommand = { + def sizeCommand(path: Path): Try[IoSizeCommand] = { buildOrDefault(_.sizeCommand, path, DefaultIoSizeCommand(path)) } - def deleteCommand(path: Path, swallowIoExceptions: Boolean = true): IoDeleteCommand = { + def deleteCommand(path: Path, swallowIoExceptions: Boolean = true): Try[IoDeleteCommand] = { buildOrDefault(_.deleteCommand, (path, swallowIoExceptions), DefaultIoDeleteCommand(path, swallowIoExceptions)) } - def copyCommand(src: Path, dest: Path, overwrite: Boolean): IoCopyCommand = { - buildOrDefault(_.copyCommand, (src, dest, overwrite), DefaultIoCopyCommand(src, dest, overwrite)) + def copyCommand(src: Path, dest: Path): Try[IoCopyCommand] = { + buildOrDefault(_.copyCommand, (src, dest), DefaultIoCopyCommand(src, dest)) } - def hashCommand(file: Path): IoHashCommand = { + def hashCommand(file: Path): Try[IoHashCommand] = { buildOrDefault(_.hashCommand, file, DefaultIoHashCommand(file)) } - def touchCommand(file: Path): IoTouchCommand = { + def touchCommand(file: Path): Try[IoTouchCommand] = { buildOrDefault(_.touchCommand, file, DefaultIoTouchCommand(file)) } - def existsCommand(file: Path): IoExistsCommand = { + def existsCommand(file: Path): Try[IoExistsCommand] = { buildOrDefault(_.existsCommand, file, DefaultIoExistsCommand(file)) } - def isDirectoryCommand(file: Path): IoIsDirectoryCommand = { + def isDirectoryCommand(file: Path): Try[IoIsDirectoryCommand] = { buildOrDefault(_.isDirectoryCommand, file, DefaultIoIsDirectoryCommand(file)) } - def readLines(file: Path): IoReadLinesCommand = { + def readLines(file: Path): Try[IoReadLinesCommand] = { buildOrDefault(_.readLinesCommand, file, DefaultIoReadLinesCommand(file)) } } diff --git a/core/src/main/scala/cromwell/core/path/PathBuilder.scala b/core/src/main/scala/cromwell/core/path/PathBuilder.scala index 5d412a10e16..91bec082853 100644 --- a/core/src/main/scala/cromwell/core/path/PathBuilder.scala +++ b/core/src/main/scala/cromwell/core/path/PathBuilder.scala @@ -5,9 +5,31 @@ import scala.util.Try trait PathBuilder { def name: String + /** + * Builds a path from a string. + * + * @param pathAsString The path to build + * @param pathBuilders Nil, or other potential builders, for example internally map GCS to a pre-signed HTTPS url + * @return A Success(path) or a Failure(throwable) + */ + def build(pathAsString: String, pathBuilders: List[PathBuilder]): Try[Path] = build(pathAsString) + + /** + * Alternative, simpler method of building a Path from a String. + */ def build(pathAsString: String): Try[Path] } +/** + * Extension of PathBuilder that attempts to pre-resolve paths to a different Path type, for example pre-resolving a + * DrsPath to a GcsPath. + */ +trait PreResolvePathBuilder extends PathBuilder { + def build(pathAsString: String, pathBuilders: List[PathBuilder]): Try[Path] + + final override def build(pathAsString: String): Try[Path] = build(pathAsString, Nil) +} + /** * A path that was built by a PathBuilder. * diff --git a/core/src/main/scala/cromwell/core/path/PathFactory.scala b/core/src/main/scala/cromwell/core/path/PathFactory.scala index 31d6262e72d..a9e074afcc7 100644 --- a/core/src/main/scala/cromwell/core/path/PathFactory.scala +++ b/core/src/main/scala/cromwell/core/path/PathFactory.scala @@ -41,19 +41,20 @@ object PathFactory { @tailrec private def findFirstSuccess(string: String, - pathBuilders: PathBuilders, - failures: Vector[String]): ErrorOr[Path] = pathBuilders match { + allPathBuilders: PathBuilders, + restPathBuilders: PathBuilders, + failures: Vector[String]): ErrorOr[Path] = restPathBuilders match { case Nil => NonEmptyList.fromList(failures.toList) match { case Some(errors) => Invalid(errors) case None => s"Could not parse '$string' to path. No PathBuilders were provided".invalidNel } case pb :: rest => - pb.build(string) match { + pb.build(string, allPathBuilders) match { case Success(path) => path.validNel case Failure(f) => val newFailure = s"${pb.name}: ${f.getMessage} (${f.getClass.getSimpleName})" - findFirstSuccess(string, rest, failures :+ newFailure) + findFirstSuccess(string, allPathBuilders, rest, failures :+ newFailure) } } @@ -69,7 +70,7 @@ object PathFactory { val path = for { preMapped <- Try(preMapping(string)).toErrorOr.contextualizeErrors(s"pre map $string") - path <- findFirstSuccess(preMapped, pathBuilders, Vector.empty) + path <- findFirstSuccess(preMapped, pathBuilders, pathBuilders, Vector.empty) postMapped <- Try(postMapping(path)).toErrorOr.contextualizeErrors(s"post map $path") } yield postMapped diff --git a/core/src/main/scala/cromwell/util/JsonEditor.scala b/core/src/main/scala/cromwell/util/JsonEditor.scala index 695c2c9f474..09263cf11eb 100644 --- a/core/src/main/scala/cromwell/util/JsonEditor.scala +++ b/core/src/main/scala/cromwell/util/JsonEditor.scala @@ -2,10 +2,10 @@ package cromwell.util import cats.data.NonEmptyList import cats.data.Validated.{Invalid, Valid} -import cats.syntax.traverse._ -import cats.syntax.validated._ import cats.instances.list._ import cats.instances.vector._ +import cats.syntax.traverse._ +import cats.syntax.validated._ import common.collections.EnhancedCollections._ import common.util.StringUtil._ import common.validation.ErrorOr._ @@ -41,6 +41,60 @@ object JsonEditor { case _ => json.validNel } + /** + * If the workflow's `calls` element contains call attempts matching the specified FQN and index, return an edited + * version of the workflow containing only those matching call attempts, otherwise return an empty JSON. + * + * @param workflowJson Full input workflow JSON. + * @param callFqn Fully qualified name of the call to be included. + * @param index Scatter index of the call to be returned, `None` if the call is not scattered. + * @return Workflow JSON edited to include only call attempts matching the specified `callFqn` and `index` + * within `calls`, or an empty JSON if there are no matching call attempts. + */ + def filterCalls(workflowJson: Json, callFqn: String, index: Option[Int]): ErrorOr[Json] = { + + def workflowAsObject: ErrorOr[JsonObject] = + workflowJson.asObject.map(_.validNel).getOrElse(s"Workflow JSON unexpectedly not an object: $workflowJson".invalidNel) + + // Return the value for the "calls" key as a JsonObject, or an empty JsonObject if "calls" is missing. + def callsAsObject(obj: JsonObject): ErrorOr[JsonObject] = obj(Keys.calls) match { + case None => JsonObject.empty.validNel + case Some(cs) => + cs.asObject map { _.validNel } getOrElse s"calls JSON unexpectedly not an object: $cs".invalidNel + } + + // Return only those calls which match the call FQN and the shard index. + def findMatchingCalls(callsObject: JsonObject): Vector[Json] = { + val effectiveIndex = index.getOrElse(-1) + + for { + callForFqn <- callsObject(callFqn).toVector + attemptsForFqn <- callForFqn.asArray.toVector + attempt <- attemptsForFqn + attemptAsObject <- attempt.asObject + shardIndexJson <- attemptAsObject(Keys.shardIndex) + shardIndex <- shardIndexJson.asNumber + shardIndexInt <- shardIndex.toInt + if shardIndexInt == effectiveIndex + } yield attempt + } + + // Assigns the workflow's `calls` entry to include only the `matchingCalls` values. + def writeMatchingCallsToWorkflow(workflow: JsonObject, matchingCalls: Vector[Json]): JsonObject = { + val callsObject = JsonObject.singleton(callFqn, Json.fromValues(matchingCalls)) + workflow.add(Keys.calls, Json.fromJsonObject(callsObject)) + } + + for { + workflowObject <- workflowAsObject + callsObject <- callsAsObject(workflowObject) + matchingCalls = findMatchingCalls(callsObject) + // Consistent with the classic metadata service, this returns a completely empty JSON object if there are + // no matching calls. + resultObject = if (matchingCalls.isEmpty) JsonObject.empty else writeMatchingCallsToWorkflow(workflowObject, matchingCalls) + } yield Json.fromJsonObject(resultObject) + } + /** A `Filter` represents a list of one or more components corresponding to a single `includeKey` or `excludeKey` parameter. * If an `includeKey` or `excludeKey` parameter specifies a nested term such as `foo:bar`, the `Filter`'s `components` * would be the two element list `[foo, bar]`. diff --git a/core/src/test/resources/application.conf b/core/src/test/resources/application.conf index c212beb98e0..7b2a2e95bbd 100644 --- a/core/src/test/resources/application.conf +++ b/core/src/test/resources/application.conf @@ -38,5 +38,10 @@ google { # ADC creds are NOT set up on Travis, etc. scheme = "mock" } + { + # For those integration tests that aren't (currently as of Nov 2020) run in CI that actually _do_ use ADC + name = "integration-test" + scheme = "application_default" + } ] } diff --git a/core/src/test/resources/hello_goodbye_papiv2.json b/core/src/test/resources/hello_goodbye_papiv2.json new file mode 100644 index 00000000000..a5431f32d9c --- /dev/null +++ b/core/src/test/resources/hello_goodbye_papiv2.json @@ -0,0 +1,242 @@ +{ + "actualWorkflowLanguage": "WDL", + "actualWorkflowLanguageVersion": "1.0", + "calls": { + "wf_hello.goodbye": [ + { + "attempt": 1, + "backend": "Papi", + "backendLogs": { + "log": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/call-goodbye/goodbye.log" + }, + "callCaching": { + "allowResultReuse": true, + "effectiveCallCachingMode": "ReadAndWriteCache", + "hashes": { + "backend name": "36EF4A8AB268D1A1C74D8108C93D48ED", + "command template": "AC223CBDD5FCB85B9E1EEBA7B093BF9E", + "input": { + "String addressee": "2F904D52F600E616095A91048BB716A3" + }, + "input count": "C4CA4238A0B923820DCC509A6F75849B", + "output count": "C4CA4238A0B923820DCC509A6F75849B", + "output expression": { + "String salutation": "0183144CF6617D5341681C6B2F756046" + }, + "runtime attribute": { + "continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA", + "docker": "09F611634147800124391D34D57A3A9F", + "failOnStderr": "68934A3E9455FA72420237EB05902327" + } + }, + "hit": true, + "result": "Cache Hit: fd07f36a-3b17-4ae1-bf50-801ba6b773bd:wf_hello.goodbye:0" + }, + "callRoot": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/call-goodbye", + "compressedDockerSize": 50310356, + "end": "2020-12-21T23:25:51.995Z", + "executionEvents": [ + { + "description": "PreparingJob", + "endTime": "2020-12-21T23:25:46.436Z", + "startTime": "2020-12-21T23:25:46.355Z" + }, + { + "description": "UpdatingCallCache", + "endTime": "2020-12-21T23:25:51.030Z", + "startTime": "2020-12-21T23:25:49.900Z" + }, + { + "description": "Pending", + "endTime": "2020-12-21T23:25:45.716Z", + "startTime": "2020-12-21T23:25:45.716Z" + }, + { + "description": "WaitingForValueStore", + "endTime": "2020-12-21T23:25:46.355Z", + "startTime": "2020-12-21T23:25:46.353Z" + }, + { + "description": "UpdatingJobStore", + "endTime": "2020-12-21T23:25:51.995Z", + "startTime": "2020-12-21T23:25:51.030Z" + }, + { + "description": "CallCacheReading", + "endTime": "2020-12-21T23:25:49.900Z", + "startTime": "2020-12-21T23:25:46.436Z" + }, + { + "description": "RequestingExecutionToken", + "endTime": "2020-12-21T23:25:46.353Z", + "startTime": "2020-12-21T23:25:45.716Z" + } + ], + "executionStatus": "Done", + "inputs": { + "addressee": "there" + }, + "outputs": { + "salutation": "Goodbye there." + }, + "returnCode": 0, + "runtimeAttributes": { + "bootDiskSizeGb": "10", + "continueOnReturnCode": "0", + "cpu": "1", + "cpuMin": "1", + "disks": "local-disk 10 SSD", + "docker": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "failOnStderr": "false", + "maxRetries": "0", + "memory": "2 GB", + "memoryMin": "2 GB", + "noAddress": "false", + "preemptible": "0", + "zones": "us-central1-b" + }, + "shardIndex": -1, + "start": "2020-12-21T23:25:45.716Z", + "stderr": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/call-goodbye/stderr", + "stdout": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/call-goodbye/stdout" + } + ], + "wf_hello.hello": [ + { + "attempt": 1, + "backend": "Papi", + "backendLogs": { + "log": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/call-hello/hello.log" + }, + "callCaching": { + "allowResultReuse": true, + "effectiveCallCachingMode": "ReadAndWriteCache", + "hashes": { + "backend name": "36EF4A8AB268D1A1C74D8108C93D48ED", + "command template": "4EAADE3CD5D558C5A6CFA4FD101A1486", + "input": { + "String addressee": "2F904D52F600E616095A91048BB716A3" + }, + "input count": "C4CA4238A0B923820DCC509A6F75849B", + "output count": "C4CA4238A0B923820DCC509A6F75849B", + "output expression": { + "String salutation": "0183144CF6617D5341681C6B2F756046" + }, + "runtime attribute": { + "continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA", + "docker": "09F611634147800124391D34D57A3A9F", + "failOnStderr": "68934A3E9455FA72420237EB05902327" + } + }, + "hit": true, + "result": "Cache Hit: fd07f36a-3b17-4ae1-bf50-801ba6b773bd:wf_hello.hello:0" + }, + "callRoot": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/call-hello", + "compressedDockerSize": 50310356, + "end": "2020-12-21T23:25:51.995Z", + "executionEvents": [ + { + "description": "Pending", + "endTime": "2020-12-21T23:25:45.716Z", + "startTime": "2020-12-21T23:25:45.716Z" + }, + { + "description": "CallCacheReading", + "endTime": "2020-12-21T23:25:49.909Z", + "startTime": "2020-12-21T23:25:46.418Z" + }, + { + "description": "UpdatingJobStore", + "endTime": "2020-12-21T23:25:51.995Z", + "startTime": "2020-12-21T23:25:51.030Z" + }, + { + "description": "UpdatingCallCache", + "endTime": "2020-12-21T23:25:51.030Z", + "startTime": "2020-12-21T23:25:49.909Z" + }, + { + "description": "RequestingExecutionToken", + "endTime": "2020-12-21T23:25:46.353Z", + "startTime": "2020-12-21T23:25:45.716Z" + }, + { + "description": "PreparingJob", + "endTime": "2020-12-21T23:25:46.418Z", + "startTime": "2020-12-21T23:25:46.354Z" + }, + { + "description": "WaitingForValueStore", + "endTime": "2020-12-21T23:25:46.354Z", + "startTime": "2020-12-21T23:25:46.353Z" + } + ], + "executionStatus": "Done", + "inputs": { + "addressee": "there" + }, + "outputs": { + "salutation": "Hello there!" + }, + "returnCode": 0, + "runtimeAttributes": { + "bootDiskSizeGb": "10", + "continueOnReturnCode": "0", + "cpu": "1", + "cpuMin": "1", + "disks": "local-disk 10 SSD", + "docker": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "failOnStderr": "false", + "maxRetries": "0", + "memory": "2 GB", + "memoryMin": "2 GB", + "noAddress": "false", + "preemptible": "0", + "zones": "us-central1-b" + }, + "shardIndex": -1, + "start": "2020-12-21T23:25:45.716Z", + "stderr": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/call-hello/stderr", + "stdout": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/call-hello/stdout" + } + ] + }, + "end": "2020-12-21T23:25:53.577Z", + "id": "9b3f9d6d-67a9-405f-a55c-70550af02127", + "inputs": {}, + "labels": { + "cromwell-workflow-id": "cromwell-9b3f9d6d-67a9-405f-a55c-70550af02127" + }, + "metadataSource": "Archived", + "outputs": { + "wf_hello.goodbyes": "Goodbye there.", + "wf_hello.hellos": "Hello there!" + }, + "start": "2020-12-21T23:25:43.044Z", + "status": "Succeeded", + "submission": "2020-12-21T23:25:42.461Z", + "submittedFiles": { + "inputs": "{}", + "labels": "{}", + "options": "{\n\n}", + "root": "", + "workflow": "version 1.0\n\ntask hello {\n input {\n String addressee\n }\n command {\n echo \"Hello ~{addressee}!\"\n }\n output {\n String salutation = read_string(stdout())\n }\n runtime {\n docker: \"ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950\"\n }\n}\n\ntask goodbye {\n input {\n String addressee\n }\n command {\n echo \"Goodbye ~{addressee}.\"\n }\n output {\n String salutation = read_string(stdout())\n }\n runtime {\n docker: \"ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950\"\n }\n}\n\n\nworkflow wf_hello {\n\n call hello { input: addressee = \"there\" }\n call goodbye { input: addressee = \"there\" }\n\n output {\n String hellos = hello.salutation\n String goodbyes = goodbye.salutation\n }\n}\n", + "workflowUrl": "" + }, + "workflowName": "wf_hello", + "workflowProcessingEvents": [ + { + "cromwellId": "cromid-5b21691", + "cromwellVersion": "55-8d04562-SNAP", + "description": "Finished", + "timestamp": "2020-12-21T23:25:53.577Z" + }, + { + "cromwellId": "cromid-5b21691", + "cromwellVersion": "55-8d04562-SNAP", + "description": "PickedUp", + "timestamp": "2020-12-21T23:25:43.040Z" + } + ], + "workflowRoot": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/9b3f9d6d-67a9-405f-a55c-70550af02127/" +} diff --git a/core/src/test/resources/hello_goodbye_scattered_papiv2.json b/core/src/test/resources/hello_goodbye_scattered_papiv2.json new file mode 100644 index 00000000000..743f0326ead --- /dev/null +++ b/core/src/test/resources/hello_goodbye_scattered_papiv2.json @@ -0,0 +1,766 @@ +{ + "actualWorkflowLanguage": "WDL", + "actualWorkflowLanguageVersion": "1.0", + "calls": { + "wf_hello.goodbye": [ + { + "attempt": 1, + "backend": "Papi", + "backendLabels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "wdl-task-name": "goodbye" + }, + "backendLogs": { + "log": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-goodbye/shard-0/goodbye-0.log" + }, + "backendStatus": "Success", + "callCaching": { + "allowResultReuse": true, + "effectiveCallCachingMode": "ReadAndWriteCache", + "hashes": { + "backend name": "36EF4A8AB268D1A1C74D8108C93D48ED", + "command template": "AC223CBDD5FCB85B9E1EEBA7B093BF9E", + "input": { + "String addressee": "2F904D52F600E616095A91048BB716A3" + }, + "input count": "C4CA4238A0B923820DCC509A6F75849B", + "output count": "C4CA4238A0B923820DCC509A6F75849B", + "output expression": { + "String salutation": "0183144CF6617D5341681C6B2F756046" + }, + "runtime attribute": { + "continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA", + "docker": "09F611634147800124391D34D57A3A9F", + "failOnStderr": "68934A3E9455FA72420237EB05902327" + } + }, + "hit": false, + "result": "Cache Miss" + }, + "callRoot": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-goodbye/shard-0", + "commandLine": "echo \"Goodbye there.\"", + "compressedDockerSize": 50310356, + "dockerImageUsed": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "end": "2020-12-21T23:19:01.015Z", + "executionEvents": [ + { + "description": "UserAction", + "endTime": "2020-12-21T23:18:26.862Z", + "startTime": "2020-12-21T23:18:24.888Z" + }, + { + "description": "Delocalization", + "endTime": "2020-12-21T23:18:42.976Z", + "startTime": "2020-12-21T23:18:26.862Z" + }, + { + "description": "UpdatingJobStore", + "endTime": "2020-12-21T23:19:01.016Z", + "startTime": "2020-12-21T23:19:00.085Z" + }, + { + "description": "Complete in GCE / Cromwell Poll Interval", + "endTime": "2020-12-21T23:18:58.823Z", + "startTime": "2020-12-21T23:18:42.976Z" + }, + { + "description": "Pulling \"gcr.io/google.com/cloudsdktool/cloud-sdk:276.0.0-slim\"", + "endTime": "2020-12-21T23:17:54.636Z", + "startTime": "2020-12-21T23:17:24.692Z" + }, + { + "description": "Worker released", + "endTime": "2020-12-21T23:18:42.976Z", + "startTime": "2020-12-21T23:18:42.976Z" + }, + { + "description": "Pending", + "endTime": "2020-12-21T23:16:37.901Z", + "startTime": "2020-12-21T23:16:37.875Z" + }, + { + "description": "PreparingJob", + "endTime": "2020-12-21T23:16:39.039Z", + "startTime": "2020-12-21T23:16:38.382Z" + }, + { + "description": "waiting for quota", + "endTime": "2020-12-21T23:16:51.040Z", + "startTime": "2020-12-21T23:16:41.292Z" + }, + { + "description": "RequestingExecutionToken", + "endTime": "2020-12-21T23:16:38.369Z", + "startTime": "2020-12-21T23:16:37.901Z" + }, + { + "description": "Localization", + "endTime": "2020-12-21T23:18:24.888Z", + "startTime": "2020-12-21T23:18:06.432Z" + }, + { + "description": "RunningJob", + "endTime": "2020-12-21T23:16:41.292Z", + "startTime": "2020-12-21T23:16:39.151Z" + }, + { + "description": "Background", + "endTime": "2020-12-21T23:18:05.979Z", + "startTime": "2020-12-21T23:18:05.629Z" + }, + { + "description": "Worker \"google-pipelines-worker-af73ecee357b72f36eea68edf2b4a7e1\" assigned in \"us-central1-b\" on a \"custom-1-2048\" machine", + "endTime": "2020-12-21T23:17:24.692Z", + "startTime": "2020-12-21T23:16:51.040Z" + }, + { + "description": "ContainerSetup", + "endTime": "2020-12-21T23:18:05.399Z", + "startTime": "2020-12-21T23:18:01.650Z" + }, + { + "description": "UpdatingCallCache", + "endTime": "2020-12-21T23:19:00.085Z", + "startTime": "2020-12-21T23:18:58.823Z" + }, + { + "description": "WaitingForValueStore", + "endTime": "2020-12-21T23:16:38.382Z", + "startTime": "2020-12-21T23:16:38.369Z" + }, + { + "description": "CallCacheReading", + "endTime": "2020-12-21T23:16:39.151Z", + "startTime": "2020-12-21T23:16:39.039Z" + }, + { + "description": "Pulling \"ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950\"", + "endTime": "2020-12-21T23:18:01.650Z", + "startTime": "2020-12-21T23:17:54.636Z" + } + ], + "executionStatus": "Done", + "inputs": { + "addressee": "there" + }, + "jes": { + "endpointUrl": "https://genomics.googleapis.com/", + "executionBucket": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci", + "googleProject": "broad-dsde-cromwell-dev", + "instanceName": "google-pipelines-worker-af73ecee357b72f36eea68edf2b4a7e1", + "machineType": "custom-1-2048", + "zone": "us-central1-b" + }, + "jobId": "projects/broad-dsde-cromwell-dev/operations/13186741046740392546", + "labels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "wdl-task-name": "goodbye" + }, + "outputs": { + "salutation": "Goodbye there." + }, + "preemptible": false, + "returnCode": 0, + "runtimeAttributes": { + "bootDiskSizeGb": "10", + "continueOnReturnCode": "0", + "cpu": "1", + "cpuMin": "1", + "disks": "local-disk 10 SSD", + "docker": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "failOnStderr": "false", + "maxRetries": "0", + "memory": "2 GB", + "memoryMin": "2 GB", + "noAddress": "false", + "preemptible": "0", + "zones": "us-central1-b" + }, + "shardIndex": 0, + "start": "2020-12-21T23:16:37.846Z", + "stderr": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-goodbye/shard-0/stderr", + "stdout": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-goodbye/shard-0/stdout" + }, + { + "attempt": 1, + "backend": "Papi", + "backendLabels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "wdl-task-name": "goodbye" + }, + "backendLogs": { + "log": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-goodbye/shard-1/goodbye-1.log" + }, + "backendStatus": "Success", + "callCaching": { + "allowResultReuse": true, + "effectiveCallCachingMode": "ReadAndWriteCache", + "hashes": { + "backend name": "36EF4A8AB268D1A1C74D8108C93D48ED", + "command template": "AC223CBDD5FCB85B9E1EEBA7B093BF9E", + "input": { + "String addressee": "2F904D52F600E616095A91048BB716A3" + }, + "input count": "C4CA4238A0B923820DCC509A6F75849B", + "output count": "C4CA4238A0B923820DCC509A6F75849B", + "output expression": { + "String salutation": "0183144CF6617D5341681C6B2F756046" + }, + "runtime attribute": { + "continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA", + "docker": "09F611634147800124391D34D57A3A9F", + "failOnStderr": "68934A3E9455FA72420237EB05902327" + } + }, + "hit": false, + "result": "Cache Miss" + }, + "callRoot": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-goodbye/shard-1", + "commandLine": "echo \"Goodbye there.\"", + "compressedDockerSize": 50310356, + "dockerImageUsed": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "end": "2020-12-21T23:19:01.021Z", + "executionEvents": [ + { + "description": "Worker released", + "endTime": "2020-12-21T23:18:43.547Z", + "startTime": "2020-12-21T23:18:43.547Z" + }, + { + "description": "Pulling \"gcr.io/google.com/cloudsdktool/cloud-sdk:276.0.0-slim\"", + "endTime": "2020-12-21T23:17:54.656Z", + "startTime": "2020-12-21T23:17:24.345Z" + }, + { + "description": "waiting for quota", + "endTime": "2020-12-21T23:16:51.041Z", + "startTime": "2020-12-21T23:16:41.331Z" + }, + { + "description": "UpdatingCallCache", + "endTime": "2020-12-21T23:19:00.085Z", + "startTime": "2020-12-21T23:18:58.823Z" + }, + { + "description": "Complete in GCE / Cromwell Poll Interval", + "endTime": "2020-12-21T23:18:58.823Z", + "startTime": "2020-12-21T23:18:43.547Z" + }, + { + "description": "Background", + "endTime": "2020-12-21T23:18:05.940Z", + "startTime": "2020-12-21T23:18:05.559Z" + }, + { + "description": "WaitingForValueStore", + "endTime": "2020-12-21T23:16:38.382Z", + "startTime": "2020-12-21T23:16:38.369Z" + }, + { + "description": "Delocalization", + "endTime": "2020-12-21T23:18:43.547Z", + "startTime": "2020-12-21T23:18:27.293Z" + }, + { + "description": "PreparingJob", + "endTime": "2020-12-21T23:16:39.039Z", + "startTime": "2020-12-21T23:16:38.382Z" + }, + { + "description": "RequestingExecutionToken", + "endTime": "2020-12-21T23:16:38.369Z", + "startTime": "2020-12-21T23:16:37.901Z" + }, + { + "description": "CallCacheReading", + "endTime": "2020-12-21T23:16:39.151Z", + "startTime": "2020-12-21T23:16:39.039Z" + }, + { + "description": "Pulling \"ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950\"", + "endTime": "2020-12-21T23:18:01.663Z", + "startTime": "2020-12-21T23:17:54.656Z" + }, + { + "description": "Worker \"google-pipelines-worker-0731feb287704fc08854dd4d0140900a\" assigned in \"us-central1-b\" on a \"custom-1-2048\" machine", + "endTime": "2020-12-21T23:17:24.345Z", + "startTime": "2020-12-21T23:16:51.041Z" + }, + { + "description": "Localization", + "endTime": "2020-12-21T23:18:25.221Z", + "startTime": "2020-12-21T23:18:06.407Z" + }, + { + "description": "Pending", + "endTime": "2020-12-21T23:16:37.901Z", + "startTime": "2020-12-21T23:16:37.875Z" + }, + { + "description": "ContainerSetup", + "endTime": "2020-12-21T23:18:05.270Z", + "startTime": "2020-12-21T23:18:01.663Z" + }, + { + "description": "UserAction", + "endTime": "2020-12-21T23:18:27.293Z", + "startTime": "2020-12-21T23:18:25.221Z" + }, + { + "description": "RunningJob", + "endTime": "2020-12-21T23:16:41.331Z", + "startTime": "2020-12-21T23:16:39.151Z" + }, + { + "description": "UpdatingJobStore", + "endTime": "2020-12-21T23:19:01.017Z", + "startTime": "2020-12-21T23:19:00.085Z" + } + ], + "executionStatus": "Done", + "inputs": { + "addressee": "there" + }, + "jes": { + "endpointUrl": "https://genomics.googleapis.com/", + "executionBucket": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci", + "googleProject": "broad-dsde-cromwell-dev", + "instanceName": "google-pipelines-worker-0731feb287704fc08854dd4d0140900a", + "machineType": "custom-1-2048", + "zone": "us-central1-b" + }, + "jobId": "projects/broad-dsde-cromwell-dev/operations/1896089790402893326", + "labels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "wdl-task-name": "goodbye" + }, + "outputs": { + "salutation": "Goodbye there." + }, + "preemptible": false, + "returnCode": 0, + "runtimeAttributes": { + "bootDiskSizeGb": "10", + "continueOnReturnCode": "0", + "cpu": "1", + "cpuMin": "1", + "disks": "local-disk 10 SSD", + "docker": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "failOnStderr": "false", + "maxRetries": "0", + "memory": "2 GB", + "memoryMin": "2 GB", + "noAddress": "false", + "preemptible": "0", + "zones": "us-central1-b" + }, + "shardIndex": 1, + "start": "2020-12-21T23:16:37.845Z", + "stderr": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-goodbye/shard-1/stderr", + "stdout": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-goodbye/shard-1/stdout" + } + ], + "wf_hello.hello": [ + { + "attempt": 1, + "backend": "Papi", + "backendLabels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "wdl-task-name": "hello" + }, + "backendLogs": { + "log": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-hello/shard-0/hello-0.log" + }, + "backendStatus": "Success", + "callCaching": { + "allowResultReuse": true, + "effectiveCallCachingMode": "ReadAndWriteCache", + "hashes": { + "backend name": "36EF4A8AB268D1A1C74D8108C93D48ED", + "command template": "4EAADE3CD5D558C5A6CFA4FD101A1486", + "input": { + "String addressee": "2F904D52F600E616095A91048BB716A3" + }, + "input count": "C4CA4238A0B923820DCC509A6F75849B", + "output count": "C4CA4238A0B923820DCC509A6F75849B", + "output expression": { + "String salutation": "0183144CF6617D5341681C6B2F756046" + }, + "runtime attribute": { + "continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA", + "docker": "09F611634147800124391D34D57A3A9F", + "failOnStderr": "68934A3E9455FA72420237EB05902327" + } + }, + "hit": false, + "result": "Cache Miss" + }, + "callRoot": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-hello/shard-0", + "commandLine": "echo \"Hello there!\"", + "compressedDockerSize": 50310356, + "dockerImageUsed": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "end": "2020-12-21T23:19:01.021Z", + "executionEvents": [ + { + "description": "UpdatingJobStore", + "endTime": "2020-12-21T23:19:01.016Z", + "startTime": "2020-12-21T23:19:00.085Z" + }, + { + "description": "Worker \"google-pipelines-worker-b0be1492647e6d51041a23e7934832b1\" assigned in \"us-central1-b\" on a \"custom-1-2048\" machine", + "endTime": "2020-12-21T23:17:24.854Z", + "startTime": "2020-12-21T23:16:51.040Z" + }, + { + "description": "UserAction", + "endTime": "2020-12-21T23:18:27.990Z", + "startTime": "2020-12-21T23:18:26.039Z" + }, + { + "description": "UpdatingCallCache", + "endTime": "2020-12-21T23:19:00.085Z", + "startTime": "2020-12-21T23:18:58.823Z" + }, + { + "description": "Pulling \"ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950\"", + "endTime": "2020-12-21T23:18:02.449Z", + "startTime": "2020-12-21T23:17:55.151Z" + }, + { + "description": "WaitingForValueStore", + "endTime": "2020-12-21T23:16:38.382Z", + "startTime": "2020-12-21T23:16:38.369Z" + }, + { + "description": "Worker released", + "endTime": "2020-12-21T23:18:44.461Z", + "startTime": "2020-12-21T23:18:44.461Z" + }, + { + "description": "Localization", + "endTime": "2020-12-21T23:18:26.039Z", + "startTime": "2020-12-21T23:18:07.188Z" + }, + { + "description": "Delocalization", + "endTime": "2020-12-21T23:18:44.461Z", + "startTime": "2020-12-21T23:18:27.990Z" + }, + { + "description": "RunningJob", + "endTime": "2020-12-21T23:16:41.280Z", + "startTime": "2020-12-21T23:16:39.151Z" + }, + { + "description": "RequestingExecutionToken", + "endTime": "2020-12-21T23:16:38.369Z", + "startTime": "2020-12-21T23:16:37.901Z" + }, + { + "description": "ContainerSetup", + "endTime": "2020-12-21T23:18:06.119Z", + "startTime": "2020-12-21T23:18:02.449Z" + }, + { + "description": "CallCacheReading", + "endTime": "2020-12-21T23:16:39.151Z", + "startTime": "2020-12-21T23:16:39.052Z" + }, + { + "description": "Background", + "endTime": "2020-12-21T23:18:06.726Z", + "startTime": "2020-12-21T23:18:06.358Z" + }, + { + "description": "PreparingJob", + "endTime": "2020-12-21T23:16:39.052Z", + "startTime": "2020-12-21T23:16:38.382Z" + }, + { + "description": "waiting for quota", + "endTime": "2020-12-21T23:16:51.040Z", + "startTime": "2020-12-21T23:16:41.280Z" + }, + { + "description": "Pending", + "endTime": "2020-12-21T23:16:37.901Z", + "startTime": "2020-12-21T23:16:37.875Z" + }, + { + "description": "Pulling \"gcr.io/google.com/cloudsdktool/cloud-sdk:276.0.0-slim\"", + "endTime": "2020-12-21T23:17:55.151Z", + "startTime": "2020-12-21T23:17:24.854Z" + }, + { + "description": "Complete in GCE / Cromwell Poll Interval", + "endTime": "2020-12-21T23:18:58.823Z", + "startTime": "2020-12-21T23:18:44.461Z" + } + ], + "executionStatus": "Done", + "inputs": { + "addressee": "there" + }, + "jes": { + "endpointUrl": "https://genomics.googleapis.com/", + "executionBucket": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci", + "googleProject": "broad-dsde-cromwell-dev", + "instanceName": "google-pipelines-worker-b0be1492647e6d51041a23e7934832b1", + "machineType": "custom-1-2048", + "zone": "us-central1-b" + }, + "jobId": "projects/broad-dsde-cromwell-dev/operations/9084347496241274837", + "labels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "wdl-task-name": "hello" + }, + "outputs": { + "salutation": "Hello there!" + }, + "preemptible": false, + "returnCode": 0, + "runtimeAttributes": { + "bootDiskSizeGb": "10", + "continueOnReturnCode": "0", + "cpu": "1", + "cpuMin": "1", + "disks": "local-disk 10 SSD", + "docker": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "failOnStderr": "false", + "maxRetries": "0", + "memory": "2 GB", + "memoryMin": "2 GB", + "noAddress": "false", + "preemptible": "0", + "zones": "us-central1-b" + }, + "shardIndex": 0, + "start": "2020-12-21T23:16:37.843Z", + "stderr": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-hello/shard-0/stderr", + "stdout": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-hello/shard-0/stdout" + }, + { + "attempt": 1, + "backend": "Papi", + "backendLabels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "wdl-task-name": "hello" + }, + "backendLogs": { + "log": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-hello/shard-1/hello-1.log" + }, + "backendStatus": "Success", + "callCaching": { + "allowResultReuse": true, + "effectiveCallCachingMode": "ReadAndWriteCache", + "hashes": { + "backend name": "36EF4A8AB268D1A1C74D8108C93D48ED", + "command template": "4EAADE3CD5D558C5A6CFA4FD101A1486", + "input": { + "String addressee": "2F904D52F600E616095A91048BB716A3" + }, + "input count": "C4CA4238A0B923820DCC509A6F75849B", + "output count": "C4CA4238A0B923820DCC509A6F75849B", + "output expression": { + "String salutation": "0183144CF6617D5341681C6B2F756046" + }, + "runtime attribute": { + "continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA", + "docker": "09F611634147800124391D34D57A3A9F", + "failOnStderr": "68934A3E9455FA72420237EB05902327" + } + }, + "hit": false, + "result": "Cache Miss" + }, + "callRoot": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-hello/shard-1", + "commandLine": "echo \"Hello there!\"", + "compressedDockerSize": 50310356, + "dockerImageUsed": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "end": "2020-12-21T23:19:36.982Z", + "executionEvents": [ + { + "description": "waiting for quota", + "endTime": "2020-12-21T23:16:51.040Z", + "startTime": "2020-12-21T23:16:41.329Z" + }, + { + "description": "Delocalization", + "endTime": "2020-12-21T23:18:42.311Z", + "startTime": "2020-12-21T23:18:26.163Z" + }, + { + "description": "PreparingJob", + "endTime": "2020-12-21T23:16:39.039Z", + "startTime": "2020-12-21T23:16:38.382Z" + }, + { + "description": "UserAction", + "endTime": "2020-12-21T23:18:26.163Z", + "startTime": "2020-12-21T23:18:24.107Z" + }, + { + "description": "Worker \"google-pipelines-worker-072f05d5b3f86e515d4a1a242bfbe403\" assigned in \"us-central1-b\" on a \"custom-1-2048\" machine", + "endTime": "2020-12-21T23:17:24.044Z", + "startTime": "2020-12-21T23:16:51.040Z" + }, + { + "description": "Pulling \"ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950\"", + "endTime": "2020-12-21T23:18:00.288Z", + "startTime": "2020-12-21T23:17:53.439Z" + }, + { + "description": "Background", + "endTime": "2020-12-21T23:18:04.542Z", + "startTime": "2020-12-21T23:18:03.943Z" + }, + { + "description": "UpdatingJobStore", + "endTime": "2020-12-21T23:19:36.984Z", + "startTime": "2020-12-21T23:19:36.027Z" + }, + { + "description": "RunningJob", + "endTime": "2020-12-21T23:16:41.329Z", + "startTime": "2020-12-21T23:16:39.151Z" + }, + { + "description": "Pulling \"gcr.io/google.com/cloudsdktool/cloud-sdk:276.0.0-slim\"", + "endTime": "2020-12-21T23:17:53.439Z", + "startTime": "2020-12-21T23:17:24.044Z" + }, + { + "description": "ContainerSetup", + "endTime": "2020-12-21T23:18:03.943Z", + "startTime": "2020-12-21T23:18:00.288Z" + }, + { + "description": "Complete in GCE / Cromwell Poll Interval", + "endTime": "2020-12-21T23:19:33.894Z", + "startTime": "2020-12-21T23:18:42.311Z" + }, + { + "description": "WaitingForValueStore", + "endTime": "2020-12-21T23:16:38.382Z", + "startTime": "2020-12-21T23:16:38.369Z" + }, + { + "description": "Worker released", + "endTime": "2020-12-21T23:18:42.311Z", + "startTime": "2020-12-21T23:18:42.311Z" + }, + { + "description": "CallCacheReading", + "endTime": "2020-12-21T23:16:39.151Z", + "startTime": "2020-12-21T23:16:39.039Z" + }, + { + "description": "RequestingExecutionToken", + "endTime": "2020-12-21T23:16:38.369Z", + "startTime": "2020-12-21T23:16:37.901Z" + }, + { + "description": "Pending", + "endTime": "2020-12-21T23:16:37.901Z", + "startTime": "2020-12-21T23:16:37.875Z" + }, + { + "description": "Localization", + "endTime": "2020-12-21T23:18:24.107Z", + "startTime": "2020-12-21T23:18:05.033Z" + }, + { + "description": "UpdatingCallCache", + "endTime": "2020-12-21T23:19:36.027Z", + "startTime": "2020-12-21T23:19:33.894Z" + } + ], + "executionStatus": "Done", + "inputs": { + "addressee": "there" + }, + "jes": { + "endpointUrl": "https://genomics.googleapis.com/", + "executionBucket": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci", + "googleProject": "broad-dsde-cromwell-dev", + "instanceName": "google-pipelines-worker-072f05d5b3f86e515d4a1a242bfbe403", + "machineType": "custom-1-2048", + "zone": "us-central1-b" + }, + "jobId": "projects/broad-dsde-cromwell-dev/operations/890602706881000377", + "labels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "wdl-task-name": "hello" + }, + "outputs": { + "salutation": "Hello there!" + }, + "preemptible": false, + "returnCode": 0, + "runtimeAttributes": { + "bootDiskSizeGb": "10", + "continueOnReturnCode": "0", + "cpu": "1", + "cpuMin": "1", + "disks": "local-disk 10 SSD", + "docker": "ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950", + "failOnStderr": "false", + "maxRetries": "0", + "memory": "2 GB", + "memoryMin": "2 GB", + "noAddress": "false", + "preemptible": "0", + "zones": "us-central1-b" + }, + "shardIndex": 1, + "start": "2020-12-21T23:16:37.846Z", + "stderr": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-hello/shard-1/stderr", + "stdout": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/call-hello/shard-1/stdout" + } + ] + }, + "end": "2020-12-21T23:19:39.491Z", + "id": "fd07f36a-3b17-4ae1-bf50-801ba6b773bd", + "inputs": {}, + "labels": { + "cromwell-workflow-id": "cromwell-fd07f36a-3b17-4ae1-bf50-801ba6b773bd" + }, + "metadataSource": "Archived", + "outputs": { + "wf_hello.goodbyes": [ + "Goodbye there.", + "Goodbye there." + ], + "wf_hello.hellos": [ + "Hello there!", + "Hello there!" + ] + }, + "start": "2020-12-21T23:16:31.284Z", + "status": "Succeeded", + "submission": "2020-12-21T23:16:30.985Z", + "submittedFiles": { + "inputs": "{}", + "labels": "{}", + "options": "{\n\n}", + "root": "", + "workflow": "version 1.0\n\ntask hello {\n input {\n String addressee\n }\n command {\n echo \"Hello ~{addressee}!\"\n }\n output {\n String salutation = read_string(stdout())\n }\n runtime {\n docker: \"ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950\"\n }\n}\n\ntask goodbye {\n input {\n String addressee\n }\n command {\n echo \"Goodbye ~{addressee}.\"\n }\n output {\n String salutation = read_string(stdout())\n }\n runtime {\n docker: \"ubuntu@sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950\"\n }\n}\n\n\nworkflow wf_hello {\n scatter (i in range(2)) {\n call hello { input: addressee = \"there\" }\n call goodbye { input: addressee = \"there\" }\n }\n output {\n Array[String] hellos = hello.salutation\n Array[String] goodbyes = goodbye.salutation\n }\n}\n", + "workflowUrl": "" + }, + "workflowName": "wf_hello", + "workflowProcessingEvents": [ + { + "cromwellId": "cromid-5b21691", + "cromwellVersion": "55-8d04562-SNAP", + "description": "PickedUp", + "timestamp": "2020-12-21T23:16:31.206Z" + }, + { + "cromwellId": "cromid-5b21691", + "cromwellVersion": "55-8d04562-SNAP", + "description": "Finished", + "timestamp": "2020-12-21T23:19:39.493Z" + } + ], + "workflowRoot": "gs://cloud-cromwell-dev-self-cleaning/cromwell_execution/ci/wf_hello/fd07f36a-3b17-4ae1-bf50-801ba6b773bd/" +} diff --git a/core/src/test/scala/cromwell/core/SimpleIoActor.scala b/core/src/test/scala/cromwell/core/SimpleIoActor.scala index 1acb275171e..589065758c5 100644 --- a/core/src/test/scala/cromwell/core/SimpleIoActor.scala +++ b/core/src/test/scala/cromwell/core/SimpleIoActor.scala @@ -10,15 +10,15 @@ import scala.concurrent.Promise import scala.util.{Failure, Success, Try} object SimpleIoActor { - def props = Props(new SimpleIoActor) + def props: Props = Props(new SimpleIoActor) } class SimpleIoActor extends Actor { - override def receive = { + override def receive: Receive = { case command: IoCopyCommand => - Try(command.source.copyTo(command.destination, command.overwrite)) match { + Try(command.source.copyTo(command.destination)) match { case Success(_) => sender() ! IoSuccess(command, ()) case Failure(failure) => sender() ! IoFailure(command, failure) } @@ -66,7 +66,7 @@ class SimpleIoActor extends Actor { // With context case (requestContext: Any, command: IoCopyCommand) => - Try(command.source.copyTo(command.destination, command.overwrite)) match { + Try(command.source.copyTo(command.destination, overwrite = true)) match { case Success(_) => sender() ! (requestContext -> IoSuccess(command, ())) case Failure(failure) => sender() ! (requestContext -> IoFailure(command, failure)) } diff --git a/core/src/test/scala/cromwell/core/TestKitSuite.scala b/core/src/test/scala/cromwell/core/TestKitSuite.scala index df1317deae3..d9fe5c3d56c 100644 --- a/core/src/test/scala/cromwell/core/TestKitSuite.scala +++ b/core/src/test/scala/cromwell/core/TestKitSuite.scala @@ -2,35 +2,36 @@ package cromwell.core import java.util.UUID -import akka.actor.ActorSystem -import akka.testkit.{TestActors, TestKit} +import akka.actor.{ActorRef, ActorSystem} +import akka.testkit.{TestActors, TestKitBase} import com.typesafe.config.{Config, ConfigFactory} import org.scalatest.{BeforeAndAfterAll, Suite} /** * A mix of Akka TestKit with ScalaTest mixed in to clean up the actor system. - * - * @param actorSystemName The name of the actor system. - * @param actorSystemConfig The config for the actor system. */ -abstract class TestKitSuite(actorSystemName: String = TestKitSuite.randomName, - actorSystemConfig: Config = TestKitSuite.config) - extends TestKit(ActorSystem(actorSystemName, actorSystemConfig)) with Suite with BeforeAndAfterAll { +abstract class TestKitSuite extends TestKitBase with Suite with BeforeAndAfterAll { - override protected def afterAll() = { + protected lazy val actorSystemName: String = this.getClass.getSimpleName + + protected lazy val actorSystemConfig: Config = TestKitSuite.config + + implicit lazy val system: ActorSystem = ActorSystem(actorSystemName, actorSystemConfig) + + override protected def afterAll(): Unit = { shutdown() } // 'BlackHoleActor' swallows messages without logging them (thus reduces log file overhead): - val emptyActor = system.actorOf(TestActors.blackholeProps, "TestKitSuiteEmptyActor") + val emptyActor: ActorRef = system.actorOf(TestActors.blackholeProps, "TestKitSuiteEmptyActor") - val mockIoActor = system.actorOf(MockIoActor.props(), "TestKitSuiteMockIoActor") - val simpleIoActor = system.actorOf(SimpleIoActor.props, "TestKitSuiteSimpleIoActor") - val failIoActor = system.actorOf(FailIoActor.props(), "TestKitSuiteFailIoActor") + val mockIoActor: ActorRef = system.actorOf(MockIoActor.props(), "TestKitSuiteMockIoActor") + val simpleIoActor: ActorRef = system.actorOf(SimpleIoActor.props, "TestKitSuiteSimpleIoActor") + val failIoActor: ActorRef = system.actorOf(FailIoActor.props(), "TestKitSuiteFailIoActor") } object TestKitSuite { - val configString = + val configString: String = """ |akka { | loggers = ["akka.testkit.TestEventListener"] @@ -96,7 +97,7 @@ object TestKitSuite { |} |""".stripMargin - val config = ConfigFactory.parseString(configString) + val config: Config = ConfigFactory.parseString(configString) def randomName = s"TestSystem-${UUID.randomUUID}" } diff --git a/core/src/test/scala/cromwell/core/io/AsyncIoSpec.scala b/core/src/test/scala/cromwell/core/io/AsyncIoSpec.scala index ce53b961fdb..4b7aff15962 100644 --- a/core/src/test/scala/cromwell/core/io/AsyncIoSpec.scala +++ b/core/src/test/scala/cromwell/core/io/AsyncIoSpec.scala @@ -1,21 +1,24 @@ package cromwell.core.io -import java.nio.file.{FileAlreadyExistsException, NoSuchFileException} +import java.nio.file.NoSuchFileException import java.util.UUID import akka.actor.{Actor, ActorLogging, ActorRef} import akka.testkit.TestActorRef import cromwell.core.TestKitSuite -import cromwell.core.path.DefaultPathBuilder +import cromwell.core.path.{DefaultPathBuilder, Path} import org.scalatest.flatspec.AsyncFlatSpecLike import org.scalatest.matchers.should.Matchers import org.scalatestplus.mockito.MockitoSugar +import scala.util.{Failure, Try} +import scala.util.control.NoStackTrace + class AsyncIoSpec extends TestKitSuite with AsyncFlatSpecLike with Matchers with MockitoSugar { behavior of "AsyncIoSpec" - implicit val ioCommandBuilder = DefaultIoCommandBuilder + implicit val ioCommandBuilder: DefaultIoCommandBuilder.type = DefaultIoCommandBuilder it should "write asynchronously" in { val testActor = TestActorRef(new AsyncIoTestActor(simpleIoActor)) @@ -66,20 +69,16 @@ class AsyncIoSpec extends TestKitSuite with AsyncFlatSpecLike with Matchers with val testPath = DefaultPathBuilder.createTempFile() val testCopyPath = testPath.sibling(UUID.randomUUID().toString) - testActor.underlyingActor.asyncIo.copyAsync(testPath, testCopyPath) map { hash => + testActor.underlyingActor.asyncIo.copyAsync(testPath, testCopyPath) map { _ => assert(testCopyPath.exists) } testPath.write("new text") - // Honor overwrite true - testActor.underlyingActor.asyncIo.copyAsync(testPath, testCopyPath, overwrite = true) map { hash => + testActor.underlyingActor.asyncIo.copyAsync(testPath, testCopyPath) map { _ => assert(testCopyPath.exists) assert(testCopyPath.contentAsString == "new text") } - - // Honor overwrite false - recoverToSucceededIf[FileAlreadyExistsException] { testActor.underlyingActor.asyncIo.copyAsync(testPath, testCopyPath, overwrite = false) } } it should "delete asynchronously" in { @@ -97,13 +96,33 @@ class AsyncIoSpec extends TestKitSuite with AsyncFlatSpecLike with Matchers with } // Honor swallow exception false - recoverToSucceededIf[NoSuchFileException] { testActor.underlyingActor.asyncIo.deleteAsync(testPath, swallowIoExceptions = false) } + //noinspection RedundantDefaultArgument + recoverToSucceededIf[NoSuchFileException] { + testActor.underlyingActor.asyncIo.deleteAsync(testPath, swallowIoExceptions = false) + } } - private class AsyncIoTestActor(override val ioActor: ActorRef) extends Actor with ActorLogging with AsyncIoActorClient { + it should "handle command creation errors asynchronously" in { + val partialIoCommandBuilder = new PartialIoCommandBuilder { + override def existsCommand: PartialFunction[Path, Try[IoExistsCommand]] = { + case _ => Failure(new Exception("everything's fine, I am an expected exists fail") with NoStackTrace) + } + } + val testActor = + TestActorRef(new AsyncIoTestActor(simpleIoActor, new IoCommandBuilder(List(partialIoCommandBuilder)))) + + val testPath = DefaultPathBuilder.createTempFile() + testPath.write("hello") + + testActor.underlyingActor.asyncIo.existsAsync(testPath).failed map { throwable => + assert(throwable.getMessage == "everything's fine, I am an expected exists fail") + } + } + + private class AsyncIoTestActor(override val ioActor: ActorRef, + override val ioCommandBuilder: IoCommandBuilder = DefaultIoCommandBuilder + ) extends Actor with ActorLogging with AsyncIoActorClient { - override lazy val ioCommandBuilder = DefaultIoCommandBuilder - override def receive: Receive = { case _ => } diff --git a/core/src/test/scala/cromwell/core/io/IoAckSpec.scala b/core/src/test/scala/cromwell/core/io/IoAckSpec.scala index c8ace526735..2d08220a7e8 100644 --- a/core/src/test/scala/cromwell/core/io/IoAckSpec.scala +++ b/core/src/test/scala/cromwell/core/io/IoAckSpec.scala @@ -12,7 +12,7 @@ class IoAckSpec extends AnyFlatSpecLike with CromwellTimeoutSpec with Matchers { "IoFailAck pattern matching" should "work for both IoFailure and IoReadForbiddenFailure" in { import DefaultPathBuilder._ - val command = DefaultIoCopyCommand(build(Paths.get("foo")), build(Paths.get("bar")), overwrite = false) + val command = DefaultIoCopyCommand(build(Paths.get("foo")), build(Paths.get("bar"))) val ioFailure = IoFailure(command, new RuntimeException("blah")) val ioReadForbiddenFailure = IoFailure(command, new RuntimeException("blah")) diff --git a/core/src/test/scala/cromwell/core/retry/RetrySpec.scala b/core/src/test/scala/cromwell/core/retry/RetrySpec.scala index 9029b8b50c2..83516f3328b 100644 --- a/core/src/test/scala/cromwell/core/retry/RetrySpec.scala +++ b/core/src/test/scala/cromwell/core/retry/RetrySpec.scala @@ -7,14 +7,14 @@ import org.scalatest.flatspec.AnyFlatSpecLike import org.scalatest.matchers.should.Matchers import org.scalatest.time.{Millis, Seconds, Span} -import scala.concurrent.Future +import scala.concurrent.{ExecutionContext, Future} -class RetrySpec extends TestKitSuite("retry-spec") with AnyFlatSpecLike with Matchers with ScalaFutures { +class RetrySpec extends TestKitSuite with AnyFlatSpecLike with Matchers with ScalaFutures { class TransientException extends Exception class MockWork(n: Int, transients: Int = 0) { - implicit val ec = system.dispatcher + implicit val ec: ExecutionContext = system.dispatcher - var counter = n + var counter: Int = n def doIt(): Future[Int] = { if (counter == 0) @@ -27,7 +27,7 @@ class RetrySpec extends TestKitSuite("retry-spec") with AnyFlatSpecLike with Mat } } - implicit val defaultPatience = PatienceConfig(timeout = Span(30, Seconds), interval = Span(100, Millis)) + implicit val defaultPatience: PatienceConfig = PatienceConfig(timeout = Span(30, Seconds), interval = Span(100, Millis)) private def runRetry(retries: Int, work: MockWork, diff --git a/core/src/test/scala/cromwell/util/JsonEditorSpec.scala b/core/src/test/scala/cromwell/util/JsonEditorSpec.scala index fe3e32156f4..a037a831ecb 100644 --- a/core/src/test/scala/cromwell/util/JsonEditorSpec.scala +++ b/core/src/test/scala/cromwell/util/JsonEditorSpec.scala @@ -6,7 +6,7 @@ import common.assertion.CromwellTimeoutSpec import common.validation.ErrorOr.ErrorOr import cromwell.util.JsonEditor._ import io.circe.parser._ -import io.circe.{DecodingFailure, FailedCursor, Json} +import io.circe.{DecodingFailure, FailedCursor, Json, JsonObject} import org.scalatest.flatspec.AnyFlatSpec import org.scalatest.matchers.should.Matchers @@ -344,14 +344,15 @@ class JsonEditorSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers actual shouldEqual expected } + val helloFqn = "wf_hello.hello" it should "handle includes nested multiple levels" in { val includeKeys = NonEmptyList.of("callCaching:hashes:backend name", "callCaching:hashes:input:String addressee") val actual = includeJson(helloWorldPapiV2, includeKeys).get val expectedJson = - """ + s""" |{ | "calls" : { - | "wf_hello.hello" : [ + | "$helloFqn" : [ | { | "callCaching" : { | "hashes": { @@ -393,11 +394,11 @@ class JsonEditorSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers val actual = includeExcludeJson(helloWorldPapiV2, includeKeys, excludeKeys).get val expectedJson = - """ + s""" |{ | "id": "d53a063a-e8b7-403f-a400-a85f089a8928", | "calls" : { - | "wf_hello.hello" : [ + | "$helloFqn" : [ | { | "backendLabels": { | "cromwell-workflow-id": "cromwell-d53a063a-e8b7-403f-a400-a85f089a8928", @@ -420,10 +421,10 @@ class JsonEditorSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers val actual = includeExcludeJson(helloWorldPapiV2, includeKeys, excludeKeys).get val expectedJson = - """ + s""" |{ | "calls": { - | "wf_hello.hello": [ + | "$helloFqn": [ | { | "callCaching": { | "allowResultReuse": true, @@ -463,10 +464,10 @@ class JsonEditorSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers val actual = includeExcludeJson(helloWorldPapiV2, includeKeys, excludeKeys).get val expectedJson = - """ + s""" |{ | "calls": { - | "wf_hello.hello": [ + | "$helloFqn": [ | { | "backendStatus": "Success", | "attempt": 1, @@ -488,10 +489,10 @@ class JsonEditorSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers val actual = includeExcludeJson(helloWorldPapiV2, includeKeys, excludeKeys).get val expectedJson = - """ + s""" |{ | "calls": { - | "wf_hello.hello": [ + | "$helloFqn": [ | { | "backendStatus": "Success", | "attempt": 1, @@ -525,6 +526,60 @@ class JsonEditorSpec extends AnyFlatSpec with CromwellTimeoutSpec with Matchers val expected = helloWorldPapiV2.mapObject(_.remove("calls")) actual shouldEqual expected } + + it should "filter unscattered calls by FQN, if so asked" in { + val actual = filterCalls(helloGoodbyePapiV2, helloFqn, None).get + val expected = helloGoodbyePapiV2.mapObject { metadata => + // We expect to see only the hello call after filtering. + val helloCall = metadata(Keys.calls).get.asObject.get(helloFqn).get + metadata.add(Keys.calls, Json.fromFields(List((helloFqn, helloCall)))) + } + actual shouldEqual expected + } + + it should "filter scattered calls by FQN and index, if so asked" in { + val actual = filterCalls(helloGoodbyeScatteredPapiV2, helloFqn, Option(1)).get + val expected = helloGoodbyeScatteredPapiV2.mapObject { metadata => + val helloCalls = metadata(Keys.calls).get.asObject.get(helloFqn).get + val helloShard1 = helloCalls.asArray.get.filter(_.asObject.get(Keys.shardIndex).get.asNumber.get.toInt.get == 1) + metadata.add(Keys.calls, Json.fromFields(List((helloFqn, Json.fromValues(helloShard1))))) + } + actual shouldEqual expected + } + + val nonexistentFqn = "wf_hello.nonexistent" + it should "gracefully handle being asked to filter an unscattered call that has no matching FQN" in { + val actual = filterCalls(helloGoodbyeScatteredPapiV2, nonexistentFqn, None).get.asObject.get + + // It seems a bit strange but this is what the classic metadata endpoint returns. + actual shouldEqual JsonObject.empty + } + + it should "gracefully handle being asked to filter an unscattered call that exists as a shard of a scatter" in { + val actual = filterCalls(helloGoodbyeScatteredPapiV2, helloFqn, None).get.asObject.get + + actual shouldEqual JsonObject.empty + } + + it should "gracefully handle being asked to filter a scattered call that has no matching FQN (or index)" in { + val actual = filterCalls(helloGoodbyeScatteredPapiV2, nonexistentFqn, Option(0)).get.asObject.get + + actual shouldEqual JsonObject.empty + } + + it should "gracefully handle being asked to filter an unscattered call which has a matching FQN but not matching index" in { + val actual = filterCalls(helloGoodbyeScatteredPapiV2, helloFqn, Option(2)).get.asObject.get + + actual shouldEqual JsonObject.empty + } + + it should "return an empty object when asked to filter calls in a workflow without calls" in { + // Not sure how/if we could end up with a Carbonited workflow that had no calls IRL but if it happens we are ready. + val noCalls = helloGoodbyePapiV2.asObject.get.remove(Keys.calls) + val actual = filterCalls(Json.fromJsonObject(noCalls), helloFqn, None).get.asObject.get + + actual shouldEqual JsonObject.empty + } } object JsonEditorSpec { @@ -545,13 +600,15 @@ object JsonEditorSpec { val treblyNestedSubworkflowCachedJson: Json = parseMetadata("trebly_nested_subworkflow_papiv2_cached.json") val helloWorldJson: Json = parseMetadata("hello_world.json") val helloWorldPapiV2: Json = parseMetadata("hello_papiv2.json") + val helloGoodbyePapiV2: Json = parseMetadata("hello_goodbye_papiv2.json") + val helloGoodbyeScatteredPapiV2: Json = parseMetadata("hello_goodbye_scattered_papiv2.json") implicit class EnhancedErrorOr[A](val e: ErrorOr[A]) extends AnyVal { def get: A = e.toEither.right.get } object JobManagerKeys { - val includeKeys = NonEmptyList.of( + val includeKeys: NonEmptyList[String] = NonEmptyList.of( "attempt", "backendLogs:log", "callCaching:hit", @@ -576,6 +633,6 @@ object JsonEditorSpec { "workflowName" ) - val excludeKeys = NonEmptyList.of("callCaching:hitFailures") + val excludeKeys: NonEmptyList[String] = NonEmptyList.of("callCaching:hitFailures") } } diff --git a/cromwell-drs-localizer/src/test/scala/drs/localizer/DrsLocalizerMainSpec.scala b/cromwell-drs-localizer/src/test/scala/drs/localizer/DrsLocalizerMainSpec.scala index d8c17ef2c56..362a9db0dd2 100644 --- a/cromwell-drs-localizer/src/test/scala/drs/localizer/DrsLocalizerMainSpec.scala +++ b/cromwell-drs-localizer/src/test/scala/drs/localizer/DrsLocalizerMainSpec.scala @@ -150,11 +150,7 @@ class MockGcsLocalizerDrsPathResolver(drsConfig: DrsConfig) extends IO.pure( MarthaResponse( size = Option(1234), - timeCreated = None, - timeUpdated = None, gsUri = gcsUrl, - googleServiceAccount = None, - fileName = None, hashes = Option(Map("md5" -> "abc123", "crc32c" -> "34fd67")) ) ) diff --git a/cromwellApiClient/src/test/scala/cromwell/api/CromwellResponseFailedSpec.scala b/cromwellApiClient/src/test/scala/cromwell/api/CromwellResponseFailedSpec.scala index 7ef94d629fe..06841bfad71 100644 --- a/cromwellApiClient/src/test/scala/cromwell/api/CromwellResponseFailedSpec.scala +++ b/cromwellApiClient/src/test/scala/cromwell/api/CromwellResponseFailedSpec.scala @@ -15,13 +15,14 @@ import org.scalatest.matchers.should.Matchers import scala.concurrent.duration._ import scala.concurrent.{Await, Future} -class CromwellResponseFailedSpec extends TestKit(ActorSystem()) with AsyncFlatSpecLike with Matchers with BeforeAndAfterAll { +class CromwellResponseFailedSpec extends TestKit(ActorSystem("CromwellResponseFailedSpec")) + with AsyncFlatSpecLike with Matchers with BeforeAndAfterAll { override def afterAll(): Unit = { Await.ready(system.terminate(), 10.seconds.dilated) super.afterAll() } - implicit val materializer = ActorMaterializer() + implicit val materializer: ActorMaterializer = ActorMaterializer() "CromwellAPIClient" should "fail the Future if the HttpResponse is unsuccessful" in { val errorMessage = diff --git a/dockerHashing/src/test/scala/cromwell/docker/DockerEmptyFlowSpec.scala b/dockerHashing/src/test/scala/cromwell/docker/DockerEmptyFlowSpec.scala index 75b623224da..0b8f578da69 100644 --- a/dockerHashing/src/test/scala/cromwell/docker/DockerEmptyFlowSpec.scala +++ b/dockerHashing/src/test/scala/cromwell/docker/DockerEmptyFlowSpec.scala @@ -6,7 +6,7 @@ import org.scalatest.matchers.should.Matchers import scala.concurrent.duration._ -class DockerEmptyFlowSpec extends DockerRegistrySpec("DockerEmptyFlowSpec") with AnyFlatSpecLike with Matchers { +class DockerEmptyFlowSpec extends DockerRegistrySpec with AnyFlatSpecLike with Matchers { behavior of "An empty docker flow" override protected def registryFlows: Seq[DockerRegistry] = Seq() diff --git a/dockerHashing/src/test/scala/cromwell/docker/DockerInfoActorSpec.scala b/dockerHashing/src/test/scala/cromwell/docker/DockerInfoActorSpec.scala index 390c93c15cf..f56f7569ac5 100644 --- a/dockerHashing/src/test/scala/cromwell/docker/DockerInfoActorSpec.scala +++ b/dockerHashing/src/test/scala/cromwell/docker/DockerInfoActorSpec.scala @@ -12,7 +12,7 @@ import org.scalatest.matchers.should.Matchers import scala.concurrent.duration._ import scala.language.postfixOps -class DockerInfoActorSpec extends DockerRegistrySpec("DockerHashActorSpec") with AnyFlatSpecLike with Matchers with BeforeAndAfterAll { +class DockerInfoActorSpec extends DockerRegistrySpec with AnyFlatSpecLike with Matchers with BeforeAndAfterAll { behavior of "DockerRegistryActor" override protected lazy val registryFlows = List( @@ -76,7 +76,10 @@ class DockerInfoActorSpec extends DockerRegistrySpec("DockerHashActorSpec") with // Send back success, failure, success, failure, ... val mockHttpFlow = new DockerRegistryMock(mockResponseSuccess, mockResponseFailure) - val dockerActorWithCache = system.actorOf(DockerInfoActor.props(Seq(mockHttpFlow), 1000, 3 seconds, 10)) + val dockerActorWithCache = system.actorOf( + props = DockerInfoActor.props(Seq(mockHttpFlow), 1000, 3 seconds, 10), + name = "dockerActorWithCache", + ) dockerActorWithCache ! request expectMsg(DockerInfoSuccessResponse(DockerInformation(hashSuccess, None), request)) @@ -99,7 +102,10 @@ class DockerInfoActorSpec extends DockerRegistrySpec("DockerHashActorSpec") with it should "not deadlock" taggedAs IntegrationTest in { - lazy val dockerActorScale = system.actorOf(DockerInfoActor.props(registryFlows, 1000, 20.minutes, 0)) + lazy val dockerActorScale = system.actorOf( + props = DockerInfoActor.props(registryFlows, 1000, 20.minutes, 0), + name = "dockerActorScale", + ) 0 until 400 foreach { _ => dockerActorScale ! makeRequest("gcr.io/google-containers/alpine-with-bash:1.0") } diff --git a/dockerHashing/src/test/scala/cromwell/docker/DockerRegistrySpec.scala b/dockerHashing/src/test/scala/cromwell/docker/DockerRegistrySpec.scala index c8d0b866b0c..ab35c216b51 100644 --- a/dockerHashing/src/test/scala/cromwell/docker/DockerRegistrySpec.scala +++ b/dockerHashing/src/test/scala/cromwell/docker/DockerRegistrySpec.scala @@ -1,28 +1,33 @@ package cromwell.docker +import akka.actor.{ActorRef, Scheduler} import akka.testkit.ImplicitSender -import cats.effect.IO +import cats.effect.{ContextShift, IO} import cromwell.core.TestKitSuite +import scala.concurrent.ExecutionContext import scala.concurrent.duration._ -abstract class DockerRegistrySpec(actorSystemName: String) extends TestKitSuite(actorSystemName) with ImplicitSender { - implicit val ex = system.dispatcher - implicit val cs = IO.contextShift(ex) - implicit val scheduler = system.scheduler +abstract class DockerRegistrySpec extends TestKitSuite with ImplicitSender { + implicit val executionContext: ExecutionContext = system.dispatcher + implicit val contextShift: ContextShift[IO] = IO.contextShift(executionContext) + implicit val scheduler: Scheduler = system.scheduler protected def registryFlows: Seq[DockerRegistry] // Disable cache by setting a cache size of 0 - A separate test tests the cache - lazy val dockerActor = system.actorOf(DockerInfoActor.props(registryFlows, 1000, 20.minutes, 0)) + lazy val dockerActor: ActorRef = system.actorOf( + props = DockerInfoActor.props(registryFlows, 1000, 20.minutes, 0), + name = "dockerActor", + ) - def dockerImage(string: String) = DockerImageIdentifier.fromString(string).get + def dockerImage(string: String): DockerImageIdentifier = DockerImageIdentifier.fromString(string).get - def makeRequest(string: String) = { + def makeRequest(string: String): DockerInfoRequest = { DockerInfoRequest(dockerImage(string)) } - override protected def afterAll() = { + override protected def afterAll(): Unit = { system.stop(dockerActor) super.afterAll() } diff --git a/dockerHashing/src/test/scala/cromwell/docker/local/DockerCliSpec.scala b/dockerHashing/src/test/scala/cromwell/docker/local/DockerCliSpec.scala index 2bbf02b72d1..04a035d06ec 100644 --- a/dockerHashing/src/test/scala/cromwell/docker/local/DockerCliSpec.scala +++ b/dockerHashing/src/test/scala/cromwell/docker/local/DockerCliSpec.scala @@ -8,7 +8,7 @@ import org.scalatest.matchers.should.Matchers import scala.concurrent.duration._ -class DockerCliSpec extends DockerRegistrySpec("DockerCliFlowSpec") with AnyFlatSpecLike with Matchers { +class DockerCliSpec extends DockerRegistrySpec with AnyFlatSpecLike with Matchers { behavior of "DockerCliFlow" override protected def registryFlows: Seq[DockerRegistry] = Seq(new DockerCliFlow) diff --git a/dockerHashing/src/test/scala/cromwell/docker/local/DockerCliTimeoutSpec.scala b/dockerHashing/src/test/scala/cromwell/docker/local/DockerCliTimeoutSpec.scala index ad9552fc321..4098b84899a 100644 --- a/dockerHashing/src/test/scala/cromwell/docker/local/DockerCliTimeoutSpec.scala +++ b/dockerHashing/src/test/scala/cromwell/docker/local/DockerCliTimeoutSpec.scala @@ -9,11 +9,11 @@ import org.scalatest.matchers.should.Matchers import scala.concurrent.TimeoutException import scala.concurrent.duration._ -class DockerCliTimeoutSpec extends DockerRegistrySpec("DockerCliTimeoutFlowSpec") with AnyFlatSpecLike with Matchers { +class DockerCliTimeoutSpec extends DockerRegistrySpec with AnyFlatSpecLike with Matchers { behavior of "A DockerCliFlow that times out" override protected def registryFlows: Seq[DockerRegistry] = Seq(new DockerCliFlow { - override lazy val firstLookupTimeout = 0.seconds + override lazy val firstLookupTimeout: FiniteDuration = 0.seconds }) it should "timeout retrieving a public docker hash" taggedAs IntegrationTest in { diff --git a/docs/Getting.md b/docs/Releases.md similarity index 55% rename from docs/Getting.md rename to docs/Releases.md index c61a4c75ca8..ea2b297c860 100644 --- a/docs/Getting.md +++ b/docs/Releases.md @@ -1,4 +1,4 @@ -**Cromwell Releases** +# Cromwell Releases Cromwell releases are available at the [GitHub Releases](https://github.com/broadinstitute/cromwell/releases/latest) page. You are strongly encouraged to use the latest release of Cromwell whenever possible. @@ -20,4 +20,20 @@ This is the main artifact in Cromwell releases that contains all executable Crom [Java 8](http://www.oracle.com/technetwork/java/javase/overview/java8-2100321.html) is required to run Cromwell. -For users running a cromwell server [a docker image](https://hub.docker.com/r/broadinstitute/cromwell) has been made available. +For users running a Cromwell server [a docker image](https://hub.docker.com/r/broadinstitute/cromwell) has been made available. + +### Apple Silicon support statement (updated 2020-11-17) + +#### Cromwell JAR works out of the box + +The Cromwell JAR works on any standard Java installation. A user can install an x86 Java runtime on an Apple Silicon Mac and the Rosetta 2 translation layer runs Cromwell at near-native speed. + +Once natively-compiled Java runtimes become available, performance will increase with no change in functionality. + +#### Docker Desktop support is in progress + +The Cromwell Docker image will not run on M1 Macs until Docker Desktop ships the appropriate update. For more details, please see [their official announcement](https://www.docker.com/blog/apple-silicon-m1-chips-and-docker/). + +By extension, the absence of Docker means that Cromwell's local Docker backend is not yet supported. + +Even when Docker Desktop goes native on Apple Silicon, any tool images running on the local backend will need to cross-compile for the x86 and Arm architectures. This is because the Rosetta 2 translation layer [does not support virtualization](https://developer.apple.com/documentation/apple_silicon/about_the_rosetta_translation_environment). Please contact the tool maintainers for more information. diff --git a/docs/RuntimeAttributes.md b/docs/RuntimeAttributes.md index c7c0e8295c5..782a80c96fe 100644 --- a/docs/RuntimeAttributes.md +++ b/docs/RuntimeAttributes.md @@ -53,6 +53,7 @@ There are a number of additional runtime attributes that apply to the Google Clo - [noAddress](#noaddress) - [gpuCount, gpuType, and nvidiaDriverVersion](#gpucount-gputype-and-nvidiadriverversion) - [cpuPlatform](#cpuplatform) +- [useDockerImageCache](#usedockerimagecache) @@ -420,3 +421,9 @@ runtime { } ``` Note that when this options is specified, make sure the requested CPU platform is [available](https://cloud.google.com/compute/docs/regions-zones/#available) in the `zones` you selected. + +### 'useDockerImageCache' + +This option is specific to the Google Cloud backend, moreover it is only supported by Google Life Sciences API starting from version v2 beta. +In order to use this feature Cromwell has to have PAPI v2 backend configured with this feature enabled. +More information about this feature and it's configuration can be found [in the Google backend section of documentation](backends/Google.md). diff --git a/docs/api/RESTAPI.md b/docs/api/RESTAPI.md index e542e2a702a..14109caf106 100644 --- a/docs/api/RESTAPI.md +++ b/docs/api/RESTAPI.md @@ -1,5 +1,5 @@