Skip to content

Latest commit

 

History

History
203 lines (160 loc) · 9.35 KB

Developing.md

File metadata and controls

203 lines (160 loc) · 9.35 KB
Raw

Developing wdlTools

Setting up your development environment

  • Install JDK 11 
    $ brew tap AdoptOpenJDK/openjdk
$ brew install adoptopenjdk11 --cask
# Use java_home to find the location of JAVA_HOME to set
$ /usr/libexec/java_home -V
$ export JAVA_HOME=/Library/Java/...
    • On Linux (assuming Ubuntu 16.04)
    $ sudo apt install openjdk-11-jre-headless
    • Note that wdlTools will compile with JDK8 or JDK11 and that JDK8 is used as the build target so the resulting JAR file can be executed with JRE8 or later.
  • Install sbt, which also installs Scala. Sbt is a make-like utility that works with the scala language.
    • On MacOS: brew install sbt
    • On Linux:
    $ wget www.scala-lang.org/files/archive/scala-2.13.7.deb
$ sudo dpkg -i scala-2.13.7.deb
$ echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
$ sudo apt-get update
$ sudo apt-get install sbt
    • Running sbt for the first time takes several minutes, because it downloads all required packages.
  • We also recommend to install Metals, which enables better integration with your IDE
    • For VSCode, install the "Scala (Metals)" and "Scala Syntax (official)" plugins
  • You will need to create a GitHub personal access token (this is required by the sbt-github-packages plugin).
    • In GitHub settings, go to "Developer settings > Personal access token" and create a new token with "write:packages" and "read:packages" scopes only.
    • Export the GITHUB_TOKEN environment variable with this token as the value. For example, in your .profile:
    export GITHUB_TOKEN=<your personal access token>
    • On macOS, you may also want to add this token into your global environment so it is visible to your IDE:
    launchctl setenv GITHUB_TOKEN $GITHUB_TOKEN

Getting the source code

  • Clone or fork the wdlTools repository (depending on whether you have commit permissions)
  • Checkout an existing branch or create a new branch (e.g. feat/42-my-feature)
  • Add pre-commit hooks:
    • Create/edit a file .git/hooks/pre-commit
    • Add the following lines
    #!/bin/bash
check=$(sbt scalafmtCheckAll)
if [[ "$?" -ne "0" ]]; then
  echo "Reformatting; please commit again"
  sbt scalafmtAll
  exit $check
fi
    • Make the file executable (e.g. chmod +x .git/hooks/pre-commit)
    • This hook runs the code formatter before committing code. You can run this command manually, but it is easiest just to have it run automatically.

Generating the parsers

wdlTools depends on parsers for each WDL version, which are generated by ANTLR4. The first time you build wdlTools, as well as any time you change the parser grammar, you need to (re)generate the parser classes. To do so, run the following command from the wdlTools root directory:

$ make

Using sbt

sbt is the build system for Scala (similar to a Makefile). The following are the main commands you will use while developing.

Compiling the code

Scala (like Java) is a compiled language. To compile, run:

$ sbt compile

If there are errors in your code, the compiler will fail with (hopefully useful) error messages.

Running unit tests

You should always run the unit tests after every successful compile. Generally, you want to run sbt testQuick, which only runs the tests that failed previously, as well as the tests for any code you've modified since the last time you ran the tests. However, the first time you checkout the code (to make sure your development environment is set up correctly) and then right before you push any changes to the repository, you should run the full test suite using sbt test.

Generating a stand-alone JAR file

$ sbt assembly
$ java -jar target/scala-2.13/wdlTools.jar ...

Other sbt tips

Cache

sbt keeps the cache of downloaded jar files in ${HOME}/.ivy2/cache. For example, the WDL jar files are under ${HOME}/.ivy2/cache/org.broadinstitute. In case of problems with cached jars, you can remove this directory recursively. This will make WDL download all dependencies (again).

Adding new code

If you want to make a change to wdlTools, do the following:

  1. Checkout the develop branch.
  2. Create a new branch with your changes. Name it something meaningful, like APPS-123-download-bug.
  3. If the current snapshot version matches the release version, increment the snapshot version.
    • For example, if the current release is 1.0.0 and the current snapshot version is 1.0.0-SNAPSHOT, increment the snapshot version to 1.0.1-SNAPSHOT.
  4. Make your changes.
  5. Always write unit tests for any new code you add, and update tests for any code you modify.
    • Unit tests should assert, and not print to the console
    • WDL test files belong in the top directory test
  6. Test locally using sbt test.
  7. Update the release notes under the top-most header (which should be "in develop").
  8. If the current snapshot version only differs from the release version by a patch, and you added any new functionality (vs just fixing a bug), increment the minor version instead.
    • For example, when you first created the branch you set the version to 1.0.1-SNAPSHOT, but then you realized you needed to add a new function to the public API, change the version to 1.1.0-SNAPSHOT.
  9. When you are done, create a pull request against the develop branch.

Style guidelines

  • We use scalafmt style with a few modifications. You don't need to worry so much about code style since you will use the automatic formatter on your code before it is committed.
  • Readability is more important than efficiency or concision - write more/slower code if it makes the code more readable.
  • Avoid using more complex features, e.g. reflection.

Building a local version for testing

  • Set version name in <project>/src/main/resources/application.conf to X.Y.Z-SNAPSHOT.
  • Run sbt publishLocal, which will publish to your Ivy local file repository.
  • In any downstream project that will depend on the changes you are making, set the dependency version in build.sbt to X.Y.Z-SNAPSHOT.

Merging the PR

When a PR is merged into develop, SNAPSHOT packages are automatically published to GitHub packages. When you push to develop (including merging a PR), you should announce that you are doing so (e.g. via GChat) for two reasons:

  • Publishing a new snapshot requires deleting the existing one. If someone is trying to fetch the snapshot from the repository at the time when the snapshot workflow is running, they will get a "package not found" error.
  • Although unlikely, it is possible that if two people merge into develop at around the same time, the older SNAPSHOT will overwrite the newer one.

Releasing

Beginning the release

  1. Checkout the develop branch (either HEAD or the specific commit you want to release)
  2. Create a release branch named with the version number, e.g. release-2.4.2
  3. Update the version numbers in application.conf files (remove "-SNAPSHOT")
  4. Update the release notes
    • Change the top header from "in develop" to "<version> (<date>)"

Releasing to GitHub

  1. Push the release branch to GitHub.
  2. Run the release action.
  3. Go to the "Releases" page on GitHub and publish the draft release.

Completing the release

If you encounter any additional issues while creating the release, you will need to make the fixes in develop and then merge them into the release branch.

To complete the release:

  1. Create branch post-release-X.Y.Z based on branch release-X.Y.Z
  2. Increment the working version from e.g. published 1.2.3 to 1.2.4-SNAPSHOT in src/main/resources/application.conf.
  3. Open pull request from branch post-release-X.Y.Z to develop. Fix release notes and resolve conflicts as needed.
  4. Do not remove the branch release-X.Y.Z and don't merge it back to main nor develop. We keep this branch for tagging purposes. The main branch is deprecated.

wdlTools CLI

Adding a new command

  1. Create a new class in a file with the same name as the command: 
    package wdlTools.cli

import scala.language.reflectiveCalls

case class MyCommand(conf: WdlToolsConf) extends Command {
  override def apply(): Unit = {
      ...
  }
}
  2. In package, add a new subcommand definition: 
    val mycommand = new WdlToolsSubcommand("mycommand", "description") {
    // add options, for example
    val outputDir: ScallopOption[Path] = opt[Path](
        descr = "Directory in which to output files",
        short = 'O'
    )
}
  3. In Main, add your command to the pattern matcher: 
    conf.subcommand match {
   case None => conf.printHelp()
   case Some(subcommand) =>
     val command: Command = subcommand match {
       case conf.mycommand => MyCommand(conf)
       ...
       case other          => throw new Exception(s"Unrecognized command $other")
     }
     command.apply()
}