Scala Rules for Bazel

Overview

Bazel is a tool for building and testing software and can handle large, multi-language projects at scale.

This project defines core build rules for Scala that can be used to build, test, and package Scala projects.

Rules

• scala_library
• scala_macro_library
• scala_binary
• scala_test
• scala_repl
• scala_library_suite
• scala_test_suite
• thrift_library
• scala_proto_library
• scala_toolchain
• scala_import
• scala_doc

Getting started

1. Install Bazel, see the compatibility table.
2. Add the following to your WORKSPACE file and update the githash if needed:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# bazel-skylib 0.8.0 released 2019.03.20 (https://github.com/bazelbuild/bazel-skylib/releases/tag/0.8.0)
skylib_version = "0.8.0"
http_archive(
    name = "bazel_skylib",
    type = "tar.gz",
    url = "https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib.{}.tar.gz".format (skylib_version, skylib_version),
    sha256 = "2ef429f5d7ce7111263289644d233707dba35e39696377ebab8b0bc701f7818e",
)

rules_scala_version="69d3c5b5d9b51537231746e93b4383384c9ebcf4" # update this as needed

http_archive(
    name = "io_bazel_rules_scala",
    strip_prefix = "rules_scala-%s" % rules_scala_version,
    type = "zip",
    url = "https://github.com/bazelbuild/rules_scala/archive/%s.zip" % rules_scala_version,
)

load("@io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()

protobuf_version="09745575a923640154bcf307fba8aedff47f240a"
protobuf_version_sha256="416212e14481cff8fd4849b1c1c1200a7f34808a54377e22d7447efdf54ad758"

http_archive(
    name = "com_google_protobuf",
    url = "https://github.com/protocolbuffers/protobuf/archive/%s.tar.gz" % protobuf_version,
    strip_prefix = "protobuf-%s" % protobuf_version,
    sha256 = protobuf_version_sha256,
)

This will load the rules_scala repository at the commit sha rules_scala_version into your Bazel project and register a Scala toolchain at the default Scala version (2.11.12)

Then in your BUILD file just add the following so the rules will be available:

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_binary", "scala_test")

You may wish to have these rules loaded by default using bazel's prelude. You can add the above to the file tools/build_rules/prelude_bazel in your repo (don't forget to have a, possibly empty, BUILD file there) and then it will be automatically prepended to every BUILD file in the workspace.

To run with a persistent worker (much faster), you need to add

build --strategy=Scalac=worker
build --worker_sandboxing

to your command line, or to enable by default for building/testing add it to your .bazelrc.

Selecting Scala version

Rules scala supports the last two released minor versions for each of Scala 2.11 and 2.12. Previous minor versions may work but are supported only on a best effort basis.

By default Scala 2.11.12 is used and to use another version you need to specify it when calling scala_repositories. scala_repositories takes a tuple (scala_version, scala_version_jar_shas) as a parameter where scala_version is the scala version and scala_version_jar_shas is a dict with sha256 hashes for the maven artifacts scala_compiler, scala_library, and scala_reflect:

scala_repositories((
    "2.12.10",
    {
       "scala_compiler": "cedc3b9c39d215a9a3ffc0cc75a1d784b51e9edc7f13051a1b4ad5ae22cfbc0c",
       "scala_library": "0a57044d10895f8d3dd66ad4286891f607169d948845ac51e17b4c1cf0ab569d",
       "scala_reflect": "56b609e1bab9144fb51525bfa01ccd72028154fc40a58685a1e9adcbe7835730"
    }
))

If you're using any of the rules twitter_scrooge, tut_repositories, scala_proto_repositories or specs2_junit_repositories you also need to specify scala_version for them. See ./test_version/WORKSPACE.template for an example workspace using another scala version.

Bazel compatible versions

bazel	rules_scala gitsha
2.0.0	HEAD
1.1.0	d681a952da74fc61a49fc3167b03548f42fc5dde
0.28.1	bd0c388125e12f4f173648fc4474f73160a5c628
0.23.x	ca655e5a330cbf1d66ce1d9baa63522752ec6011
0.22.x	f3113fb6e9e35cb8f441d2305542026d98afc0a2
0.16.x	f3113fb6e9e35cb8f441d2305542026d98afc0a2
0.15.x	3b9ab9be31ac217d3337c709cb6bfeb89c8dcbb1
0.14.x	3b9ab9be31ac217d3337c709cb6bfeb89c8dcbb1
0.13.x	3c987b6ae8a453886759b132f1572c0efca2eca2

Breaking changes

If you're upgrading to a version containing one of these commits, you may encounter a breaking change where there was previously undefined behavior.

• 929b318 on 2020-01-30: Fixed a bug in the JMH benchmark build that was allowing build failures to creep through. Previously you were able to build a benchmark suite with JMH build errors. Running the benchmark suite would only run the successfully-built benchmarks.

Usage with bazel-deps

Bazel-deps allows you to generate bazel dependencies transitively for maven artifacts. Generally we don't want bazel-deps to fetch scala artifacts from maven but instead use the ones we get from calling scala_repositories. The artifacts can be overridden in the dependencies file used by bazel-deps:

replacements:
  org.scala-lang:
    scala-library:
      lang: scala/unmangled
      target: "@io_bazel_rules_scala_scala_library//:io_bazel_rules_scala_scala_library"
    scala-reflect:
      lang: scala/unmangled
      target: "@io_bazel_rules_scala_scala_reflect//:io_bazel_rules_scala_scala_reflect"
    scala-compiler:
      lang: scala/unmangled
      target: "@io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler"

  org.scala-lang.modules:
    scala-parser-combinators:
      lang: scala
      target:
        "@io_bazel_rules_scala_scala_parser_combinators//:io_bazel_rules_scala_scala_parser_combinators"
    scala-xml:
      lang: scala
      target:
        "@io_bazel_rules_scala_scala_xml//:io_bazel_rules_scala_scala_xml"

[Experimental] Dependency options

There are a number of dependency options which can be set in the scala toolchain. These include dependency_mode, strict_deps_mode, unused_dependency_checker_mode, and dependency_tracking_method.

[Experimental] Recommended options

We recommend one of the following sets of options

Option A Accept the defaults, which might work well enough for you. The defaults are

  dependency_mode = "direct",
  strict_deps_mode = "off",
  unused_dependency_checker_mode = "off",
  dependency_tracking_method = "high-level",

but you do not need to include this in the toolchain as they are the defaults.

Option B

  dependency_mode = "plus-one",
  strict_deps_mode = "error",
  unused_dependency_checker_mode = "error",
  dependency_tracking_method = "ast",

Should the first option result in too much effort in handling build files and the like due to confusing dependencies and you becoming confused as to why some specific dependency is needed when the code being compiled never references it, consider this set of options. It will include both dependencies and dependencies of dependencies, which in practice is enough to stop almost all strange missing dependency errors at the cost of somewhat more incremental compile cost in certain cases.

With these settings, we also will error on dependencies which are unneeded, and dependencies which should be included in deps due to be directly referenced in the code, but are not.

The dependency tracking method ast is experimental but so far proves to be better than the default for computing the direct dependencies for plus-one mode code. In the future we hope to make this the default for plus-one mode and remove the option altogether.

[Experimental] Dependency mode

There are three dependency modes. The reason for the multiple modes is that often scalac depends on jars which seem unnecessary at first glance. Hence, in order to reduce the need to please scalac, we provide the following options.

• dependency_mode = "direct" - only include direct dependencies during compiliation; that is, those in the deps attribute
• dependency_mode = "plus-one" - only include deps and deps of deps during compiliation.
• dependency_mode = "transitive" - all transitive dependencies are included during compiliation. That is, deps, deps of deps, deps of deps of deps, and so on.

Note when a dependency is included, that means its jars are included on the classpath, along with the jars of any targets that it exports.

When using direct mode, there can be cryptic scalac errors when one mistakenly depends on a transitive dependency or, as more often the case for some, a transitive dependency is needed to please scalac itself.

As one goes down the list, more dependencies are included which helps reduce confusing requirements to add deps, at the cost of increased incremental builds due to a greater number of dependencies. In practice, using plus-one deps results in almost no confusing deps entries required while still being relatively small in terms of the number of total dependencies included.

Caveats for plus_one and transitive:

• Extra builds- Extra dependencies are inputs to the compilation action which means you can potentially have more build triggers for changes the cross the ijar boundary
• Label propagation- since label of targets are needed for the clear message and since it's not currently supported by JavaInfo from bazel we manually propagate it. This means that the error messages have a significantly lower grade if you don't use one of the scala rules or scala_import (since they don't propagate these labels)
• javac outputs incorrect targets due to a problem we're tracing down. Practically we've noticed it's pretty trivial to understand the correct target (i.e. it's almost a formatting problem)

Note: the last two issues are bugs which will be addressed by [bazelbuild#839].

[Experimental] Strict deps mode

We have a strict dependency checker which requires that any type referenced in the sources of a scala target should be included in that rule's deps. To learn about the motivation for this you can visit this Bazel blog post on the subject.

The option strict_deps_mode can be set to off, warn, or error. We highly recommend setting it to error.

In both cases of warn or error you will get the following text in the event of a violation:

...
Target '//some_package:transitive_dependency' is used but isn't explicitly declared, please add it to the deps.
You can use the following buildozer command:
buildozer 'add deps //some_package:transitive_dependency' //some_other_package:transitive_dependency_user

Note that if you have buildozer installed you can just run the last line and have it automatically apply the fix for you.

[Experimental] Unused dependency checking

To allow for better caching and faster builds we want to minimize the direct dependencies of our targets. Unused dependency checking makes sure that all targets specified as direct dependencies are actually used. If unused_dependency_checker_mode is set to either error or warn you will get the following message for any dependencies that are not used:

error: Target '//some_package:unused_dep' is specified as a dependency to //target:target but isn't used, please remove it from the deps.
You can use the following buildozer command:
buildozer 'remove deps //some_package:unused_dep' //target:target

Unused dependency checking can either be enabled globally for all targets using a scala toolchain or for individual targets using the unused_dependency_checker_mode attribute.

The feature is still experimental and there can thus be cases where it works incorrectly, in these cases you can enable unused dependency checking globally through a toolchain and disable reports of individual misbehaving targets with unused_dependency_checker_ignored_targets which is a list of labels.

[Experimental] Dependency tracking method

The strict dependency tracker and unused dependency tracker need to track the used dependencies of a scala compilation unit. This toggle allows one to pick which method of tracking to use.

• dependency_tracking_method = "high-level" - This is the existing tracking method which has false positives and negatives but generally works reasonably well for direct dependency mode.
• dependency_tracking_method = "ast" - This is a new tracking method which is being developed for plus-one and transitive dependency modes. It is still being developed and may have issues which need fixing. If you discover an issue, please submit a small repro of the problem.

Note we intend to eventually remove this flag and use high-level as the method for direct dependency mode, and ast as the method for plus-one and transitive dependency modes.

In the meantime, if you are using plus-one or transitive dependency modes, you can use ast dependency tracking mode and see how well it works for you.

[Experimental] Turning on strict_deps_mode/unused_dependency_checker_mode

It can be daunting to turn on strict deps checking or unused dependency mode checking on a large codebase. However, it need not be so bad if this is done in phases

1. Have a default scala toolchain A with the option of interest set to off (the starting state)
2. Create a second scala toolchain B with the option of interest set to warn or error. Those who are working on enabling the flag can run with this toolchain as a command line argument to help identify issues and fix them.
3. Once all issues are fixed, change A to have the option of interest set to error and delete B.

We recommend turning on strict_deps_mode first, as rule A might have an entry B in its deps, and B in turn depends on C. Meanwhile, the code of A only uses C but not B. Hence, the unused dependency checker, if on, will request that B be removed from A's deps. But this will lead to a compile error as A can no longer depend on C. However, if strict dependency checking was on, then A's deps is guaranteed to have C in it.

[Experimental] Migrating from deprecated configurations

There are a few deprecated configuration methods which we will be removing in the near future.

• plus_one_deps_mode = "on" on the scala toolchain. Instead, set dependency_mode = "plus-one" on the scala toolchain. plus_one_deps_mode will be removed in the future.
• The command line argument --strict_java_deps=WARN/ERROR. Instead, set dependency_mode = "transitive" on the scala toolchain, and if only a warning is desired set strict_deps_mode = "warn" on the toolchain. In the future, strict_java_deps will no longer affect how scala files are compiled. Note that strict_java_deps will still control java compilation.

Advanced configurable rules

To make the ruleset more flexible and configurable, we introduce a phase architecture. By using a phase architecture, where rule implementations are defined as a list of phases that are executed sequentially, functionality can easily be added (or modified) by adding (or swapping) phases.

Phases provide 3 major benefits:

• Consumers are able to configure the rules to their specific use cases by defining new phases within their workspace without impacting other consumers.
• Contributors are able to implement new functionalities by creating additional default phases.
• Phases give us more clear idea what steps are shared across rules.

See Customizable Phase for more info.

Phase extensions

• Scala Format

Building from source

Test & Build:

bash test_all.sh

You can also use:

bazel test //test/...

Note bazel test //... will not work since we have a sub-folder on the root folder which is meant to be used in a failure scenario in the integration tests. Similarly to only build you should use bazel build //src/... due to that folder.

Updates

This section contains a list of updates that might require action from the user.

• 043ba58: Updates default scrooge version to 18.6.0
• 76d4ab9: Updates naming for scala artifact replacements in bazel-deps. See Usage with bazel-deps

Contributing

See CONTRIBUTING.md for more info.

Adopters

Here's a (non-exhaustive) list of companies that use rules_scala in production. Don't see yours? You can add it in a PR!

• Ascend
• Canva
• Etsy
• Grand Rounds
• Kitty Hawk
• Meetup
• Spotify
• Stripe
• VSCO
• Wix