Nix+Bazel Codelab

This set of exercises is a practical introduction to building polyglot projects with Bazel, supported by Nix for reproducible, hermetic build environments. It is self-contained and works on Linux and macOS.

Please open an issue if you have questions or encounter problems running the examples.

Please make a pull request if you found and fixed an issue.

Background

This codelab is based on Google's Bazel Codelab. It has been adapted to

• Use more recent versions of the required Bazel rule sets.
• Follow best practices such as formatting with buildifier or using BUILD.bazel files instead of BUILD files.
• Use Nix to provide Bazel and other developer tools in a Nix shell.
• Use rules_nixpkgs to import Nix provided toolchains from nixpkgs.

Project Layout

• direnv used to automatically set up shell environment, see .envrc.
• Nix shell used to provide developer tools like Bazel itself, see shell.nix
• Nix dependencies pinned under nix/.
• Bazel layout
  • MODULE.bazel defines root of Bazel project and defines external module dependencies
    • Uses bazel_dep to import Bazel dependencies from the Bazel Central Registry (BCR)
    • Uses register_toolchains to register toolchains from nixpkgs
    • Uses use_repo to bring repositories into scope
    • Uses other rule sets' dedicated module extensions to import their dependencies, e.g. Go or NodeJS.
  • non_module_dependencies.bzl contains legacy WORKSPACE rules that are not available as native module extensions
    • Uses nixpkgs_*_configure to create toolchains
    • Can use http_archive to import Bazel dependencies
    • Can use nixpkgs_package to import Nix dependencies into Bazel
    • Can use other rule sets' dedicated repository rules to import their dependencies
  • .bazelrc configures Bazel
  • BUILD.bazel files define Bazel packages and targets in that package

Check your files against a known solution

The repository contains bazel build files that are known to work, they are shipped in the solutions subdirectory of the project.

A script named lab is added to your environment with nix-shell, it helps using the solutions files.

lab can be used in multiple ways:

• lab help to get help and examples of use
• lab compare $file to view a difference between your file and the solution if exists
• lab display $file to view its solution if any
• lab install $file to copy the solution if any (this will replace your work!)
• lab install_all install all solutions file, the project should compile fine after running this command

Before you get started

Read up on Bazel Concepts and Terminology.

Install Nix and enter nix-shell

Install the Nix package manager with

sh <(curl -L https://nixos.org/nix/install) --daemon

All commands from the instructions are expected to run in the environment created by entering

nix-shell

If you want to use your favorite shell (e.g. fish) installed on your system:

nix-shell --run fish

Use direnv to automatically set up the environment when changing to this project's directory.

Getting started

This project contains missing pieces that you will have to write, with some guidance, to make it compile again.

In the files that require editing, you will find comments such as # section 1 code to add here helping you ensure that you are looking at the right place for the current exercise.

The lab script can be used if you need to compare your changes with the solution.

Feel free to open an issue to ask questions, or if you think something is wrong in the codelab.

Section 1: Hello, Bazel!

Build an example Java application that prints

Welcome to the Bootcamp! Let's get going :)

to the console.

1. Edit java/src/main/java/bazel/bootcamp/BUILD.bazel
2. Add a binary target for HelloBazelBootcamp.java
   • use java_binary()
   • name it HelloBazelBootcamp
3. Run the binary using bazel run //java/src/main/java/bazel/bootcamp:HelloBazelBootcamp
   • WARNING: The first time you run this command, it may take quite a long time. Bazel is downloading and installing all the dependencies required to build a Java project.

Section 2: Go server

Build a Go server that receives log messages in the format defined in proto/logger/logger.proto.

1. Edit proto/logger/BUILD.bazel
   1. Add a target for the protocol description
      • use proto_library()
      • name it logger_proto
      • ignore the deprecation warning in the documentation, the API is still valid
   2. Add a library target for a Go protobuf library based on logger.proto
      • use go_proto_library()
        • name it logger_go_proto
        • importpath
          • allows importing the library from a known package path in server.go
        • compilers
          • setting to ["@rules_go//proto:go_grpc"] builds functions implementing the Logger service for gRPC
2. Edit go/cmd/server/BUILD.bazel
   1. The go_library target server_lib is already written for you
   2. Add a binary target from server.go
      • use go_binary()
      • name it go-server
      • it should be compiled together with the sources of server_lib
3. Run the go binary using bazel run //go/cmd/server:go-server
   • run implies the build step
4. Check http://localhost:8081
   • it should display

     "No log messages received yet."

Section 3: Java client

Build a Java client which sends log messages to the server, in the format defined in proto/logger/logger.proto.

1. Edit proto/logger/BUILD.bazel

   1. Add a Java library for data structures in logger.proto
      • use java_proto_library()
      • name it logger_java_proto
   2. Add a Java library implementing the gRPC service Logger defined in logger.proto
      • use java_grpc_library()
      • name it logger_java_grpc

2. Edit java/src/main/java/bazel/bootcamp/BUILD.bazel

   1. Add a Java library implementing the logging client
      • use java_library()
      • name it JavaLoggingClientLibrary
      • use JavaLoggingClientLibrary.java as source
      • declare depencies on *_proto and *_grpc targets created in previous steps
      • since JavaLoggingClientLibrary uses the io.grpc package, you will also need to declare a dependency on @grpc-java//core, as well as a transport impementation such as @grpc-java//netty.
   2. Add a Java binary for the client
      • use java_binary()
      • name it JavaLoggingClient
      • use JavaLoggingClient.java as source
      • declare a dependency on JavaLoggingClientLibrary

3. Run the Java binary with

   bazel run //java/src/main/java/bazel/bootcamp:JavaLoggingClient
   

   • workaround for an open nixpkgs issue on macOS

     env CC=gcc bazel run //java/src/main/java/bazel/bootcamp:JavaLoggingClient
     

     Note that it will not work by fixing CC to either clang or clang++, as both C and C++ dependencies are present!

4. Run the Go binary from Section 2 from another nix-shell

5. Type messages to send from the client to the server and view them on http://localhost:8081

Section 4: Java client unit tests

1. Edit java/src/main/java/bazel/bootcamp/BUILD.bazel
   1. Add a test target for JavaLoggingClientLibrary
      • java_test()
      • name it JavaLoggingClientLibraryTest
        • names matter for tests!
        • target name, test class, and file name must be identical
      • use JavaLoggingClientLibraryTest.java as source
      • add JavaLoggingClientLibrary target from Section 3 as dependency
   2. Repeat the same for the client binary JavaLoggingClient
2. Run tests
   • NOTE: this is not a unit test, it depends on the server to make any sense.
     • If you skipped section 2 exercise, you can simply run lab install go/cmd/server/BUILD.bazel to get the solution of this exercise and bazel run //go/cmd/server:go-server to run it.
   • start Go server in another nix-shell, as in Section 2
   • run Java client tests

     bazel test //java/src/main/java/bazel/bootcamp:JavaLoggingClientLibraryTest
     

     • NOTE: If you simply turn off the server and rerun the test, you will most likely see that the test passed, with the information that it is "cached". Indeed, Bazel does not rerun the tests if none of its dependencies changed. If you really want to rerun it to acknowledge that it fails when the server is off, you have to specify --cache_test_results=no in the command line.
   • Do not forget to run the tests of JavaLoggingClient
     • NOTE: Those tests do not use the server.

Section 5: Typescript web frontend

1. Edit proto/logger/BUILD.bazel
   1. Add a TypeScript protobuf library implementing logger.proto
      • use ts_proto_library()
      • name it logger_ts_proto
2. Edit typescript/BUILD.bazel
   1. Add a target for the TypeScript app app.ts
      • use ts_project()
      • add logger_ts_proto as a dependency
      • set transpiler = "tsc"
      • you need to reference the local tsconfig.json
   2. Bundle the JavaScript modules using esbuild
      • use esbuild()
      • name it bundle
      • set a valid entrypoint (ts_project generates a file with the ending .js)
   3. Add a target for a development server
      • use js_run_devserver()
      • name it devserver
      • add the esbuild target and index.html as data dependencies
      • set args = ["typescript"] to serve files from the typescript directory
      • set tool = ":http_server" to make it work correctly
3. Run the webserver using bazel run //typescript:devserver
   • it will print a link which you can click on
   • if the link does not work, try http://localhost:5432
4. Run the Go server and Java client from the previous steps.
   • send messages from the Java client to the Go server
   • click the button on the web front end to see them in your web browser

Section 6: Integration test

1. Edit tests/BUILD.bazel
   • add a shell test target for integrationtest.sh
   • use sh_test
   • add the Go server and Java client targets from previous steps as data dependencies
2. Run the test using bazel test <target> and make sure that it passes
   • NOTE: You may have to modify your solution of Section 2 and 3 to make those tests pass. In particular, you most likely did not set the visibility of the binaries.
3. Run the test multiple times using bazel test <target> --runs_per_test=10
   • QUESTION: Does it pass? If not, can you figure out why?
   • HINT: the section on test run behaviour in the tags documentation may prove useful.

Section 7: Code formatting with buildifier

1. Edit BUILD.bazel
   1. Add a load instruction to import the rule buildifier.
      • NOTE: the documentation linked above assumes the use of the upstream buildifier rules. This project currently depends on a fork that ships precompiled binaries and is already available as a Bazel module.
   2. Create a target called buildifier-print which warns about lint error and only prints the changes buildifier would make.
      • NOTE: Adding such a rule will provide a warning that using mode = "diff" is deprecated, and suggesting to use a buildifier_test rule instead. There is a known issue preventing a buildifier_test from being run on every folders of a project, hence, for now, let's simply ignore this deprecation warning.
   3. Now create a target called buildifier-fix which fixes all the formatting errors (including linting ones).
   4. Run the 2 targets to format your solution to the exercises.

Section 8: Query dependency graph

1. Read the Bazel Query How-To and try the examples to display the dependency graph of the packages in this exercise.
2. SOLUTION: This exercise is solved here.