Skip to content

Latest commit

 

History

History
272 lines (198 loc) · 12.2 KB

CONTRIBUTING.md

File metadata and controls

272 lines (198 loc) · 12.2 KB
Raw

Contributing to Nessie

How to contribute

Everyone is encouraged to contribute to the Nessie project. We welcome of course code changes, but we are also grateful for bug reports, feature suggestions, helping with testing and documentation, or simply spreading the word about Nessie.

There are several ways to get in touch with other contributors:

More information are available at https://projectnessie.org/develop/

Code of conduct

You must agree to abide by the Project Nessie Code of Conduct.

Reporting issues

Issues can be filed on GitHub. Please use the template and add as much detail as possible. Including the version of the client and server, how the server is being run (eg docker image) etc. The more the community knows the more it can help :-)

Feature Requests

If you have a feature request or questions about the direction of the project please join the slack channel and ask there. It helps build a richer discussion and more people can be involved than when posting as an issue.

Large changes or improvements

We are excited to accept new contributors and larger changes. Please join the mailing list and post a proposal before submitting a large change. This helps avoid double work and allows the community to arrive at a consensus on the new feature or improvement.

Code changes

IntelliJ IDEA tips

Nothing special for IntelliJ IDEA, just trust the project and let IntelliJ import it as a Gradle project.

Gradle tips

Common Gradle tasks:

  • Check whether everything compiles (no style checks): ./gradlew jar testClasses
  • Automatically fix code style issues: ./gradlew spotlessApply (abbreviated: sAp)
  • Publish to local Maven repo: ./gradlew publishToLocalMaven (abbreviated: pTML)
  • Run unit tests: ./gradlew test, also ./gradlew build
  • Run integration tests: ./gradlew intTest
  • Run all checks (including tests): ./gradlew check

It is fine to just run all test tasks, Gradle will only execute a task, when anything that the task depends on has been changed.

Abbreviations

You can abbreviate project and task names, see docs:

  • For ./gradlew spotlessApply you can write ./gradlew sAp
  • For ./gradlew :nessie-versioned-persist-serialize-proto:tasks you can write ./gradle :n-v-p-s-p:tasks

Local Maven repository

Using the local Maven repository is discouraged.

If you really have to use the local Maven repository, you can use it by explicitly instructing the build to do so by passing -DwithMavenLocal=true. Be aware that Gradle does not cache anything from the local Maven repository and builds will be significantly slower.

Note: -DwithMavenLocal=true allows using the local Maven repository for dependencies. This is different from the Gradle task publishToMavenLocal, which publishes the Nessie artifacts to the local Maven repository.

Improving the build time for native images

The native image builds require quite some amount of Java heap for the native image compilation. The value can be bumped by putting for example the following value to your local ~/.gradle/gradle.properties file:

quarkus.native.native-image-xmx=8g

Heap size and JVM arguments for tests

Tests run with the default Java heap size, which is sufficient for all tests. If you really need to bump the heap size for tests, for example during development, you can do so via the project property testHeapSize. For example: ./gradlew -PtestHeapSize=4g :nessie-client:test --tests TestMyStuff

The project property testJvmArgs allows specifying JVM arguments for tests. Example: ./gradlew -PtestJvmArgs="-Xmx8g -XX:+UnlockExperimentalVMOptions -XX:+UseZGC" :nessie-client:test --tests TestMyStuff. Multiple JVM arguments can be specified via the testJvmArgs property, separated by spaces.

Note: if you need these JVM settings regularly, you can also specify those in the ~/.gradle/gradle.properties file or set those via the environment, for example via export ORG_GRADLE_PROJECT_testHeapSize=4g.

javadoc task fail

If the javadoc task fails, check that there's a javadoc executable. If not, install a "full" JDK.

Migrating an existing Nessie clone using Maven

In June 2022 the Nessie code tree changed to the Gradle build tool. Existing clones using Maven can be migrated as follows:

  1. Close your IDE
  2. Run git clean -xdf
  3. Run git pull
  4. Run ./gradlew testClasses to ensure the build works fine. The very first build will be slower, because it has to assemble the build plugins and download dependencies.
  5. Make sure that your IDE has no more "references to Maven" (nothing to do when using IntelliJ)
  6. Open Nessie in your IDE

Note: The Gradle build does not use the local Maven repository and dependencies there will not be used by default. See Local Maven reposotiry.

Development process

The development process doesn't contain many surprises. As most projects on github anyone can contribute by forking the repo and posting a pull request. See GitHub's documentation for more information. Small changes don't require an issue. However, it is good practice to open up an issue for larger changes. If you are unsure of where to start ask on the slack channel or look at existing issues. The good first issue label marks issues that are particularly good for people new to the codebase.

For the Spark tests to run with Java 16 or newer, you need to have Java 11 installed. Gradle will most likely find the Java 11 runtime required to run the Spark tests. Run ./gradlew javaToolchains to see the Java toolchains that Gradle discovered. If Gradle could not locate your Java 11 runtime, consult the docs.

Building with Java 17 (and 16)

Due to JEP 396, introduced in Java 16, a couple JVM options are required for google-java-format and errorprone to work. These options are harmless when using Java 11.

Apache Spark does only work with Java 11 (or 8), so all tests using Spark use the Gradle toolchain mechanism to force Java 11 for the execution of those tests.

Development environments

Nessie's primary development environment is Linux. The code base can be built on macOS and Windows. All tests, unit and integration tests, must run and successfully complete on Linux. Some tests do not work on macOS and/or Windows. Primary reason is that Docker is not natively available on those platforms and some integration tests run into issues. Another reason are obvious and non-obvious platform differences.

TL;DR It is fine to build code on macOS and Windows, but the reference platform is Linux.

Running tests on macOS

In our CI we use Podman for macOS, initialized using the following sequence of commands, which can be used locally as well.

# Install podman using Homebrew
brew install podman
# Initialize the default instance
podman machine init
# Start the podman machine
podman machine start
# Symlink `docker` to `podman` (rather a convenience)
ln -s /usr/local/bin/podman /usr/local/bin/docker
# Export the DOCKER_HOST with the location of the Unix socket
export DOCKER_HOST="unix://${HOME}/.local/share/containers/podman/machine/podman-machine-default/podman.sock"

The "Docker machine" is actually a virtual machine running Linux and therefore CPU and memory are constrained. If you encounter issues when running integration tests that use Docker, consider limiting the number of concurrent integration tests by setting / using the system property nessie.intTestParallelism to a low value (1 or 2), the actual value depends on your machine.

Testcontainers via Podman regularly fails to fetch information about the requested Docker image. Unfortunately, there is no workaround, because the issue happens even if the images are present locally. The error during integration tests looks like this:

caused by ... ContainerFetchException: Can't get Docker image: RemoteDockerImage...
caused by ... NoHttpResponseException: localhost:2375 failed to respond

Running tests on Windows & WSL2

Developing Nessie on Windows (not in WSL) is possible, but not supported. Since testcontainers does not support Windows (not WSL), running a lot of important integration tests that use Docker is not possible. Running those tests inside WSL2 should work, but has not been verified.

Windows line ending issue in nessie-ui repository

Nessie UI is in the projectnessie/nessie-ui repository.

Windows uses a different line ending (CR LF) than Linux and macOS (LF). The prettier tool that we use in the ui/ module for linting however expects LF even on Windows. This is an inconvenience at the moment, meaning that you cannot run tests and checks on the Nessie UI module.

It would be possible to work around the limitation by setting the following Git configuration options and re-checkout Nessie. But this is neither tested nor supported. We use the approach in CI though.

git chckout ca95c806f43f470b9f17e4cc9305c1cc0910e55d
git config core.autocrlf false
git config core.eol lf
git checkout main

Style guide

Changes must adhere to the style guide and this will be verified by the continuous integration build.

Java and Scala code style is checked by Spotless with google-java-format and scalafmt during build.

Python code style is checked by flake8/black.

Automatically fixing code style issues

Java, Scala and Kotlin code style issues can be fixed from the command line using ./gradlew spotlessApply.

Python code style issues can be fixed from the command line using

cd python/

[ ! -d venv/ ] && virtualenv venv
. venv/bin/activate
pip install -U -r requirements_lint.txt
 
black pynessie tests

Configuring the Code Formatter for Intellij IDEA and Eclipse

Follow the instructions for Eclipse or IntelliJ, note the required manual actions for IntelliJ.

Code coverage

Code coverage is measured using jacoco plus codecov.

Submitting a pull request

Upon submission of a pull request you will be asked to sign our contributor license agreement. Anyone can take part in the review process and once the community is happy and the build actions are passing a Pull Request will be merged. Support must be unanimous for a change to be merged.

All pull-requests automatically trigger CI runs. Two long-running parts of the CI workflow are skipped for PRs by default, but can be enabled using "labels" on the PR.

  • Quarkus native image generation + tests against the native image. The label pr-native label enables this. The label pr-native label enables this, CI results do not appear as a separate job, because those run as part of the "Java/Maven" workflow job.
  • Nessie-client tests against various combinations of Jackson versions. The label pr-jackson label enables this and CI result will appear as a separate check.

Reporting security issues

Please see our Security Policy