proposal: Release OpenDAL Components Seperately

[Xuanwo]

Summary

Release OpenDAL Components Seperately with different version for each component.

Motivation

OpenDAL has a lot components like

• bindings: c, cpp, java, python, nodejs, ...
• applications: ofs, oay, oli
• integrations: object_store, dav-server

They are at different stages of development. The bindings for java, python, and nodejs have reached a level of maturity where they can be released with the core for every version update. However, most of them are still in development and do not have regular releases. I believe they cannot reach maturity until we expose them to users and gather feedback. We should release them separately from the core, assigning most of them a version number like 0.1.0 or even 0.0.1.

In addition, mature components encounter their own issues. Bindings such as java, python, and nodejs require a minor version update for each change in the opendal core. It is important to remember that in the semver world, this constitutes a breaking change for version 0.x. We must discover a method to enable bindings like java and python to maintain their own versions so that users can make informed choices.

Besides, our current style has encountered issues with certain bindings such as ocaml: #4057.

Proposal

So I propose to release opendal components seperately with different version for each component.

Instead of a single large release of apache-opendal, we will be releasing apache-opendal-core, apache-opendal-binding-python, and apache-opendal-bin-oli separately. This allows users to build and verify each component individually, without needing to refer to ../../core or similar paths.

Each component will have its own version with embedded build metadata. For example, under the tag 0.50.0, we will have:

• apache-opendal-core-0.50.0
• apache-opendal-binding-python-0.46.0+core.0.50.0: python binding depends on core v0.50.0
• apache-opendal-bin-ofs-0.0.1-core.0.50.0: app ofs depends on core v0.50.0
• apache-opendal-binding-go-0.0.1+binding.c.0.50.0: binding go depends on c binbding v0.50.0

If there are no API changes in the Python binding, we can keep its version unchanged and only update the build metadata of the core to a new version. For languages like Node.js that do not consider build metadata for version resolution, we can simply increment the patch version instead.

The entire release workflow remains unchanged. We still release Apache OpenDAL as a whole. The difference is that during the release discussion, we need all code owners of the components to decide which version to use in their respective component. The Release Manager is responsible for updating them accordingly.

We will modify release.sh to generate all the tarballs with the correct version. Additionally, we will include a release note for each component version along with their corresponding change logs.

Drawbacks

Tag on repo is not the same with component version

After this change, the tag on the repository is for the entire Apache OpenDAL project instead of just a component. It's possible that the Python binding has version 0.46.0, but the repository has a tag v0.50.

Rationale and alternatives

Don't change release style

We can maintain our current processes without making any changes.

Release every component seperately

We have over 10 components, but our community is not prepared for releasing them individually. That means we need to set up the release process for each component and vote on them individually, which is not interesting.

Split opendal monorepo into different repos

A monorepo is beneficial for development as it allows for increased visibility. Having separate repositories makes it more challenging for maintainers to track and involves tedious setup work, such as tests.

Prior art

None

Unresolved questions

None

Future possibilities

None

[simonsan]

WRT 'Tag on repo is not the same with component version':

gitoxide is a monorepo with a lot of component/crates, maybe it is helpful to make this decision. Here are their releases to get an impression. https://github.com/Byron/gitoxide/releases

[Xuanwo]

Thanks for sharing. My proposal sticks to a single tag for:

• One page for all component changes.
• Simplify our release management process.
• Simplify our release workflows.

The main difference is that I plan to release the entire opendal project as a whole, rather than releasing individual components one by one. This will help reduce our burden for verification and voting.

[tisonkun]

apache-opendal-binding-python-0.46.0+core.0.50.0

I don't like this version strategy. Either a binding should be compatible with a version range of core package, or we do releases simultaneously. Fixing the core version for a binding version increases the burden for downstream adopting, or if we do fix the core version, we omit the core version so that downstream users just pick against binding version.

The point here is whether we make core package version associated to the bindings' version a public interface and how we define its semantic.

[tisonkun]

The most important thing is that the OpenDAL project is always released as a whole at the same time, so there's no range of core versions to choose from.

Thank you! It looks clear to me that we don't release packages totally separately, but use a different semver strategy to help downstream users to understand the semantic of the version. Then it should work :D

[tisonkun]

While IIRC Cargo doesn't read build metadata at all, I'd refer to Protobuf's version strategy where they can introduce breaking changes for specific languages implementation - https://protobuf.dev/support/version-support/.

The protobuf core remains using "feature.patch" pattern but it may not be suitable for OpenDAL Rust core.

However, as you state "Node.js completely ignores them" and it can be a more complicated issue, while I suppose Java (Maven) just ignore the semantic but treat the version string as a literal so it happens to work.

We may leave the discussion open for a while and highlight these challenges. Before we meet a real requirement to split the version we can use a synchronized version number for all bindings & core for now.

[Xuanwo]

However, as you state "Node.js completely ignores them" and it can be a more complicated issue, while I suppose Java (Maven) just ignore the semantic but treat the version string as a literal so it happens to work.

This part is fine from my point of view. I feel like we can bump a patch version if no meaningful changes to nodejs or java.

[Xuanwo]

While IIRC Cargo doesn't read build metadata at all

Sorry for not make this part more clear. While I'm talking about build metadata, I'm saying the build metadata in semver instead of in cargo. This means that it also a part of versioning semantics.

For example, librocksdb uses the same way to distinguish itself API changes and rocksdb's API changes: https://crates.io/crates/librocksdb-sys/versions

[Xuanwo]

The semver spec said:

Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85, 1.0.0+21AF26D3----117B344092BD.

So I think we can always bump a patch version of only core changed for other components.