Move (back) to a Monorepo

[devjgm]

TL;DR: All the google-cloud-cpp-* GitHub repos will be combined into this repo as a single "monorepo" in Q2-2020. We apologize for any confusion this may cause our customers, but we think this will be much simpler for our customers going forward. This should not result in any build breaks for our customers, but in order for customers to pick up the newest monorepo versions of our libraries, they will need to make minimal changes to their build scripts.

Background

This GitHub repo is currently a "monorepo" in the sense that it contains the code for several client libraries. It contains the GCS client code, the Bigtable client code, and the beginnings of a Firestore client. However, our plan has been to change to a multi-repo approach where each client library lives in its own GitHub respository. This plan was outlined in #3350.

We started doing the multi-repo approach about a year ago with with our -spanner repo. At this point we now have the following GitHub repos:

• https://github.com/googleapis/google-cloud-cpp (this "mono" repo)
• https://github.com/googleapis/google-cloud-cpp-spanner
• https://github.com/googleapis/google-cloud-cpp-common
• https://github.com/googleapis/google-cloud-cpp-pubsub
• https://github.com/googleapis/google-cloud-cpp-bigquery

These separate repos are becoming exceedingly difficult to manage, there's lots of duplication across repos, and redundant work to keep things in sync, and the separate repos make it difficult for customers to explore all the C++ clients that we have available. This latter point will become an even bigger problem as the number of supported client libraries increase.

Given the new data we have from the experience of managing multiple repos for about a year, we have decided to reverse this decision and to instead move back to a monorepo.

The Plan

Our plan is to combine all of the "google-cloud-cpp-*" projects into a single GitHub repo, which will be this repo. The code, samples, and documentation for each client library will live in its own directory at google/cloud/$library, for example, google/cloud/bigtable, google/cloud/storage, and google/cloud/spanner.

Our plan is to begin and complete this work in Q2-2020 (roughly April - June, 2020). The work to do this is captured in the following milestones:

1. Optimize CI Builds
2. Prepare to be a monorepo
3. Prepare builds for incoming repos
4. Move google-cloud-cpp-pubsub
5. Move google-cloud-cpp-bigquery
6. Move google-cloud-cpp-spanner
7. Move google-cloud-cpp-common

Expected User Impact

We apologize for any confusion or difficulty this change may cause our customers. We are committed to minimizing this friction. We also believe that the end result will be simpler, clearer, and better for our current and new customers going forward.

We know this change impacts existing users, but we believe users will be able to migrate to this new scheme with small amounts of effort. We anticipate that no changes to the C++ code will be needed. We expect that some changes to the build and/or packaging scripts will be needed.

Users Downloading from GitHub

We expect that a number of our customers download the source from GitHub and then incorporate the code into their build scripts. Existing releases of google-cloud-cpp and the other google-cloud-cpp-* repos will continue to work, and will have exactly the same content as before. Therefore, customers who have pinned their scripts to use a release will have no impact until such time as they decide to upgrade to the new version in the new "monorepo". Users who download master will break immediately, but we hope this is a small number, and that they are cognizant of the risks of depending on a non-release branch.

Changes for CMake Users depending on pre-installed libraries

No changes to their CMakeLists.txt files should be needed for these customers. Some changes to their build / installation scripts would be needed at the time they decide to upgrade.

Changes for CMake Users with Super Builds

These customers will need to change their top-level CMakeLists.txt to add new external projects.

Bazel Users

Bazel users will need to change their WORKSPACE file to introduce google-cloud-cpp as a new dependency.

Package Maintainers

Package maintainers who use google-cloud-cpp-spanner will need to create a new package for google-cloud-cpp, if one does not already exist.

[devjgm]

Moved this issue to the "Final Cleanups" milestone so that we can keep this issue open during the entire monorepo move, while still closing other milestones.

[svasan]

Hi, I'm using v0.19.0 of the storage client and want to upgrade at some point to at least catch up to the initial mono-repo releases (mostly to make it easier to upgrade later on if needed). I see a new version 1.13.0 which seems to combine pubsub, common, bigquery and common. Is this the first stable monorepo release and something I can upgrade to or should I wait some more.

[coryan]

1.13.0 is stable, and something you can update to. It does not include spanner, that will come with 1.14.0. Whether you want to wait until then is entirely up to you.

[svasan]

When is 1.14.0 expected? Looking at the release history, it looks like there is a release in the beginning of each month which would make the next one early June. Is that the current plan?

I don't need spanner, so 1.13 should be okay but I'll probably update next week and if the next release is around the corner, would just wait to get that.

[coryan]

Yes, we expect to cut the 1.14.0 release early in June.

[devjgm]

@svasan FYI, the 1.14.0 release is now available at https://github.com/googleapis/google-cloud-cpp/releases/tag/v1.14.0