[Spike] Create monorepo for juggler and connectors

[emonddr]

Description / Steps to reproduce / Feature proposal

Create a monorepo as a POC to store/test juggler and connectors together.

import juggler as-is using lerna import
import one connector using lerna import

Create a 3.x branch and a 4.x branch.

In 3.x branch, test juggler 3.x with version 3.x of the connector.
In 4.x branch, test juggler 4.x with version 4.x of the connector.

Acceptance Criteria

• get a test suite for 3.x to run locally and in Travis
• get a test suite for 4.x to run locally and in Travis

[emonddr]

This is related to: https://github.com/strongloop-internal/scrum-apex/issues/416

[raymondfeng]

I created a PoC at https://github.com/raymondfeng/connector-next.

A few observations:

1. The --flatten option is required to run lerna import for juggler and connector repos
2. I wish lerna import can rewrite the commit message with a prefix

[raymondfeng]

I was able to use git subtree add --prefix=packages/loopback-connector-postgresql [email protected]:strongloop/loopback-connector-postgresql.git master to import loopback-connector-postgresql repo.

[raymondfeng]

Another interesting approach - https://github.com/ingydotnet/git-subrepo

[bajtos]

Few comments on this approach:

1. At the moment, we a single version of a connector (e.g. [email protected]) supports multiple juggler versions (e.g. loopback-datasource-juggler versions 2.x.y, 3.x.y and 4.x.y). In strongloop-internal/scrum-apex#416, I want to find a way how to test the same connector code against multiple juggler versions. How do you envision to address this aspect in a monorepo?

2. At the moment, we have 27 repositories with a name starting with loopback-connector- (see here query). If each connector requires 3 minutes to run the test suite, and we import all connectors to the monorepo, then every CI build will take 3*27=81 minutes. I find this as unacceptable slow down of our development process. I see two solutions: (1) incremental build & test (test only things affected by the pull request - e.g. don't test MongoDB connector when the pull request is fixing MySQL only) (2) run tests in parallel (preferably on multiple virtual machines!). We need to find or build a solution for running tests this way, effectively replacing infrastructure provided by cis-jenkins tooling. It's important to account for this work in our plans to migrate connectors to a monorepo.

3. Based on the work we have done so far in loopback-next, I think many (if not all) parts of juggler will become obsolete soon. At that point, it may be better to move connectors to the main loopback-next monorepo and say good bye to loopback-datasource-juggler. In that light, I feel that moving juggler & connectors to a new monorepo is not worth the effort right now.

[bajtos]

run tests in parallel (preferably on multiple virtual machines!)

This may be relatively easy to achieve using Travis stages & tasks, see e.g. https://boneskull.com/mocha-and-travis-ci-build-stages/ for inspiration.

[bajtos]

Another question: how are we going to publish new releases from the monorepo?

So far, we are using slt-release to make individual releases of juggler or connectors.

Will we keep doing that in the new monorepo? Are we going to switch to conventional commits & release process, as we do in loopback-next?

From what I've seen so far, this creates a lot of noise, because lerna is not able to release only a package that has been changed, it always publishes a new release of every package :(

---

To put my questions in a wider context: there are many ramifications when moving to a monorepo and hidden costs that we will have to pay down in the (near) future. I would like us to be aware of them and account for the additional effort when deciding whether to pursue the monorepo approach at this moment in time.

[dhmlau]

Closing in favor of the spike result from https://github.com/strongloop-internal/scrum-apex/issues/416.