[RFC] - Carbon v10 Migration Strategy

[tay1orjones]

I've attempted to keep this as brief as possible, but wanted to capture some of the various concerns and ideas we've heard on this topic.

This is a request for comments - a proposal of what we could do. Please leave feedback in the comments below.

Summary

Carbon has updated to v10, the components as of today in this package are built with v9. Carbon v10 contains breaking changes which can make adoption difficult. In addition, products may opt to upgrade in sections, so a massive v10 adoption may not be practical. The components here need to have the Carbon components they rely on updated to Carbon v10 without impacting current consumers of the v9 version.

Motivation

Portfolio unification is an imperative for the products that consume this package. Without a v10 migration, stagnation of the components will cause product teams to develop their own v10-compatible components, instead of contributing back here.

Unifying the efforts of our product teams is also crucial - product feature deliveries will slow if individual product teams each have to develop their own custom v10 implementation of the Watson IoT Design System and associated components.

With a migration plan in place and features to support incremental adoption, we can continue to house all our components in one place across multiple Carbon versions.

Detailed design

@davidicus and I have discussed some potential solutions to this problem. The proposal we think might work best consists of 5 primary steps:

1. Implement a feature flag system
2. Create a branch to house all v10 updates to our components
3. Update our components to support Carbon v10 behind a feature flag
4. Publish the new branch to npm under alpha/pre-release/@next tag
5. Write documentation to support the migration path
6. Merge into master

1. Implement a feature flag system

We can export a new feature flag module to enable set/get of feature flags.

The Carbon team has these, we could follow a similar configuration:

export default FeatureFlag = {
 buttonEnhanced: 'v9',
 table: 'v9',
 // ... another component, etc
};

Internal Component Conditional Example:

import {FeatureFlag} from 'carbon-addons-iot-react';

const ButtonEnhanced = props => {
  const { children, loading, disabled, className, ...other } = props;
  return FeatureFlag.buttonEnhanced === 'v10' ? (
    <StyledButton
      {...other}
      className={className}
      disabled={disabled || (loading !== undefined && loading !== false)}>
      {loading ? <Loading small withOverlay={false} /> : null}
      {children}
    </StyledButton>
  ) : (
    <StyledButtonv10
      {...other}
      className={className}
      disabled={disabled || (loading !== undefined && loading !== false)}>
      {loading ? <Loading small withOverlay={false} /> : null}
      {children}
    </StyledButtonv10>
  );
};

External Consumer Example:

// ...
import { FeatureFlag } from 'carbon-addons-iot-react';
FeatureFlag.buttonEnhanced = 'v10'; // or could be set at root, or their own encapsulated feature flag component or module
// ...
return (
  <div>
    <p> An Example v10 ButtonEnhanced</p>
    <ButtonEnhanced>
  </div>
);

These feature flags would be v9 by default. At some point in the future we can switch the flag to 'v10' by default. Even further down the road, we can deprecate v9 support altogether and remove the v9 conditions in the components.

Definetly open to suggestions here on the format of these feature flags. Could alternatively do a boolean structure, buttonEnhanced@next = true || false

2. Create a branch to house all v10 updates to the components

A longstanding new branch will house all work updating the components to v10. The first step will be to update the dev deps for carbon to v10 and squashing any compliation/build bugs.

This will allow work on components on v9 to continue as we work to bring all the components stable under Carbon v10.

3. Update our components to support Carbon v10 behind a feature flag

Components will be examined one by one and made to be compliant from a design perspective in the new branch. PRs should be reviewed by designers to ensure there aren't any defects.

4. Publish the new branch under alpha/pre-release/@next tag

Manually on a case-by-case basis, we should release these updated components for testing in products/consumers codebases. Potentially after a component has been completed and updated to v10. Also, for teams who are already on v10 and just waiting for us to support v10, they will be able to consume these components as they become available.

Consumers who want to use both v9 and v10 versions of our components can use yarn's aliasing feature to install both versions.

5. Write documentation to support the migration path

Installation steps will need to be revised. This is also a good opportunity to improve our getting started documentation and more clearly state how to contribute, what our purpose is, etc.

Details on consuming/testing the pre-release version should be included.

A table showing status of v10 adoption would be beneficial, to know at a glance which components are v10-ready.

6. Merge into master

Once all components are updated, finalize the update by merging the pre-release branch into master, cut a new major version in github/npm. Carbon v10 will be the offically 'supported' version moving forward. v9 development can continue, but is discouraged and will require a manual minor/patch release to the previous major version.

See unresolved questions below regarding v9 bugfixes and peer dependency versions.

Drawbacks

• All work has to be done in a separate branch, any updates to components made during this time will require manual merging of master into the pre-release branch, solving any conflicts and retaining new fixes/features. This might get really difficult to manage if there's a ton of work happening in v9.
• Pre-releases are manual
• ... I'm sure there are others, please leave feedback!

Alternatives

Some things discussed:

• Refactoring to a monorepo structure
  • Each component would be it's own published and versioned package. Would allow us to adopt v10 component-by-component, but would have the same drawbacks with the added cost to refactor into a monorepo structure plus set up the CI/Deployments for that.
• Updating components "live", one-by-one, to v10. Effectively halting v9 support for a component immediately after it's v10 update PR is approved and merged.
  • Consuming projects would be required to install Carbon v9 and v10 by using yarn's aliasing feature to install both versions at the same time. Sorta messy, and this would likely increase the consumer's bundle size by quite a bit.
  • Unable to provide bugfixes for v9 after a component has been updated to v10.

Adoption strategy

After the pre-release branch is merged, consumers would update to Carbon v10, update our package, and things should "just work".

The components' overall API shouldn't change with this update, but we should anticipate encoutering bugs or defects. The update is mainly a visual/styling change from what I can tell on the components we have today. Although that might not entirely be true for heavily extended components like Table.

How we teach this

Overhaul the readme.md to include clearer getting-started documentation, the purpose of this repo, how to contribute, and links to a small migration guide.

As consumers update and run across issues, we can tag them with a new label in GitHub to understand the continued impact of this change.

Unresolved questions

• Should we leave the Carbon dev dep version at 9 even after our components are upgraded to use v10? - If we update dev deps to v10, what would be the workflow to develop bugfixes for v9 components?
• Should feature flags default be set to 'v10' once the pre-release branch is merged into master?
• How does this process fall in line with new components being added to the system, compliant with v10 from day-1. Similarly, will there be new components added without v10 support?

[stuckless]

@tay1orjones Not sure I understand the reasoning to use both branches and feature flags? If I'm in the V10 branch, then I'm wanting to use the V10 components, so why would I need to set a feature flag? Just seems like unnecessary work for you and me. Maybe I'm missing something? It would seem that merging from V9 to V10 would be almost impossible with feature flags existing in V10 and not in V9. ie, the Table component in V10 will be vastly different with introduction of the feature flag in the main component.

As part of the V10 branch, I assume there is a pipeline to publish the V10 Storybook? (otherwise how will people review the V10 components)

As for 1 repo per component. I go back and forth on this. Conceptually I like the idea, but, practically it almost never makes sense, and generally makes things harder to consume. Something that I've considered in our project is to have a base repo for the "common" component, but, put more volatile components like charts, and tables, into a separate repos. Using multiple repos also makes is harder for ZenHub integration and issue tracking. ie, sometimes issues get put into the wrong repo.. zenhub has a limit on the # of repos, etc.

[lukefirth]

I'll speak from a personal, UX practitioner perspective and try to keep away from the things I know little about (the cost vs benefits of monorepo vs alternatives for example). @tay1orjones Let me know if this isn't the apt place for these thoughts/comments and I can happily move them to a general discussion space or something 😄

First thing I believe we need is a commitment from teams on when they will target V10 adoption. Once this date has been established and agreed, then we call that our 'V9 halt of support day'. At this point, we resolve to only maintain V10 versions of all of our custom components. Equally, designers can then gauge release dates of whatever flow they are working on, and understand if they should design it with v9 or v10 in mind.

Topic 1: Component by component upgrades vs The whole hog.

• While we could follow a path of moving UI over to V10 on a component by component basis to slice up and contain the work. Personally at least (and I think others will echo this) I believe this would bring more work and complexity in total.

• From a purely design side I would says its far easier to make the jump and refactor all the flows and screens to comply with V10 than to evolve them over time with given components (requiring us to keep track of when a flow may be released, and which components will be in which version at that time).

• Also, from both a design and I suspect an implementation side, the piecemeal approach would likely involve doing a lot of things to make a V10 component fit nicely in a predominantly v9 environment (tiny example - button colors would likely need to be the v9 color until later on even if the component was v10), most of these 'fit' changes would eventually need to be redone or removed when its a generally v10 context. Adding all these up would probably equate to a lot of extra person days all around.

• Because of this, from my personal perspective: A route of creating a second branch for V10 and creating V10 versions of the existing components (plus new) makes sense. Teams stick with their v9 branch until they are up and ready to move to V10. The longer past V9 halt of support day they go, the more out of date the V9 set will get, putting more and more value on them migrating to V10.

Topic 2: If we do that, how should teams transition from 9 to 10?

1. Firstly, designers on all teams should begin work on V10 designs of all of their key flows from their existing products being careful to minimize changes that aren't direct component swaps - we all have things we wish we could have done differently, but this is not for fixing designs, but instead for enabling the simple migration to be as simple as possible for front end teams without them having to constantly go back to ask UX&Viz how to resolve conflicts where v9 can't just be swapped with v10.

The benefits of this are threefold:

• To give development teams who are trying to migrate an answer to their questions on design breaking changes that come with V10. They can run asynchronously from design upgrading screens from 9->10 and use the reference designs as guidance.
• To give designers the screens they can use as a base for new things going forwards. (Eg: when I start some work, usually I'm extending one of my existing UX flows, so I grab my v9 screens for it and build up. Doing this means I can start from V10 screens).
• To give us all a bit of practice in applying V10 without simultaneously rethinking flows.

2. Secondly the hard part, and again I'm speaking UX only and I'm sure there's other factors that might make this less desirable than it is to me. Each product moving up from V9-V10 would try to progress through the process as quickly as possible, but more importantly, they should create a branched version of their UI where the upgrade takes place, then when complete they should release it all at once and make it a big announcement deal to their users 'welcome to your new productX'. Its a great enhancement and improvement after all, lets benefit from it and create a buzz.

• During this time, all new UI development should also go directly into the V10 branch. This will near certainly delay the release of some features in the UI while they wait for everything V10 to be ready before they can ship but will mean we aren't building things in 9 as we're moving other things to 10, short term pain of delay, massively long term gain of not building twice and even more buzz on release day of the V10 version if it comes with new features too.

This roadmap for each product could look like:

• Date 1: Design migration begins - I imagine this could be very rapid if its purely a 9->10 reskin.

• Date 2: Migrated designs of all existing flows get delivered.

• Date 3: New UI branch created, V10 upgrades begin in this branch. All new UI is also built directly in this branch.

• Date 4: V10 launch day. Today the user logs in and is greeted with a short message explaining the changes, the new design and any new features that were produced between when v9 dev stopped and today.

• One other, important thought is if we want a 'softer' release of V10 versions of each product where users can for a while, choose to switch back to what it used to be like (without new features). Then we did something similar in Platform when moving from Prometheus -> Kratos where we actually created new pages and URLS for the V10 work, and customers can pick if they wanna work in the V1 or V2 of the application, until we finally moved everything to V2 and resolved all the duplicate URLs.

Topic Three: So what does that mean for our core component library:**

Quite simply, we get the V9 halt day date. We create a V10 branch and upgrade our existing components in there. Anything new needs to be asked 'do you need it before or after V9 day?'. If before, then we do double work (create in 9 and 10 together), if after then we build directly in 10. At V9 day, we freeze the V9 branch and focus all work on the V10 branch, support etc for V9 would be frozen.

[tay1orjones]

@stuckless Very good food for thought!

Why a branch?

I think it would be best to have a branch to contain the v10 updates, mainly because we're going to have to bump our Carbon dev dependencies to v10 versions in order to make the updates. Without the branch, we'd have to do this for each feature branch and would be prone to version mismatches between developer environments. It would also facilitate publishing pre-release versions to npm a bit more cleanly, maybe even automated with semantic-release. Even still, consumers won't be installing a v9 or v10 branch, it would just be an alternate version published to npm.

Why feature flags?

Your question about feature flags rings true - we might be able to ditch feature flags altogether. It might be easier to sum this whole RFC up as:

"We're working on our next major version release (v2.x) in a new branch, which will solely support Carbon v10."

No feature flags means less overhead. I like it, and it doesn't feel like we need them.

What's the upgrade path without feature flags?

Consuming projects can stay on v1.x until they're ready to update to v2.x with the Carbon v10 updates. If there's a severe bugfix or other issue in v1.x, we can cut a manual release to npm. At that point, master would be representative of the latest 2.x release, so we'll need to make a longstanding v1 branch to facilitate making those updates to previous versions.

@lukefirth This line of thinking above matches with your proposal - a hard line between Carbon v9 and Carbon v10 support. We also did hear that the Carbon team has found the feature flags to be difficult to manage in certain situations. Less overhead overall would be ideal. In addition we can follow this procedure again for future Carbon releases, major or minor.

v10 Storybook

Yes, we can configure netlify to publish the pre-release branch to a separate url so we can view without having the development environment running.

Reviews will still happen in PRs - branching off of pre-release branch to make changes, then PR to merge back into pre-release.

Monorepo follow up

@stuckless - Carbon has moved to a monorepo architecture. They're considering recommending the same for Addons and potentially opening up their npm scope (@carbon) in some form to publish Addons packages like ours there. Everything there lives in one repo, individual packages and components are within the packages folder, they're managed through lerna and publish independently of one another. This monorepo architecture facilitates Zenhub integration, etc as everything lives in one place. The RFC and the associated PR discussion, are particularly great thorough reads.

Carbon v10 transition

@lukefirth I very much agree with your comment and would love to publish some migration guidance just like you suggest in addition to the guidance provided by Carbon. They recommend the same approach, doing it all at once. Branching the UI repo, making the updates, pushing all at once including new UI development. Love the idea that v10 support might come with new features as well.

These shared components here are a small part of the migration for consuming projects. The bulk of the work won't be the instances where they consume this repo, it's Carbon proper.

I think we can begin this work without a total v9 halt day, as long as there's an agreed-upon set of components that will have v10 support. Those agreed-upon components will need a v9 halt date.

[tay1orjones]

It looks like this could be how we set up semantic release for publishing the pre-release branch to npm

semantic-release/semantic-release#381 (comment)

[tay1orjones]

Semantic release supports prerelease channels in their beta release, we'll just need to update and then set configs for different branches

[tay1orjones]

Also would like to propose a couple additional items here for the v2 release of this package:

1. Update Storybook theming to be in line with the internal Watson IoT Design System website
2. Wrap all carbon components, even if there are no changes to stock carbon
   1. Why?
      1. Consumers would only need to import components from one package, vs carbon-components-react and this lib. There shouldn't be a need for consumers to use stock Carbon alongside our package, but if they choose to, they can. This would be outside the support realm of this package though.
      2. Small tweaks to Carbon components can be made without requiring arbitrary naming conventions and it eliminates confusing usage requirements ("Do I use this one or that one, what's different?") - ie ButtonEnhanced would be no more, we'd have simply Button that extends Carbon's Button to our liking
      3. Issues would be consolidated here and can be decided if we should contain fixes/features here or push upstream to Carbon itself, rather than consumers having to deal with the confusion of where to look for updates or fixes.
      4. We'll frequently pull in upstream Carbon to maintain feature parity - this is an additional cost, but is outweighed by the benefits we get above
      5. This approach has been explored by some other teams in IBM and have found the pattern to work well with teams in various upgrade cycles

[tay1orjones]

Closed via #304