Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split RHCOS into layers #1637

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Split RHCOS into layers #1637

wants to merge 1 commit into from

Conversation

jlebon
Copy link
Member

@jlebon jlebon commented Jun 7, 2024

This enhancement describes improvements to the way RHEL CoreOS (RHCOS) is built so that it will better align with image mode for RHEL, all while also providing benefits on the OpenShift side. Currently, RHCOS is built as a single layer that includes both RHEL and OCP content. This enhancement proposes splitting it into three layers. Going from bottom to top:

  1. the (RHEL-versioned) bootc layer (i.e. the base rhel-bootc image shared with image mode for RHEL)
  2. the (RHEL-versioned) CoreOS layer (i.e. coreos-installer, ignition, afterburn, scripts, etc...)
  3. the (OCP-versioned) node layer (i.e. kubelet, cri-o, etc...)

The terms "bootc layer", "CoreOS layer", and "node layer" will be used throughout this enhancement to refer to these.

The details of this enhancement focus on doing the first split: creating the node layer as distinct from the CoreOS layer (which will not yet be rebased on top of a bootc layer). The two changes involved which most affect OCP are:

  1. bootimages will no longer contain OCP components (e.g. kubelet, cri-o, etc...)
  2. the rhel-coreos payload image will be built in Prow/Konflux (as any other)

Tracked at: https://issues.redhat.com/browse/OCPSTRAT-1190

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Jun 7, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@jlebon jlebon mentioned this pull request Jun 7, 2024
Closed
@jlebon jlebon force-pushed the pr/split-rhcos-into-layers branch from 067ece5 to f79684b Compare June 7, 2024 15:38
cgwalters
cgwalters reviewed Jun 7, 2024
View reviewed changes
Copy link
Member

@cgwalters cgwalters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work on this!

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
@rphillips
Copy link
Contributor

openshift/kubernetes has a specific workflow where jobs will build a new kubelet to use during the job run. This helps with rebase work and validating new kubernetes versions coming into OpenShift. We should preserve this workflow when migrating to RHCOS layering.

/cc @soltysh

@openshift-ci openshift-ci bot requested a review from soltysh June 12, 2024 16:00
@jlebon
Copy link
Member Author

jlebon commented Jun 12, 2024

openshift/kubernetes has a specific workflow where jobs will build a new kubelet to use during the job run. This helps with rebase work and validating new kubernetes versions coming into OpenShift. We should preserve this workflow when migrating to RHCOS layering.

/cc @soltysh

I don't expect any issues there. That workflow should keep working as is.

@jlebon jlebon force-pushed the pr/split-rhcos-into-layers branch from f79684b to a6a7438 Compare June 20, 2024 21:15
zaneb
zaneb reviewed Jun 24, 2024
View reviewed changes
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
@zaneb
Copy link
Member

zaneb commented Jun 24, 2024

/cc @cybertron @andfasano

@soltysh
Copy link

soltysh commented Jun 26, 2024

I don't expect any issues there. That workflow should keep working as is.

I believe this was the pre-req work done in openshift/kubernetes#1805, which ensured we won't have problems in o/k.

jlebon added a commit to jlebon/installer that referenced this pull request Jul 16, 2024
@jlebon
WIP: bootstrap: pivot into node image before bootstrapping
42890ca 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be `oc`, `kubelet`, or `crio`
binaries for example, which bootstrapping obviously relies on.

Instead, now we change things up so that early on when booting the
bootstrap node, we pull down the node image, unencapsulate it (this just
means convert it back to an OSTree commit), then mount over its `/usr`,
and import new `/etc` content.

This is done by isolating to a different systemd target to only bring
up the minimum number of services to do the pivot and then carry on
with bootstrapping.

This does not incur additional reboots and should be compatible
with AI/ABI/SNO. But it is of course, a huge conceptual shift in how
bootstrapping works. With this, we would now always be sure that we're
using the same binaries as the target version as part of bootstrapping,
which should alleviate some issues such as AI late-binding (see e.g.
https://issues.redhat.com/browse/MGMT-16705).

The big exception of course being the kernel. Relatedly, currently
`/usr/lib/modules` is also shadowed by the mount, but we could re-mount
it if needed.

To be conservative, the new logic only kicks in when using bootimages
which do not have `oc`. This will allow us to ratchet this in more
easily.

Down the line, we should be able to replace some of this with
`bootc apply-live` once that's available (and also works in a live
environment). (See containers/bootc#76.)

For full context, see the linked enhancement and discussions there.
@jlebon jlebon mentioned this pull request Jul 16, 2024
Draft
@jlebon
Copy link
Member Author

jlebon commented Jul 16, 2024

OK, so let's resume the bootstrapping issue. Restating some of the things from above and from researching further:

  • We can't run the kubelet in a container because it's no longer supported.
  • The delta between kubelet and podman play is too large to make the latter a feasible replacement.
  • systemctl soft-reboot is not in RHEL9.
  • In the AI/ABI/SNO cases, bootstrapping happens in the live environment where e.g. rebooting is not possible.
  • I considered cobbling something around kexec, but in the limit, there are potential issues with kexec and hardware reliability, as well as how it meshes with Secure Boot.

What I'm playing with now is basically to have a special node-image-pivot.target that the node isolates to first. There, we pull the node image, unencapsulate it, check out its contents, and then mount over /usr and do a rough 3-way /etc merge. We then isolate back to multi-user.target to continue with the bootstrapping process.

This is in effect like a more aggressive bootc/rpm-ostree apply-live, though that doesn't currently work in live environments. (Though even in the non-live case, there are some issues there that would need to be resolved.) It's close to what OKD currently does when using FCOS live media today, though using the ostree stack and isolating targets should make this more robust.

WIP for this in openshift/installer#8742.

@rphillips
Copy link
Contributor

@jlebon That sounds like it might work. Where will the Kubelet be coming from? An OpenShift built image?

@zaneb
Copy link
Member

zaneb commented Jul 17, 2024

Won't doing systemctl isolate node-image-pivot.targethave the effect of stopping the assisted/agent services that we need to avoid stopping?

@jlebon
Copy link
Member Author

jlebon commented Jul 17, 2024

@jlebon That sounds like it might work. Where will the Kubelet be coming from? An OpenShift built image?

From the node image (i.e. for OCP, the rhel-coreos image in the release payload).

Won't doing systemctl isolate node-image-pivot.targethave the effect of stopping the assisted/agent services that we need to avoid stopping?

No. The system boots into node-image-pivot.target first. Any other service hooked into multi-user.target aren't started until after we've finished the live pivot.

@cgwalters
Copy link
Member

The system boots into node-image-pivot.target first.

Via a generator overriding default.target?

@jlebon jlebon mentioned this pull request Aug 22, 2024
Open
jlebon added a commit to jlebon/installer that referenced this pull request Sep 6, 2024
@jlebon
WIP: bootstrap: pivot into node image before bootstrapping
fd7af8f 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be `oc`, `kubelet`, or `crio`
binaries for example, which bootstrapping obviously relies on.

Instead, now we change things up so that early on when booting the
bootstrap node, we pull down the node image, unencapsulate it (this just
means convert it back to an OSTree commit), then mount over its `/usr`,
and import new `/etc` content.

This is done by isolating to a different systemd target to only bring
up the minimum number of services to do the pivot and then carry on
with bootstrapping.

This does not incur additional reboots and should be compatible
with AI/ABI/SNO. But it is of course, a huge conceptual shift in how
bootstrapping works. With this, we would now always be sure that we're
using the same binaries as the target version as part of bootstrapping,
which should alleviate some issues such as AI late-binding (see e.g.
https://issues.redhat.com/browse/MGMT-16705).

The big exception of course being the kernel. Relatedly, note we do
persist `/usr/lib/modules` from the booted system so that loading kernel
modules still works.

To be conservative, the new logic only kicks in when using bootimages
which do not have `oc`. This will allow us to ratchet this in more
easily.

Down the line, we should be able to replace some of this with
`bootc apply-live` once that's available (and also works in a live
environment). (See containers/bootc#76.)

For full context, see the linked enhancement and discussions there.
jlebon added a commit to jlebon/installer that referenced this pull request Sep 6, 2024
@jlebon
WIP: bootstrap: pivot into node image before bootstrapping
f181329 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be `oc`, `kubelet`, or `crio`
binaries for example, which bootstrapping obviously relies on.

Instead, now we change things up so that early on when booting the
bootstrap node, we pull down the node image, unencapsulate it (this just
means convert it back to an OSTree commit), then mount over its `/usr`,
and import new `/etc` content.

This is done by isolating to a different systemd target to only bring
up the minimum number of services to do the pivot and then carry on
with bootstrapping.

This does not incur additional reboots and should be compatible
with AI/ABI/SNO. But it is of course, a huge conceptual shift in how
bootstrapping works. With this, we would now always be sure that we're
using the same binaries as the target version as part of bootstrapping,
which should alleviate some issues such as AI late-binding (see e.g.
https://issues.redhat.com/browse/MGMT-16705).

The big exception of course being the kernel. Relatedly, note we do
persist `/usr/lib/modules` from the booted system so that loading kernel
modules still works.

To be conservative, the new logic only kicks in when using bootimages
which do not have `oc`. This will allow us to ratchet this in more
easily.

Down the line, we should be able to replace some of this with
`bootc apply-live` once that's available (and also works in a live
environment). (See containers/bootc#76.)

For full context, see the linked enhancement and discussions there.
jlebon added a commit to jlebon/assisted-installer that referenced this pull request Sep 6, 2024
@jlebon
Overlay node image before bootstrapping if necessary
ebd45cd 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be oc, kubelet, or crio
binaries for example, which bootstrapping obviously relies on.

To adapt to this, the OpenShift installer now ships a new
`node-image-overlay.service` in its bootstrap Ignition config. This
service takes care of pulling down the node image and overlaying it,
effectively updating the system to the node image version.

Here, we accordingly also adapt assisted-installer so that we run
`node-image-overlay.service` before starting e.g. `kubelet.service` and
`bootkube.service`.

See also: openshift/installer#8742
jlebon added a commit to jlebon/assisted-installer that referenced this pull request Sep 6, 2024
@jlebon
WIP: overlay node image before bootstrapping if necessary
23f9c3f 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be oc, kubelet, or crio
binaries for example, which bootstrapping obviously relies on.

To adapt to this, the OpenShift installer now ships a new
`node-image-overlay.service` in its bootstrap Ignition config. This
service takes care of pulling down the node image and overlaying it,
effectively updating the system to the node image version.

Here, we accordingly also adapt assisted-installer so that we run
`node-image-overlay.service` before starting e.g. `kubelet.service` and
`bootkube.service`.

See also: openshift/installer#8742
@jlebon jlebon mentioned this pull request Sep 6, 2024
Draft
jlebon added a commit to jlebon/installer that referenced this pull request Sep 6, 2024
@jlebon
WIP: overlay node image before bootstrapping if necessary
d920a70 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be `oc`, `kubelet`, or `crio`
binaries for example, which bootstrapping obviously relies on.

Instead, now we change things up so that early on when booting the
bootstrap node, we pull down the node image, unencapsulate it (this just
means convert it back to an OSTree commit), then mount over its `/usr`,
and import new `/etc` content.

This is done by isolating to a different systemd target to only bring
up the minimum number of services to do the pivot and then carry on
with bootstrapping.

This does not incur additional reboots and should be compatible
with AI/ABI/SNO. But it is of course, a huge conceptual shift in how
bootstrapping works. With this, we would now always be sure that we're
using the same binaries as the target version as part of bootstrapping,
which should alleviate some issues such as AI late-binding (see e.g.
https://issues.redhat.com/browse/MGMT-16705).

The big exception of course being the kernel. Relatedly, note we do
persist `/usr/lib/modules` from the booted system so that loading kernel
modules still works.

To be conservative, the new logic only kicks in when using bootimages
which do not have `oc`. This will allow us to ratchet this in more
easily.

Down the line, we should be able to replace some of this with
`bootc apply-live` once that's available (and also works in a live
environment). (See containers/bootc#76.)

For full context, see the linked enhancement and discussions there.
jlebon added a commit to jlebon/installer that referenced this pull request Sep 7, 2024
@jlebon
WIP: overlay node image before bootstrapping if necessary
21b03ea 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be `oc`, `kubelet`, or `crio`
binaries for example, which bootstrapping obviously relies on.

Instead, now we change things up so that early on when booting the
bootstrap node, we pull down the node image, unencapsulate it (this just
means convert it back to an OSTree commit), then mount over its `/usr`,
and import new `/etc` content.

This is done by isolating to a different systemd target to only bring
up the minimum number of services to do the pivot and then carry on
with bootstrapping.

This does not incur additional reboots and should be compatible
with AI/ABI/SNO. But it is of course, a huge conceptual shift in how
bootstrapping works. With this, we would now always be sure that we're
using the same binaries as the target version as part of bootstrapping,
which should alleviate some issues such as AI late-binding (see e.g.
https://issues.redhat.com/browse/MGMT-16705).

The big exception of course being the kernel. Relatedly, note we do
persist `/usr/lib/modules` from the booted system so that loading kernel
modules still works.

To be conservative, the new logic only kicks in when using bootimages
which do not have `oc`. This will allow us to ratchet this in more
easily.

Down the line, we should be able to replace some of this with
`bootc apply-live` once that's available (and also works in a live
environment). (See containers/bootc#76.)

For full context, see the linked enhancement and discussions there.
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 9, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign miciah for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jlebon jlebon marked this pull request as ready for review September 9, 2024 20:16
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2024
@jlebon
Copy link
Member Author

jlebon commented Sep 9, 2024

Updated and officially ready for review!

  • Filled in the remaining sections
  • Added details about versioning scheme
  • Added details about solving the bootstrapping issue
  • Added @yuqi-zhang and @jupierce as reviewers

@cgwalters
Copy link
Member

@omertuc did you see this one?

jlebon added a commit to jlebon/assisted-installer that referenced this pull request Sep 10, 2024
@jlebon
WIP: overlay node image before bootstrapping if necessary
48c6cc1 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be oc, kubelet, or crio
binaries for example, which bootstrapping obviously relies on.

To adapt to this, the OpenShift installer now ships a new
`node-image-overlay.service` in its bootstrap Ignition config. This
service takes care of pulling down the node image and overlaying it,
effectively updating the system to the node image version.

Here, we accordingly also adapt assisted-installer so that we run
`node-image-overlay.service` before starting e.g. `kubelet.service` and
`bootkube.service`.

See also: openshift/installer#8742
jlebon added a commit to jlebon/installer that referenced this pull request Sep 10, 2024
@jlebon
WIP: bootstrap: pivot into node image before bootstrapping
4a5738b 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be `oc`, `kubelet`, or `crio`
binaries for example, which bootstrapping obviously relies on.

Instead, now we change things up so that early on when booting the
bootstrap node, we pull down the node image, unencapsulate it (this just
means convert it back to an OSTree commit), then mount over its `/usr`,
and import new `/etc` content.

This is done by isolating to a different systemd target to only bring
up the minimum number of services to do the pivot and then carry on
with bootstrapping.

This does not incur additional reboots and should be compatible
with AI/ABI/SNO. But it is of course, a huge conceptual shift in how
bootstrapping works. With this, we would now always be sure that we're
using the same binaries as the target version as part of bootstrapping,
which should alleviate some issues such as AI late-binding (see e.g.
https://issues.redhat.com/browse/MGMT-16705).

The big exception of course being the kernel. Relatedly, note we do
persist `/usr/lib/modules` from the booted system so that loading kernel
modules still works.

To be conservative, the new logic only kicks in when using bootimages
which do not have `oc`. This will allow us to ratchet this in more
easily.

Down the line, we should be able to replace some of this with
`bootc apply-live` once that's available (and also works in a live
environment). (See containers/bootc#76.)

For full context, see the linked enhancement and discussions there.
jlebon added a commit to jlebon/installer that referenced this pull request Sep 10, 2024
@jlebon
WIP: bootstrap: pivot into node image before bootstrapping
fa023a9 
As per openshift/enhancements#1637, we're trying
to get rid of all OpenShift-versioned components from the bootimages.

This means that there will no longer be `oc`, `kubelet`, or `crio`
binaries for example, which bootstrapping obviously relies on.

Instead, now we change things up so that early on when booting the
bootstrap node, we pull down the node image, unencapsulate it (this just
means convert it back to an OSTree commit), then mount over its `/usr`,
and import new `/etc` content.

This is done by isolating to a different systemd target to only bring
up the minimum number of services to do the pivot and then carry on
with bootstrapping.

This does not incur additional reboots and should be compatible
with AI/ABI/SNO. But it is of course, a huge conceptual shift in how
bootstrapping works. With this, we would now always be sure that we're
using the same binaries as the target version as part of bootstrapping,
which should alleviate some issues such as AI late-binding (see e.g.
https://issues.redhat.com/browse/MGMT-16705).

The big exception of course being the kernel. Relatedly, note we do
persist `/usr/lib/modules` from the booted system so that loading kernel
modules still works.

To be conservative, the new logic only kicks in when using bootimages
which do not have `oc`. This will allow us to ratchet this in more
easily.

Down the line, we should be able to replace some of this with
`bootc apply-live` once that's available (and also works in a live
environment). (See containers/bootc#76.)

For full context, see the linked enhancement and discussions there.
jupierce
jupierce reviewed Sep 12, 2024
View reviewed changes
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved
enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved
@jlebon
Copy link
Member Author

jlebon commented Oct 25, 2024

Updated this for feedback!

  • Clarified node image CI build triggers
  • Clarified ART tooling changes needed for proper change detection
  • Elaborated on release controller changes needed
  • Clarified ART plashet changes needed for weekly RHEL kernel
  • Mentioned bootimage bump potential follow-up
  • Mentioned the hotfix case

@jlebon
Split RHCOS into layers
0f934fb 
This enhancement describes improvements to the way RHEL CoreOS (RHCOS)
is built so that it will better align with image mode for RHEL, all
while also providing benefits on the OpenShift side. Currently, RHCOS
is built as a single layer that includes both RHEL and OCP content. This
enhancement proposes splitting it into three layers. Going from bottom
to top:
1. the (RHEL-versioned) bootc layer (i.e. the base rhel-bootc image
   shared with image mode for RHEL)
2. the (RHEL-versioned) CoreOS layer (i.e. coreos-installer, ignition,
   afterburn, scripts, etc...)
3. the (OCP-versioned) node layer (i.e. kubelet, cri-o, etc...)

The terms "bootc layer", "CoreOS layer", and "node layer" will be used
throughout this enhancement to refer to these.

The details of this enhancement focus on doing the first split: creating
the node layer as distinct from the CoreOS layer (which will not yet be
rebased on top of a bootc layer). The two changes involved which most
affect OCP are:
1. bootimages will no longer contain OCP components (e.g. kubelet,
   cri-o, etc...)
2. the `rhel-coreos` payload image will be built in Prow/Konflux (as
   any other)

Tracked at: https://issues.redhat.com/browse/OCPSTRAT-1190
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Oct 25, 2024

@jlebon: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LorbusChris LorbusChris LorbusChris left review comments

@cverna cverna cverna left review comments

@eranco74 eranco74 eranco74 left review comments

@rphillips rphillips rphillips left review comments

@mrunalp mrunalp mrunalp left review comments

@jupierce jupierce jupierce left review comments

@travier travier travier left review comments

@wking wking wking left review comments

@bthurber bthurber bthurber left review comments

@mhrivnak mhrivnak mhrivnak left review comments

@sdodson sdodson sdodson left review comments

@yuqi-zhang yuqi-zhang yuqi-zhang left review comments

@andfasano andfasano andfasano left review comments

@zaneb zaneb zaneb left review comments

@cgwalters cgwalters cgwalters left review comments

@romfreiman romfreiman romfreiman left review comments

@soltysh soltysh Awaiting requested review from soltysh

@cybertron cybertron Awaiting requested review from cybertron

@jaypoulz jaypoulz Awaiting requested review from jaypoulz

@shawn-hurley shawn-hurley Awaiting requested review from shawn-hurley

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.