Enable leader election in ws-manager-mk2 (v3) #18539

aledbf · 2023-08-17T12:26:41Z

Description

Change:

The previous revert was related to the disabled maintenance mode not detected by the stand-by replica.
Now, the issue is related to grpc request/s being processed by the stand-by replica and storing the state in memory.
The fix is to not bind the Subscribers helper to the grpc server instance connected to; instead, an informer is the source of truth.

Here we remove the activity and switch the state to the workspace status.

Summary generated by Copilot

`🤖 Generated by Copilot at 27cc1a0`

Refactor and improve the ws-manager-mk2 component, update its dependencies and tests, and add a new field LastActivity to the WorkspaceStatus struct.

Related Issue(s)

Fixes #

How to test

Documentation

Preview status

Gitpod was successfully deployed to your preview environment.

🏷️ Name - aledbf-leader-v3
🔗 URL - aledbf-leader-v3.preview.gitpod-dev.com/workspaces.
📚 Documentation - See our internal documentation for information on how to interact with your preview environment.
📦 Version - aledbf-leader-v3-gha.15566

Build Options

Build

/werft with-werft
Run the build with werft instead of GHA
leeway-no-cache
/werft no-test
Run Leeway with --dont-test

Publish

/werft publish-to-npm
/werft publish-to-jb-marketplace

Installer

analytics=segment
with-dedicated-emulation
workspace-feature-flags
Add desired feature flags to the end of the line above, space separated

Preview Environment / Integration Tests

/werft with-local-preview
If enabled this will build install/preview
/werft with-preview
/werft with-large-vm
/werft with-gce-vm
If enabled this will create the environment on GCE infra
with-integration-tests=workspace
Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
with-monitoring

/hold

aledbf · 2023-08-17T14:10:05Z

Integration tests https://github.com/gitpod-io/gitpod/actions/runs/5891737721/job/15979818090

aledbf · 2023-08-17T14:15:29Z

/gh run recreate-vm=true

Comment triggered a workflow run

Started workflow run: 5892091350

recreate_vm: true

Furisto · 2023-08-17T14:25:24Z

The manifest for the CRD has not been updated. Run make manifests.

WVerlaek · 2023-08-17T14:41:38Z

Integration tests https://github.com/gitpod-io/gitpod/actions/runs/5891737721/job/15979818090

This actually used a build from main (main-gha.15563) 😅, you can either specify the installer version from this PR or tick the checkbox in the PR description to run the tests from there instead

aledbf · 2023-08-17T20:39:29Z

/gh run recreate-vm=true

Comment triggered a workflow run

Started workflow run: 5895723228

recreate_vm: true

aledbf · 2023-08-18T09:33:01Z

The manifest for the CRD has not been updated. Run make manifests.

done

aledbf · 2023-08-18T11:57:42Z

/gh run recreate-vm=true

Comment triggered a workflow run

Started workflow run: 5902316472

recreate_vm: true

aledbf · 2023-08-19T15:22:45Z

/gh run recreate-vm=true

Comment triggered a workflow run

Started workflow run: 5912178632

recreate_vm: true

dev/preview/workflow/preview/deploy-gitpod.sh

components/ws-manager-mk2/controllers/subscriber_controller.go

csweichel · 2023-08-22T08:05:03Z

Change lgtm with some minor comments.
Where can I find the benchmarks/measurements for the additional load imposed by moving the activity to the CR?

WVerlaek · 2023-08-23T08:51:46Z

components/ws-manager-mk2/controllers/suite_test.go

-	wsMetrics  *controllerMetrics
+	ctx       context.Context
+	cancel    context.CancelFunc
+	wsMetrics *controllerMetrics


how do the ws-manager metrics behave when two replicas are enabled? Do both report metrics, or only one?

Metrics will be scrapped from all available replicas

this will break some of the dashboards then (and maybe alerts too). For instance we sum all workspaces by cluster, this will then double the amount of reported workspaces

Please check this gist (integration tests are running there)

Only the leader will have metrics related to the controller. If the current leader is not elected anymore, the pod is restarted.

thanks for the gists. Both replicas report workspace metrics, e.g.:

gitpod_ws_manager_mk2_workspace_phase_total{class="g1-standard",phase="Running",type="Regular"} 1

is in both. Summing them as done in dashboards/alerts would give 2 replicas, while there's only 1

gist updated after running the integration test

components/ws-manager-mk2/controllers/subscriber_controller.go

components/ws-manager-mk2/controllers/timeout_controller.go

components/ws-manager-mk2/service/manager.go

components/ws-manager-mk2/main.go

components/ws-manager-mk2/controllers/subscriber_controller.go

WVerlaek · 2023-08-23T09:42:43Z

Where can I find the benchmarks/measurements for the additional load imposed by moving the activity to the CR?

I'd also like to see some benchmarks, as both a new replica is added watching for events, and the large increase in CR activity.

I don't think though that loadgen will be able to simulate the additional activity as the workspaces aren't marked as active. How would we test that?

roboquat added do-not-merge/hold do-not-merge/work-in-progress size/XL size/XXL and removed size/XL labels Aug 17, 2023

aledbf marked this pull request as ready for review August 17, 2023 14:11

aledbf requested review from a team as code owners August 17, 2023 14:11

roboquat removed the do-not-merge/work-in-progress label Aug 17, 2023

github-actions bot added team: team-experience team: team-engine labels Aug 17, 2023

aledbf force-pushed the aledbf/leader-v3 branch from 2575ccb to b44fb71 Compare August 17, 2023 20:12

aledbf marked this pull request as draft August 17, 2023 20:21

roboquat added the do-not-merge/work-in-progress label Aug 17, 2023

aledbf force-pushed the aledbf/leader-v3 branch from bd7f036 to 7f968fa Compare August 18, 2023 08:07

aledbf force-pushed the aledbf/leader-v3 branch 2 times, most recently from 36c751a to c73dc1d Compare August 18, 2023 20:14

aledbf force-pushed the aledbf/leader-v3 branch 4 times, most recently from db24b0f to cd14a5b Compare August 21, 2023 12:55

aledbf marked this pull request as ready for review August 21, 2023 12:55

Furisto reviewed Aug 21, 2023

View reviewed changes

dev/preview/workflow/preview/deploy-gitpod.sh Show resolved Hide resolved

Furisto approved these changes Aug 21, 2023

View reviewed changes

csweichel reviewed Aug 22, 2023

View reviewed changes

components/ws-manager-mk2/controllers/subscriber_controller.go Outdated Show resolved Hide resolved

csweichel reviewed Aug 22, 2023

View reviewed changes

components/ws-manager-mk2/controllers/subscriber_controller.go Show resolved Hide resolved

WVerlaek reviewed Aug 23, 2023

View reviewed changes

components/ws-manager-mk2/main.go Outdated Show resolved Hide resolved

components/ws-manager-mk2/controllers/subscriber_controller.go Show resolved Hide resolved

aledbf added 17 commits August 23, 2023 09:57

Enable leader election in ws-manager-mk2

766766b

Update go modules

16b1aad

Move workspace activity to CRD

201990a

Remove workspace activity

0cd9523

Cleanup

1594aac

Update ws-manager-mk2 CRD

615faa8

Cleanup

1067f75

Restore lastActivity logic

ec385e4

TEST

c152222

Disable observability

51aa070

Start the grpc server after leader election

07c995f

Bount the source of subscribers to an informer

477b688

Cleanup

ecf1b95

Avoid deepCopy

08ae889

Remove goroutine to execute OnReconcile

1ef3f80

Refactor last activity to be consistent acrtoss the controllers

08537ae

Address feedback

d390827

aledbf force-pushed the aledbf/leader-v3 branch from bba6c41 to d390827 Compare August 23, 2023 09:57

akosyakov approved these changes Aug 24, 2023

View reviewed changes

aledbf removed the do-not-merge/hold label Aug 26, 2023

roboquat merged commit 687f337 into main Aug 26, 2023
16 checks passed

roboquat deleted the aledbf/leader-v3 branch August 26, 2023 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable leader election in ws-manager-mk2 (v3) #18539

Enable leader election in ws-manager-mk2 (v3) #18539

aledbf commented Aug 17, 2023 •

edited

Loading

aledbf commented Aug 17, 2023

aledbf commented Aug 17, 2023 •

edited by github-actions bot

Loading

Furisto commented Aug 17, 2023

WVerlaek commented Aug 17, 2023

aledbf commented Aug 17, 2023 •

edited by github-actions bot

Loading

aledbf commented Aug 18, 2023

aledbf commented Aug 18, 2023 •

edited by github-actions bot

Loading

aledbf commented Aug 19, 2023 •

edited by github-actions bot

Loading

csweichel commented Aug 22, 2023

WVerlaek Aug 23, 2023

aledbf Aug 23, 2023

WVerlaek Aug 23, 2023

aledbf Aug 23, 2023 •

edited

Loading

WVerlaek Aug 23, 2023

aledbf Aug 23, 2023

WVerlaek commented Aug 23, 2023

Enable leader election in ws-manager-mk2 (v3) #18539

Enable leader election in ws-manager-mk2 (v3) #18539

Conversation

aledbf commented Aug 17, 2023 • edited Loading

Description

🤖 Generated by Copilot at 27cc1a0

Related Issue(s)

How to test

Documentation

Preview status

Build Options

aledbf commented Aug 17, 2023

aledbf commented Aug 17, 2023 • edited by github-actions bot Loading

Comment triggered a workflow run

Furisto commented Aug 17, 2023

WVerlaek commented Aug 17, 2023

aledbf commented Aug 17, 2023 • edited by github-actions bot Loading

Comment triggered a workflow run

aledbf commented Aug 18, 2023

aledbf commented Aug 18, 2023 • edited by github-actions bot Loading

Comment triggered a workflow run

aledbf commented Aug 19, 2023 • edited by github-actions bot Loading

Comment triggered a workflow run

csweichel commented Aug 22, 2023

WVerlaek Aug 23, 2023

Choose a reason for hiding this comment

aledbf Aug 23, 2023

Choose a reason for hiding this comment

WVerlaek Aug 23, 2023

Choose a reason for hiding this comment

aledbf Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

WVerlaek Aug 23, 2023

Choose a reason for hiding this comment

aledbf Aug 23, 2023

Choose a reason for hiding this comment

WVerlaek commented Aug 23, 2023

aledbf commented Aug 17, 2023 •

edited

Loading

`🤖 Generated by Copilot at 27cc1a0`

aledbf commented Aug 17, 2023 •

edited by github-actions bot

Loading

aledbf commented Aug 17, 2023 •

edited by github-actions bot

Loading

aledbf commented Aug 18, 2023 •

edited by github-actions bot

Loading

aledbf commented Aug 19, 2023 •

edited by github-actions bot

Loading

aledbf Aug 23, 2023 •

edited

Loading