Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VCluster API refresh: allow create db to proceed if db is already running #652

Merged
merged 4 commits into from
Jan 4, 2024

Conversation

spilchen
Copy link
Collaborator

@spilchen spilchen commented Jan 3, 2024

This builds on a change made in vclusterops. It can now emit a specific error if it discovers that the database is already running. We add a feature flag to tolerate this (the default is true). This can allow us to proceed if we are in an environment with few resources available, which can cause instability during the creation process. If we have previously progressed far enough that Vertica is running, we will skip the create db step and proceed to the next reconciler. This will update the status condition to indicate that the database has been initialized.

@spilchen spilchen self-assigned this Jan 3, 2024
@spilchen spilchen changed the title Allow create db to proceed if db is already running VCluster API refresh: allow create db to proceed if db is already running Jan 4, 2024
@spilchen spilchen marked this pull request as ready for review January 4, 2024 15:13
Nodes: s.RestartHosts,
}
// timeout option
opts := vops.VStartNodesOptionsFactory()
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restructured code here so that we call the factor function for opts. This was needed to properly initialize a new field added to the opts.

Copy link
Collaborator

@roypaulin roypaulin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

// was finished. By ignoring the error this time, we will be able to
// update the status condition and start any nodes that may be down in
// order to bring the cluster online.
dbIsRunningError := &vops.DBIsRunningError{}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by partially complete? Do you mean on the operator side?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is the server side. The cases we saw is that create db fails during the package installation. If we got far enough to bring up a vertica process, chances are the database is fine to use.

@@ -238,6 +248,12 @@ func GetTerminationGracePeriodSeconds(annotations map[string]string) int {
return lookupIntAnnotation(annotations, TerminationGracePeriodSecondsAnnotaton)
}

// FailCreateDBIfVerticaIsRunning returns how to handle failures during create
// db if vertica is found to be running.
func FailCreateDBIfVerticaIsRunning(annotations map[string]string) bool {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you tell what returning true means?

@spilchen spilchen merged commit c33e209 into main Jan 4, 2024
27 checks passed
@spilchen spilchen deleted the spilchen/improve-create-db-failure branch January 4, 2024 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants