Add option to ignore provider errors when removing a runner in error state #160
Comments
gabriel-samfira
added a commit
to gabriel-samfira/garm
that referenced
this issue
Oct 12, 2023
This branch adds the ability to forcefully remove a runner from GARM. When the operator wishes to manually remove a runner, the workflow is as follows: * Check that the runner exists in GitHub. If it does, attempt to remove it. An error here indicates that the runner may be processing a job. In this case, we don't continue and the operator gets immediate feedback from the API. * Mark the runner in the database as pending_delete * Allow the consolidate loop to reap it from the provider and remove it from the database. Removing the instance from the provider is async. If the provider errs out, GARM will keep trying to remove it in perpetuity until the provider succedes. In situations where the provider is misconfigured, this will never happen, leaving the instance in a permanent state of pending_delete. A provider may fail for various reasons. Either credentials have expired, the API endpoint has changed, the provider is misconfigured or the operator may just have removed it from the config before cleaning up the runners. While some cases are recoverable, some are not. We cannot have a situation in which we cannot clean resources in garm because of a misconfiguration. This change adds the pending_force_delete instance status. Instances marked with this status, will be removed from GARM even if the provider reports an error. The GARM cli has been modified to give new meaning to the --force-remove-runner option. This option in the CLI is no longer mandatory. Instead, setting it will mark the runner with the new pending_force_delete status. Omitting it will mark the runner with the old status of pending_delete. Fixes: cloudbase#160 Signed-off-by: Gabriel Adrian Samfira <[email protected]>
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In case we misconfigure a provider, or if we simply remove it from the garm config while we still have runners defined, we may get into a situation where there is no way to remove it because we can't contact the provider anymore.
Another situation is if the provider is misconfigured and will always err out. Garm will fail to remove runners that are in error state.
We need a way to force remove runners from the DB and github, even if the provider is in error, or if it was simply removed.
The text was updated successfully, but these errors were encountered: