Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pods stuck in ContainerCreating after cluster autoscaling (AWS CNI race condition) #247

Closed
deliahu opened this issue Jul 22, 2019 · 0 comments · Fixed by #261 or #291
Closed

Pods stuck in ContainerCreating after cluster autoscaling (AWS CNI race condition) #247

deliahu opened this issue Jul 22, 2019 · 0 comments · Fixed by #261 or #291
Assignees
@deliahu
Labels
blocked Blocked on another task or external event bug Something isn't working

Comments

@deliahu
Copy link
Member

deliahu commented Jul 22, 2019 •
edited
Loading

Description

See:

To reproduce

  1. Run cluster with two t3.mediums and cluster autoscaling enabled
  2. Deploy iris and let run to completion
  3. Deploy pipelines/iris

This will trigger cluster autoscaling, and once the new node has joined the cluster, the pending Spark job will get scheduled but will be stuck in ContainerCreating
@deliahu deliahu added bug Something isn't working blocked Blocked on another task or external event labels Jul 22, 2019
@deliahu deliahu self-assigned this Jul 25, 2019
@deliahu deliahu changed the title Pods get stuck in ContainerCreating after cluster autoscaling Pods get stuck in ContainerCreating after cluster autoscaling (AWS CNI race condition) Jul 25, 2019
This was referenced Jul 25, 2019
Closed
Merged
@deliahu deliahu removed the blocked Blocked on another task or external event label Jul 26, 2019
@deliahu deliahu reopened this Jul 27, 2019
@deliahu deliahu added the blocked Blocked on another task or external event label Jul 27, 2019
@deliahu deliahu changed the title Pods get stuck in ContainerCreating after cluster autoscaling (AWS CNI race condition) Pods stuck in ContainerCreating after cluster autoscaling (AWS CNI race condition) Jul 29, 2019
@deliahu deliahu mentioned this issue Aug 7, 2019
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@deliahu deliahu

Labels
blocked Blocked on another task or external event bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Replace AWS CNI with weave-net cortexlabs/cortex
Set AWS CNI version to 1.5.2 cortexlabs/cortex
1 participant
@deliahu