Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ECS: Fargate service replacement doesn't preserve desired task count #26189

Closed
zkello opened this issue Jun 30, 2023 · 2 comments
Closed

ECS: Fargate service replacement doesn't preserve desired task count #26189

zkello opened this issue Jun 30, 2023 · 2 comments
Labels
@aws-cdk/aws-ecs Related to Amazon Elastic Container bug This issue is a bug. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. p1 response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.

Comments

@zkello
Copy link

zkello commented Jun 30, 2023

Describe the bug

My team recently ran into this bug which caused our Fargate service to be replaced instead of updated during a deployment. Not ideal, but during the replacement we observed some behaviors which surprised us. Basically the service was replaced as soon as a single task in the new service was up and running. This caused the service to suddenly be under-scaled for the traffic it was receiving. Replacements of Fargate services are rare, but it seems like CDK/ECS could be smarter about how it's done.

Expected Behavior

  1. The old Fargate service shouldn't be replaced until the new service reaches its desired task count or the very least its minimum task count.
  2. The new service should have the desired task count set to that of the old service at time of deployment. This will prevent the service from being brought online under-scaled.

Current Behavior

  1. Traffic from our NLB to the old Fargate service was cut off as soon as a single task was up in running in the new Fargate service, despite the minimum task count in the service being set to 3
  2. The desired task count was set the minimum task count despite the desired task count of the old service being set much higher.

Reproduction Steps

  • Have a Fargate service deployed with the following:
    • Minimum task count set above 1
    • Desired task count set above minium task count based on autoscaling policy (cpu in our case)
  • Make a CDK change that requires a replacement of the service.
  • Observe service is replaced as soon as new service has one task healthy

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.59.0

Framework Version

No response

Node.js Version

18

OS

Linux

Language

Typescript

Language Version

No response

Other information

No response

@zkello zkello added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 30, 2023
@github-actions github-actions bot added the @aws-cdk/aws-ecs Related to Amazon Elastic Container label Jun 30, 2023
@peterwoodworth
Copy link
Contributor

Could you provide reproduction code please?

@peterwoodworth peterwoodworth added p1 response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Jun 30, 2023
@github-actions
Copy link

github-actions bot commented Jul 3, 2023

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Jul 3, 2023
@github-actions github-actions bot closed this as completed Jul 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-ecs Related to Amazon Elastic Container bug This issue is a bug. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. p1 response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Projects
None yet
Development

No branches or pull requests

2 participants