Overrides all of task definition parameters

[hasimo]

Start-Task and Run-Task API can overrides the task definition.
But it supports a few parameters only. (command, environment and taskRoleArn).
I would like to overrides other parameters. (ex: hostname,logConfiguration ..)
It would be good to be able to overrides all of task definition parameters.

[egafni]

+1, especially resource requirements

[dinvlad]

Same here, in our application it's the volumes:host:sourcePath that needs to be set for each individual task, and currently we have to register a new task definition just for that (all other parameters are kept the same). But there's a hard limit on task definition registration rate of 1/sec, which could be very limiting if we need to do that for thousands of tasks at a time. I imagine other applications have similar problems.

[billyshambrook]

We would greatly benefit if you could override CPU and memory limits. For us, these parameters change depending on the input message. Our current workaround is to create multiple task definitions.

[samuelkarp]

@egafni @billyshambrook CPU, memory, and memory reservation can now be overridden through the RunTask and StartTask API.

I'll leave this issue open as the other task definition fields are not yet supported for overrides.

[jellevandenhooff]

Hi @samuelkarp, it would be great if we could the new Fargate task CPU and memory limits as well!

[dene14]

2 years have passed. only 5 parameters it's possible to override... Great progress!

[vincentkwok]

stuck in aws-cli doc https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html and finally find this thread...

[ppdeassis]

Overriding logConfiguration would be really helpful.

[toredash]

Support @ppdeassis

We have one common container where the application is built, but also all scheduled tasks files are placed. The application and scheduled tasks are related, so building them and having them together in the same Docker image make sense for us.

But, we would like to override the logConfiguration option. That way we have one common task definition which is the master, but depending on which command is run inside the container, the log group name can vary. E.g. "applicationsLogs" vs. "db-table-optimize",

[mrwillis]

Overriding logs would be a great benefit.

[fcoelho]

I'm also looking to override logConfiguration. My use case is to use the same base task definition in a multi-tenant application and replace the stream prefix for each tenant to have log streams of the form <tenant id>/<container name>/<task id>. I can generate one task definition per tenant, but it would be much easier to manage this with a handful of task definitions instead of thousands

[wjlow]

Would love to override logConfiguration on my end.

[hleb-rubanau]

Another use case for logConfiguration override:

We have an administrative task definition which can be run with different parameters injected into container environment (such as project_id). It would be great to be able to reflect the parameter(s) value in the log stream prefix as well, during task run. Otherwise one cannot easily identify relevant log stream for particular task run.

[joanayma]

Overriding entryPoint will be also a good thing.

[DarwinJS]

Overriding the container (full registry path) is critical for implementing more dynamic container references. In the case of leveraging a Fargate configuration for processing CI pipelines, the pipeline definitions allow the CI developer to specify containers from any public container registry or private elastic container registry they have access to.

[jimbob687]

Would like to be able to override image or image tag to pull from ECR. If we tag an image we need to create an new task definition afaik.

[himesh-gosvami]

We would love to hear for logConfiguration update from you guys. eagerly waiting for it. it was raised since 2016 hope we will have this update soon.

[tkram01]

+1 especially image

[remidebette]

Indeed especially for overriding the image (or at least the tag)
In the context of a CI that generates as many tags as github commits sha, if we want to test a task run for those we would then need to create as many task-definition versions...
Anyone knows a workaround applicable currently?

[deuscapturus]

I just want the ability to override the entryPoint

[sgrilux]

+1 on image override.

Is there any ETA for this? I am setting up a serverless Gitlab runner on Fargate/ECS and would be nice to have this implemented. At the moment we have to use generic images, which is fine for now as we don't have complex pipelines but we may need in future.

[dsinghvi]

+1 on image override as well

[DougTrajano]

+1 on image override as well

[nicholasserra]

+1 on image override as well

[ozeebee]

+1 on image override

[konstantinj]

+1 on image override

[pete-leese]

hard to believe this has been open since 2016 and still not implemented.

It's really important to be able to override image and other parameters.

[gyanglz]

yeah, I think AWS is probably giving up or has already given up ECS, just dont want people to rush out

[yotantoya]

+1 on image override

[pimperator]

+1 on image override

[mreferre]

Thanks for the interest in this topic to those that have been providing inputs. This thread has triggered some interesting internal discussions and, while we may not update these threads on a daily basis, rest assured they are closely monitored and are a great way to interact with service owners. We think we owe an update on this front.

For background, ECS Task Definitions are intended to describe the application runtime environment and dependencies for one or more containers. During the life of an application the versions of the binaries that comprise the application would likely mutate – to support this ECS supports image tags like LATEST in the image URI. However if the entire image being used were to change ECS typically views this as a new application and so expects a new definition for that applications runtime. Allowing the image URI to be mutated in place introduces some side effects. Specifically there are security considerations, potential impact on resource utilization/description and unanticipated failure modes that may be hard to recreate as a result of this override.

The team recommends creating a new Task Definition (or Task Definition Revision) in cases where the images being used by the application need to change. There are other Task Definition parameters that today are immutable and that would not have these side effects. We will evaluate those on a single parameter basis based on specific feedback.

In general, and for future guidance, it’s better to open issues that are more granular than this one to enable better tracking and better feedback. Thanks!

The ECS team

[DarwinJS]

Thanks for the interest in this topic to those that have been providing inputs. This thread has triggered some interesting internal discussions and, while we may not update these threads on a daily basis, rest assured they are closely monitored and are a great way to interact with service owners. We think we owe an update on this front.

For background, ECS Task Definitions are intended to describe the application runtime environment and dependencies for one or more containers. During the life of an application the versions of the binaries that comprise the application would likely mutate – to support this ECS supports image tags like LATEST in the image URI. However if the entire image being used were to change ECS typically views this as a new application and so expects a new definition for that applications runtime. Allowing the image URI to be mutated in place introduces some side effects. Specifically there are security considerations, potential impact on resource utilization/description and unanticipated failure modes that may be hard to recreate as a result of this override.

The team recommends creating a new Task Definition (or Task Definition Revision) in cases where the images being used by the application need to change. There are other Task Definition parameters that today are immutable and that would not have these side effects. We will evaluate those on a single parameter basis based on specific feedback.

In general, and for future guidance, it’s better to open issues that are more granular than this one to enable better tracking and better feedback. Thanks!

The ECS team

Thanks for the explanation @mreferre !

In the case of the GitLab Runner implementation on ECS Fargate, container image dynamism is critical when ECS is used to process a pipeline that consists of many container references for build and/or deploy.

Do you know if EKS backed by Fargate can allocate any container image we wish (assuming docker secrets are registered per the standard in the cluster)?

Container image dynamism is not limited in Kubernetes - so I think a lot of us pursuing ECS import that expectation - even though I can see how it might be a challenging ask to implement.

[mreferre]

@DarwinJS Using either Fargate or EC2 won't change the mechanics of the task definitions and their behavior. Thanks for the additional context and feedback.

[frosforever]

During the life of an application the versions of the binaries that comprise the application would likely mutate – to support this ECS supports image tags like LATEST in the image URI. However if the entire image being used were to change ECS typically views this as a new application and so expects a new definition for that applications runtime

@mreferre would being able to override just the image tag instead of forcing the use of a mutable tag like LATEST still suffer from the same issues?

[pimperator]

Hi Guys,
for us replacing the tag would be sufficient since we anyway do not allow to pull other than from a specific registry via iam-policy.

Our use case is to run database-migrations and afterwards update the 'real' taskdefinition both in ci/cd which for an existing and running ecs-service looks like:

• build new container (with updated database-scripts)
• upload container to ECR
• fetch active taskdefinition for ecs-service, replace CMD (and hopefully also image-tag), start new container as a single ecs-task and run database-migrations; wait until task has finished (i.e. is STOPPED)
• deploy application container by creating a new taskdefinition revision (just modify the image-tag) and update ecs-service

Of course we could use the existing taskdefinition that was created for the database migration since it already contains the proper tag but in our automation (in which we use a shell script for the last step) it is easier for the db-migration to overwrite the tag.

[mreferre]

@mreferre would being able to override just the image tag instead of forcing the use of a mutable tag like LATEST still suffer from the same issues?

Yes, it would. Basically what we were trying to convey is that for different apps you'd need different task definitions and for different versions (of the same app) you'd need different tags. Deploying LATEST and moving the tag to the latest version of the application is not good bad practice (bad choice of an example in my previous post perhaps) because you'd lose sight of which actual app version is deployed where.

[DarwinJS]

Tag would not work for my GitLab Fargate Runner use case because container based CI needs full flexibility for any image to be specified on each job.

[GitBytes]

+1 on image override

[mwayop]

I have created a cdk-fargate-gitlab-runner setup based on this example.

While all was working well for a "hello world: echo build success" pipeline, I quickly bumped into issues. I noticed the image override in the gitlab-ci.yml was ignored and found out it was due to a "limitation" in the AWS::ECS:TaskDefinition.

@mreferre:

The team recommends creating a new Task Definition (or Task Definition Revision) in cases where the images being used by the application need to change.

We would like to use Gitlab AutoDevOps which provides great of the box functionality. However the AutoDevops template consists out of many jobs, using many different docker images that may change with every new AutoDevOps version.

If I understand correctly this would mean we would have create a separate Gitlab runner (Defined as TaskDefinition) foreach image:version used during a pipeline (apart from overriding each job in the AutoDevops template to add the appropriate FARGATE_TASK_DEFINITION).

This will quickly result in tens of different Gitlab Runners, each having a specific Dockerfile and TaskDefinition needing to stay aligned with whatever images are used in the AutoDevOps template.

With my current knowledge, I would conclude that the image override limitation in the TaskDefinition (Gitlab runner on Fargate) renders a solution like Gitlab AutoDevops practically useless. Would you agree? Corrections or suggestions are most welcome of course.

[mreferre]

Unfortunately I don't have a good handle on how these runners works so I can't comment on the specific flow. But one minor correction: you don't need to create a new task definition. You need to create a new task definition revision (that is, the task definition will remain the same but it will be rev'ed). I would need to double check this but, when you launch a task or create a service, if you do not specify a revision, the last revision is picked. That is to say, you can rev your task definition without touching the deployment artifact (again I'd need to double check this). This would not be a best practice because it would be like pointing to the LATEST tag of an image but depending on the scenario and what you are testing it may work for you.

[DarwinJS]

@mwayop - I worked at GitLab and I'm the person who added the screenshotted callouts to GitLab CIs Fargate page here: https://docs.gitlab.com/runner/configuration/runner_autoscale_aws_fargate/

And we are tracking it as a known issue here: GitLab ECS Fargate Runners Ignore gitlab-ci.yml 'image:' tag

@mreferre - I had passed this use case through to the ECS team with reference to this issue a while back - we can coordinate a meeting if you wish to learn more.

Also, I have not been able to fully experiment yet, but I believe fargate backed EKS does not have this limitation (the fargate pods can run arbitrary images). I will be experimenting with EKS Blueprints soon.

[mwayop]

@mreferre

if you do not specify a revision, the last revision is picked.

Thanks for that addition, I was not aware of that and can confirm the latest revision is selected when I leave out a specific TaskDefinition version tag.

However:

This would not be a best practice because it would be like pointing to the LATEST

Agreed

On top, we would still:

• need to create a TaskDefinition (revision), and IaC, for each Docker image:tag used in the Gitlab AutoDevOps template
• override each Gitlab AutoDevOps job to specify the concerning Task Definition (leaving us with a lot of custom CICD code)

This would lead to quite some overhead in code, maintenance and AWS resources (having a separate Gitlab runner container online for each image flavor used in Auto Devops).

With every new version of Gitlab AutoDevOps our CICD might break and we need to make sure the TaskDefinition revisions are updated and aligned with all docker images used.

@DarwinJS
Thanks for your involvement and stepping in!

[chitzinwin]

c'mon man, please let us override the image in tdef.

[TomD3Wiz]

I don't care about overriding the image, but since you can override command, why can't you override the entrypoint?

[DarwinJS]

@mwayop and others who want a GitLab Fargate Runner that does not ignore the 'image' tag I have a blog article (in draft as of this writing) that let's you create an EKS Fargate Runner in less than an hour - and scale to 200 parallel jobs immediately: Iteration 1 of GitLab EKS Fargate Runners in 1 Hour and Zero Lines of Code

[duggan]

Allowing the image URI to be mutated in place introduces some side effects. Specifically there are security considerations, potential impact on resource utilization/description and unanticipated failure modes that may be hard to recreate as a result of this override.

The team recommends creating a new Task Definition (or Task Definition Revision) in cases where the images being used by the application need to change. There are other Task Definition parameters that today are immutable and that would not have these side effects. We will evaluate those on a single parameter basis based on specific feedback.

Hey @mreferre

Any sort of container overrides, including the ones already available, introduce side effects. Unless you're referring to some AWS internal systems idiosyncrasies, do you not agree that this is something for an application developer to anticipate and manage themselves?

In my own case, not having the option to override the image (or at least tag) at deploy time makes the prospect of deploying canary releases to specific users unwieldy. My task definitions are managed via Terraform, and their deployment is managed via the deploy-task-defintion Github Action. I'm not even sure how I should go about doing this in a way that doesn't result in writing brittle and verbose duplicates of everything from workflows to infrastructure code.

[bryantbiggs]

@duggan this should be possible with https://github.com/terraform-aws-modules/terraform-aws-ecs

It is not ideal, but with the use of this data source, you should be able to update the image which advances the task definition revision without causing conflicts with Terraform. The only caveat is that you should also save the image value (w/ tag) in an SSM parameter that Terraform can reference. This will ensure that if you do make any changes to the task/container definition, the correct/latest container image is still pulled when rendering those

[duggan]

Thanks @bryantbiggs – unless I'm misreading though, this only helps reduce the amount of Terraform code needed to configure any given task definition (useful though, and wish I'd spotted it a few months ago). In my case, I want to use information from my application at runtime to spin up containers with different Docker image tags for different users.

To me, a natural workflow would be to push code to a branch, build that branch and push to ECR with a particular image tag, then use that tag as part of a boto based run_task command within my application.

I'm doing this already for environment variables to turn on different features, but really there's only so much that it makes sense to configure via environment variables, particularly for risky changesets that I don't want to roll out to all users at once.

[bryantbiggs]

I think what you are looking for is AppConfig - https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-containers-agent.html

Or, the alternative is to do a for_each over your own "base" task/container definition where only the unique components are being modified for each user

[DarwinJS]

@duggan - my comment here #196 (comment) is likely to become the GitLab advised way to do GitLab runners on Fargate. The entire EKS cluster is fargate backed - so zero nodes for everything.

[mreferre]

Unless you're referring to some AWS internal systems idiosyncrasies

@duggan thanks for pushing on this. I love this discussion.

In a way, that's the core of the discussion. ECS works in a way where the source of truth/consistency is the task definition revision (e.g. when a service needs to scale out it knows that it needs to deploy the same task dev revision even though there are newer). There is a 1:1 relation between the TD revision and the image name/tag. If we were to allow overriding name/tag with the current implementation, ECS would not be able to track which image/tag was used for a particular service as that information would not be tracked.

This isn't to say that it's an implementation that can't be changed but it has a considerable amount of ramifications.

Thanks again. While I know this answer does not move the needle for you, the discussion is helping us!

[aelgn]

We are also building a new docker image:tag for each version we want to be able to deploy to test on an adhoc basis. The TaskDefinition is set up so that the application has all its dependencies - the only thing that needs to change is the :tag to deploy the new version to test.

Would love to be able to override image :tag directly instead of creating a new revision of the TD.

[jbddc]

+1 on overriding entryPoint

[a13zen]

+1 to image overrides. We're using EKS just because of it's support for image overrides instead of ECS.