Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EcsCommandExecutor: return failure status when exit code is empty #1724

Conversation

builtinnya
Copy link
Contributor

@builtinnya builtinnya commented Mar 10, 2022

Problem

When a Docker container can't be started for some reason, Amazon ECS agent emits DockerTimeoutError: Could not transition to started; timed out after waiting 3m0s and the ECS task has STOPPED status and no exitCode.

Currently, EcsCommandExecutor treats the stopped containers without exitCode as successful (exit code = 0) and workflows continue.

nextStatus.put("status_code", task.getContainers().get(0).getExitCode());

@Override
public int getStatusCode()
{
return json.get("status_code").intValue();
}

(JsonNode#intValue() returns 0 for non-number nodes such as null.)

Containers that are not even started should not be treated as sucessful.

What this PR does

This PR tries to fix the issue by treating the exit status of containers without exitCode as 1 (error) to make it fail.

On reproduction

It's difficult to establish stable steps to reproduce this issue because it's hard to make ECS agents fail with DockerTimeoutError appropriately but it should be easy to see what happens when ECS tasks have no exitCode and why it should be treated as error.

@myui myui self-requested a review April 23, 2022 19:07
@szyn szyn added this to the v0.10.5 milestone Feb 9, 2023
Copy link
Member

@szyn szyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for creating this PR, too. This also sounds reasonable and looks good to me.

@szyn szyn merged commit 10843b3 into treasure-data:master Feb 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants