Potential AWS ECS Fargate container ID incompatibility

[stevejgordon]

Through a customer support case, we've been notified of a failure to parse the container ID from /proc/self/cgroup for a service running on ECS Fargate. In this customer's case, the container ID does not seem to conform to the expected format of the Docker ID in AWS ECS Fargate, which, per this documentation, is expected to be 32 hex characters followed by a dash and then ten numeric digits.

In this customer case, in two out of 30 services, there are only nine numeric digits after the dash. It is unclear if this is a bug in AWS, but as it is not documented, it could well be. AWS support has been unable to reproduce this on their end. Since our specification uses a strict regex to match the possibility of an ECS Fargate format, for this customer, our APM agents will be unable to parse the container ID while using our regex.

I've also reached out to a contact at AWS to see if they have any thoughts. In the meantime, should we loosen our regex to allow for the possibility of nine digits?

[trentm]

@Mpdreamz brought this up at the last APM Aegnts weekly. I think, probably yes we should relax the regex in our agents spec.

It would be wonderful if there was a reference to code or something authoritative from AWS for how those cgroup names are generated.

[Mpdreamz]

See also: #886

I had no luck finding anything authoritative on how these get generated. I am very interested in understanding this better too.

[stevejgordon]

I also dug around with no luck finding any specific implementation details.

[xrmx]

Has it been checked that the value in /proc/self/cgroup is the same as the one in the metadata endpoint response?

[stevejgordon]

Has it been checked that the value in /proc/self/cgroup is the same as the one in the metadata endpoint response?

I've asked if the customer can provide this to compare them.

[xrmx]

Has it been checked that the value in /proc/self/cgroup is the same as the one in the metadata endpoint response?

I've asked if the customer can provide this to compare them.

Looks like the id is the same but they managed to get a 8 digits long one 😓

[xrmx]

At 28/8 APM weekly call we decided to relax the match to {1,10} digits, would this work when parsing the cgroup file?

[stevejgordon]

At 28/8 APM weekly call we decided to relax the match to {1,10} digits, would this work when parsing the cgroup file?

Yes, this seems the most pragmatic option since AWS sometimes does not align with their documented length!

[Mpdreamz]

Closing in favour of meta issue: #888