ECS-Sumologic Plugin integration

[rh-lphan]

Summary

Trying to use sumologic logging driver with ECS

Description

After following instructions from

#992

Using latest AMI from amazon
Docker Vers: 17.03.2-ce
Amazon Agent: 1.14.5

As part of User data, I have added
#!/bin/bash

echo ECS_CLUSTER=PROD-MS1-Cluster >> /etc/ecs/ecs.config
echo ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","syslog","sumologic"]>> /etc/ecs/ecs.config
docker plugin install store/sumologic/docker-logging-driver:1.0.2 --grant-all-permissions --alias sumologic

I verified on the ECS instance:
docker plugin ls
ID NAME DESCRIPTION ENABLED
67f33caf0336 sumologic:latest Sumo Logic logging driver true

Expected Behavior

Container would start up without any issue

Observed Behavior

Task has error starting:

Status reason	CannotCreateContainerError: API error (500): logger: no log driver named 'sumologic' is registered
["sh","-c"]
["echo hello sumologic!"]

Environment Details

curl http://localhost:51678/v1/metadata
{"Cluster":"PROD-MS1-Cluster","ContainerInstanceArn":"arn:aws:ecs:us-west-1:312817442124:container-instance/b37bed2d-be4c-47a7-a389-dfba2481004a","Version":"Amazon ECS Agent - v1.14.5 (0dcd02c)"}

[root@ip-10-27-21-11 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 720M 7.0G 10% /
devtmpfs 2.0G 88K 2.0G 1% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/dm-3 9.8G 47M 9.2G 1% /var/lib/docker/devicemapper/mnt/606efdb898ddd5034619ba752e37cecd7a5050e6c2198fefb4c5f25928deb4a8
shm 64M 0 64M 0% /var/lib/docker/containers/b148925b664c654d9449b3428acb4b7786e506937820bb433c8cda4f27634773/shm
/dev/dm-5 9.8G 816M 8.5G 9% /var/lib/docker/devicemapper/mnt/b16710045b6d574f987a88fee172c4e458909f4d3b34da36b0b6a762bb57cc5b
shm 64M 0 64M 0% /var/lib/docker/containers/fa99885f65f988400b2a11534ec49553127cdf8bfea2b75b25725b0f812a1f0a/shm
/dev/dm-6 9.8G 830M 8.4G 9% /var/lib/docker/devicemapper/mnt/b4a26ae5f8c78ed9020bfe4cd559e3c24c6a135b2170c6759607aaabb237ddb2
shm 64M 0 64M 0% /var/lib/docker/containers/04292304c1b19563d4d661f2c7d6dd453a3a98a4ddf56841cdbbc36b2b143702/shm

Supporting Log Snippets

ecs.json.txt

Here is the ECS Agent Log

2017-10-06T21:07:53Z [DEBUG] Updating task's known status, task: FETA-miracledatetest-POC:1 arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4, TaskStatus: (NONE->STOPPED) Containers: [miracledatetest (STOPPED->STOPPED),]
2017-10-06T21:07:53Z [DEBUG] Container with earliest known container is [miracledatetest(rhartifactory-rhrelease.jfrog.io/domain/miracledate:PROD68) (STOPPED->STOPPED)] for task: FETA-miracledatetest-POC:1 arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4, TaskStatus: (NONE->STOPPED) Containers: [miracledatetest (STOPPED->STOPPED),]
2017-10-06T21:07:53Z [DEBUG] Updating task's known status to: STOPPED, task: FETA-miracledatetest-POC:1 arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4, TaskStatus: (NONE->STOPPED) Containers: [miracledatetest (STOPPED->STOPPED),]
2017-10-06T21:07:53Z [DEBUG] Updating task: [FETA-miracledatetest-POC:1 arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4, TaskStatus: (STOPPED->STOPPED) Containers: [miracledatetest (STOPPED->STOPPED),]]
2017-10-06T21:07:53Z [DEBUG] Updating task desired status to stopped because of container: [miracledatetest]; task: [FETA-miracledatetest-POC:1 arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4, TaskStatus: (STOPPED->STOPPED) Containers: [miracledatetest (STOPPED->STOPPED),]]
2017-10-06T21:07:53Z [DEBUG] Container change also resulted in task change module="TaskEngine" task="FETA-miracledatetest-POC:1 arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4, TaskStatus: (STOPPED->STOPPED) Containers: [miracledatetest (STOPPED->STOPPED),]"
2017-10-06T21:07:53Z [INFO] Task change event module="TaskEngine" event="{Attachment: TaskARN:arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4 Status:STOPPED Reason: Containers:[] Task:FETA-miracledatetest-POC:1 arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4, TaskStatus: (STOPPED->STOPPED) Containers: [miracledatetest (STOPPED->STOPPED),]}"
2017-10-06T21:07:53Z [INFO] TaskHandler, batching container event: arn:aws:ecs:us-west-1:312817442124:task/2f7fdaf8-4f69-46ae-813d-1b2d49cfdde4 miracledatetest -> STOPPED, Reason CannotCreateContainerError: API error (500): logger: no log driver named 'sumologic' is registered
, Known Sent: NONE

[samuelkarp]

Docker Vers: 17.03.2-ce

You need at least Docker 17.05.0 in order for logging driver plugins to work. See the changelog.

[rh-lphan]

Based on this link

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html

Looks like the latest ECS AMI only supports: 17.03.2

The developer who reported the fix from #992 says it worked from him. I wonder if he was using these AMI and somehow upgraded the Docker version.

I will check with him.

[samuelkarp]

You can install the ECS agent on any modern Linux AMI, including ones that have newer versions of Docker available. The requirements for a supported AMI are described here.

[mhumeSF]

I am running ubuntu 16.04 with ecs-agent running in docker at startup. But cannot start task definitions that reference sumologic as custom logging driver

[samuelkarp]

@mhumeSF What error are you receiving? What Docker version are you running? What ECS agent version are you running? What value did you configure for ECS_AVAILABLE_LOGGING_DRIVERS?

[mhumeSF]

I hadn't set ECS_AVAILABLE_LOGGING_DRIVERS. That fixed it!

[samuelkarp]

@mhumeSF Great to hear!

@rh-lphan Were you able to get the SumoLogic plugin to work with a newer version of Docker?

[rh-lphan]

Yes, I cooked up my own Centos and followed the instructions to set it up. Works great, thank you!

[samuelkarp]

Great to hear!