FIFO queue

[davidtheclark]

We'd like to use a FIFO SQS queue instead of the standard SQS queue that Watchbot currently ships with.

@rclark I know you started work on this in https://github.com/mapbox/ecs-watchbot/compare/fifo. Do you have any idea how far out this might be? And is there anything we could do to help it along?

In some discussion you suggested this FIFO branch "hard-wires you to performing builds one at a time". In theory, I think a FIFO queue should allow multiple simultaneous jobs as long as they have different message group IDs. It sounds like the downside of this is that the new scaling logic wouldn't work nicely. If you're expecting the queue to include only one message group, then you'd only ever need one machine and it would be wasteful to spin up more machines based on the length of the queue. However, if you're expecting the queue to include multiple message groups (as we are), then with only one machine you'd be slowing things down, requiring messages in unrelated groups to wait for each other.

One idea: What if the scaling behavior for a FIFO queue is configurable? You could decide whether you accept one build at a time (because you won't be using multiple message groups at all or very much), or you accept the additional costs of scaling (because you expect that you'll rarely have messages from the same group overlapping).

Another idea: Maybe I'm thinking about this wrong, and should instead make our own FIFO queue that triggers a CodeBuild project?

cc @mapbox/frontend-platform @jseppi @emilymdubois

[rclark]

Let's push this discussion forward in #279 and land FIFO queues as a new feature.