Update buildkite-metrics to use the agent metrics api

[sj26]

This is super rough, but it works.

[lox]

This is a good reduction in code bulk. Will probably needs some explanation in the README

[lox]

Random idea, but what if we didn't pass the agent registration token over the wire and instead used a HMAC'd value with the account_id? Still vulnerable to reply attacks, but at least you can't then use that token to register an agent.

[sj26]

@lox I'd love to look into a better authentication exchange, especially I'd love to homogenize our tokens so that a cluster token can do some limited agent api stuff, and an agent token can do limited rest api stuff, etc. It'd be a little interesting, trying to create tokens which included enough information to do a mac exchange. But I think we're sufficiently protected against replay by TLS for the moment:

https://www.ssllabs.com/ssltest/analyze.html?d=agent.buildkite.com

[lox]

I wasn't proposing any sort of key exchange @sj26, just rather than passing in BUILDKITE_AGENT_TOKEN, we'd pass in hmac($BUILDKITE_AGENT_TOKEN, $ACCOUNT_ID) and use that as the secret.

It means we aren't encouraging people to put the BUILDKITE_AGENT_TOKEN in env anywhere.

[sj26]

Right, right — for the agent setup and env, not the request channel. Yeah, that also kinda gels with what I mean. I was proposing opaque authorization tokens which actually pack information, like our graphql IDs, e.g. Authorization: Token base64(pack($TOKEN_UUID+$NONCE+hmac($TOKEN_SECRET,$TOKEN_UUID+$NONCE))), so BK can unpack the uuid to lookup the agent registration token and then validate the hmac — and you could generate that on the buildkite.com side to feed in as an env. I was misinterpreting and thinking the metrics agent would also then perform hmac for the request channel, but that's probably superfluous.

But yeah, step 2.

[lox]

Gotcha! That makes sense. Reckon getting it working like it is makes sense for now anyway.

[lox]

Tests are looking awesome @sj26

[sj26]

@lox thanks! They're rough, but they work. I think maybe some README updates and this is good for release, with some published caveats that it drops a bunch of metrics from the previous version around builds and historical — this is purely for job/agent workload metrics. Do you know many folks using it beyond the elastic stack?

[lox]

Happy to merge with some README changes!

[lox]

🚢