Critical CVE-2021-44906 in minimist in image summerwind/actions-runner

[lhaussknecht]

Trivy reports a critical CVE CVE-2021-44906 in a minimist dependency. I would love to create a PR but have no idea, where this is comming from...

The image sha, that was tested is: sha256:0580b846ff609f9bd6092eb173b8d0d967848e1ea349937bf4a441ce773b0f46

[mumoshu]

Perhaps it's a part of nodejs dependencies that actions/runner needs to run node-based actions? 🤔

[lhaussknecht]

Yes. It is a dependency of the runner. It was fixed in actions/runner#1783. But it seems, that the old version is included in the runner release.

[mumoshu]

Thanks. actions/runner@0343e76 says it should be included since v2.290.0. What was the tag of our runner image on which you ran trivy? If it was tagged v0.290.0 or greater, can it be a bug in actions/runner's release process that the release archive contains outdated dependencies? 🤔

[lhaussknecht]

Yes. The current release downloads have Minimist in ./externals/node12/lib/node_modules/npm/node_modules/minimist.

The package.json there is pointing to "_id": "[email protected]",. So this seems to be a problem in the release pipeline in actions/runner.

Will open a ticket there.

[BeyondEvil]

Could the issue I'm seeing be related to the CVE?

[Mon May  9 17:38:41 UTC 2022] [INFO] [/usr/local/bin/startup.sh] Using /etc/docker/daemon.json with the following content
{}
[Mon May  9 17:38:41 UTC 2022] [INFO] [/usr/local/bin/startup.sh] Using /etc/supervisor/conf.d/dockerd.conf with the following content
[program:dockerd]
command=/usr/local/bin/dockerd
autostart=true
autorestart=true
stderr_logfile=/var/log/dockerd.err.log
stdout_logfile=/var/log/dockerd.out.log
[Mon May  9 17:38:41 UTC 2022] [INFO] [/usr/local/bin/startup.sh] Starting supervisor
[Mon May  9 17:38:41 UTC 2022] [INFO] [/usr/local/bin/startup.sh] Waiting for processes to be running
[Mon May  9 17:38:41 UTC 2022] [INFO] [/usr/local/bin/startup.sh] Process dockerd is not running yet. Retrying in 1 seconds
[Mon May  9 17:38:41 UTC 2022] [INFO] [/usr/local/bin/startup.sh] Waited 0 seconds of 30 seconds
[Mon May  9 17:38:42 UTC 2022] [INFO] [/usr/local/bin/startup.sh] Process dockerd is not running yet. Retrying in 1 seconds
[Mon May  9 17:38:42 UTC 2022] [INFO] [/usr/local/bin/startup.sh] Waited 1 seconds of 30 seconds
[Mon May  9 17:38:43 UTC 2022] [INFO] [/usr/local/bin/startup.sh] dockerd is running
Docker enabled runner detected and Docker daemon wait is enabled
Waiting until Docker is avaliable or the timeout is reached
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
Github endpoint URL https://github.com/
Passing --ephemeral to config.sh to enable the ephemeral runner.
Passing --disableupdate to config.sh to disable automatic runner updates.
Configuring the runner.

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------
Cannot configure the runner because it is already configured. To reconfigure the runner, run 'config.cmd remove' or './config.sh remove' first.
Runner successfully configured.
{
  "agentId": 4410,
  "agentName": "proxyco-small-runner-bwl57-46zq9",
  "poolId": 1,
  "poolName": "Default",
  "disableUpdate": true,
  "ephemeral": true,
  "serverUrl": "https://pipelines.actions.githubusercontent.com/yFmGis1EotTFQMY1Zckk0xewcjSbf7p5TvcDl4NVLmtdn4kvAG",
  "gitHubUrl": "https://github.com/proxyco",
  "workFolder": "/home/runner/work/"
mkdir: cannot create directory './externals': File exists
mv: cannot stat './externalstmp/*': No such file or directory
}
√ Connected to GitHub

Current runner version: '2.289.1'
2022-05-09 17:38:58Z: Listening for Jobs
An error occurred: Runner version v2.289.1 is deprecated and cannot receive messages.

I'm not sure I understand the last line?

Should we simply upgrade to the latest Runner version?

[mumoshu]

@BeyondEvil Hey! Not at all. It's saying what the error message tells- you're using an outdated version of actions/runner and you should definitely upgrade to the latest.

[BeyondEvil]

@BeyondEvil Hey! Not at all. It's saying what the error message tells- you're using an outdated version of actions/runner and you should definitely upgrade to the latest.

Is it common practice to deprecate runners that are only ~2 months old?

That's why I wondered if it was related to the CVE.

I looked and looked, and I couldn't find any information about the deprecation.

[mumoshu]

@BeyondEvil You might better ask actions/runner maintainers. Anyway, my take is that it is generally recommended to stick to the latest version if you have no facility to patch what are included in the actions/runner release archives yourself.

It's a single release channel(there's no alpha, beta, stable release channels like linux distros for actions/runner, right?) and my straight interpretation of that fact is that you can't expect patches or security fixes to older runner versions.

[ethomson]

Is it common practice to deprecate runners that are only ~2 months old?

@BeyondEvil The runner updates itself automatically. If you disable this automatic update, you must update manually. If you do not update your runners within 30 days, they will stop receiving jobs. There's more information in the documentation.

[BeyondEvil]

I see @ethomson, thanks for making that clear(er).

I do have one question tho, if I understand correctly, it's not the summerwind/actions-runner-dind image that I have to bump, it's basically setting DISABLE_RUNNER_UPDATE to false (or omitting it, since false seems to be the default).

Would a consequence of not updating the summerwind image, be that the runner will be updated on every invocation of an ephemeral runner?

[ethomson]

Would a consequence of not updating the summerwind image, be that the runner will be updated on every invocation of an ephemeral runner?

Correct. Once the runner is out of date it will update itself before any job is run. It will add unnecessary network ingress and a delay in starting every job.

[BeyondEvil]

Would a consequence of not updating the summerwind image, be that the runner will be updated on every invocation of an ephemeral runner?

Correct. Once the runner is out of date it will update itself before any job is run. It will add unnecessary network ingress and a delay in starting every job.

So what's best practice here considering we're using ephemeral runners? 😅

[toast-gear]

So what's best practice here considering we're using ephemeral runners? 😅

Keep your runner images current. I mean it's not so much best practice as enforced by GitHub. Much of the Actions logic to support a new feature is in the actions/runner software and so it needs to be current to support new features, that's why they enforce people updating with a short 30 day grace period. We automatically build a new image if actions/runner produce a new release so you either need to:

• reference 1 of our immutable tags and automate bumping to the newer image with the newer immutable tag when we release one
• reference our mutable latest tag and always pull it or detect when it changes and pull the new image on the new tag
• reference our mutable latest tag and pull the image on a schedule, daily or weekly or whatever

The main point I'm driving at here is ARC can't really help here, there is some automation required in your environment.

We've got reports the self-update process can result in large delays in job allocation #1387 and so disabling the update is probably best, I also have some reports from ex-employers of this too. Additionally, it breaks the immutable concept that a container is supposed to provide so it's not ideal to begin with. The downside of turning self-updates off though is you'll need to implement some automation.

[BeyondEvil]

Keep your runner images current. I mean it's not so much best practice as enforced by GitHub. Much of the Actions logic to support a new feature is in the actions/runner software and so it needs to be current to support new features, that's why they enforce people updating with a short 30 day grace period. We automatically build a new image if actions/runner produce a new release so you either need to:

• reference 1 of our immutable tags and automate bumping to the newer image with the newer immutable tag when we release one
• reference our mutable latest tag and always pull it or detect when it changes and pull the new image on the new tag
• reference our mutable latest tag and pull the image on a schedule, daily or weekly or whatever

Even using the latest tag would require restarting the actions-runner-controller pod, right? Including setting the ImagePullPolicy to Always.

The main point I'm driving at here is ARC can't really help here, there is some automation required in your environment.

Yeah, totally get that. But I figured you of anyone probably have some ideas of how to deal with it. 😊

We've got reports the self-update process can result in large delays in job allocation #1387 and so disabling the update is probably best, I also have some reports from ex-employers of this too. Additionally, it breaks the immutable concept that a container is supposed to provide so it's not ideal to begin with. The downside of turning self-updates off though is you'll need to implement some automation.

Yeah, it was partly due to this we turned it off.

[toast-gear]

Even using the latest tag would require restarting the actions-runner-controller pod, right? Including setting the ImagePullPolicy to Always.

Yup, it's probably easier to reference an immutable tag and automate bumping it but it's all quite environment specific, people deploy and manage runners is lots and ways

[BeyondEvil]

Even using the latest tag would require restarting the actions-runner-controller pod, right? Including setting the ImagePullPolicy to Always.

Yup, it's probably easier to reference an immutable tag and automating bumping it but it's all quite environment specific, people deploy and manage runners is lots and ways

Thanks, this was really helpful! I now have a path forward. 🙇

With a little luck I can trigger a workflow based on new releases in the actions/runner repo.

[toast-gear]

@BeyondEvil. if you do end up getting something together, sharing what you can as a show and tell in discussions is always welcomed! This is going to be an extremely common problem to tackle so I'm sure the community will appreciate some examples of some working solutions out there!

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.