feat: add docker autoscaler executor

[mmoutama09]

Description

Provides a new executor using the new GitLab autoscaler executor. I've been using the fleeting plugin for AWS only.

Prerequisite: Docker must already be installed on the AMI used by worker machines (the Docker autoscaler does not install it, unlike the Docker machine). Additionally, the user used to connect to the workers must also be added to the Docker group.

Related to issue #624

Verification

Built an AMI with Docker based on Amazon Linux 2023. Set up the new executor according to the example. Works!

[github-actions]

Hey @mmoutama09! 👋

Thank you for your contribution to the project. Please refer to the contribution rules for a quick overview of the process.

Make sure that this PR clearly explains:

• the problem being solved
• the best way a reviewer and you can test your changes

With submitting this PR you confirm that you hold the rights of the code added and agree that it will published under this LICENSE.

The following ChatOps commands are supported:

• /help: notifies a maintainer to help you out

Simply add a comment with the command in the first line. If you need to pass more information, separate it with a blank line from the command.

This message was generated automatically. You are welcome to improve it.

[Tiduster]

Hi @kayman-mk.
I am a colleague of @mmoutama09 and @Kadeux.

This change could be the next major release of the module.

Gitlab is still on track to make their plugin GA this summer: https://gitlab.com/groups/gitlab-org/-/epics/6995

We are still NOT using this version in our production setup, but we will deploy it on part of our runners in June.

What should be the next steps for this PR?

Best regards,

[kayman-mk]

Sounds quite promising to get rid of the outdated docker machine. As soon as GitLab has published their module, we can integrate it here.

As far as I can see the docker machine can still be used, so we can create a feature release. Before the next major release I will check if we can get rid of docker machine to simplify the code.

Could you please post the settings to test this change?

At the moment I am working on #1117. That change will be merged before to support zero downtime during deployment of a new version.

[Tiduster]

Thanks @kayman-mk for your answer.

I was not aware of this zero downtime PR, very interesting, we can test it as well in our environment.
I will try to look at it and gave a feedback if I find something interesting

[kayman-mk]

@Tiduster Could you please post a minimal configuration showing which AMIs to use to get this up and running?

[kayman-mk]

Just tried it, but with no success. Runner is up and working. But in case a job is processed, GitLab shows

Running with gitlab-runner 16.4.2 (e77af703)
  on prod-gitlab-ci-runner-test-Gitlab-Runner-TEST-A PsqsZYpLQ, system ID: s_0a07de49d04b
Resolving secrets 00:00
Preparing the "docker-autoscaler" executor 00:50
Dialing instance i-05caf9a7284ccaxxx...
Instance i-05caf9a7284ccaxxx connected
ERROR: Failed to remove network for build
ERROR: Preparation failed: error during connect: Get "http://internel.tunnel.invalid/v1.24/info": dialing environment connection: ssh: rejected: connect failed (open failed) (docker.go:826:0s)

The Runner shows in Cloudwatch

{
    "external-address": "",
    "instance-id": "i-05caf9a7284ccaxxx",
    "internal-address": "100.64.30.16",
    "job": 5314382,
    "level": "info",
    "msg": "Dialing instance",
    "project": 987,
    "runner": "PsqsZYpLQ",
    "time": "2024-06-06T07:40:32Z",
    "use-external-address": true
}
{"error":"networksManager is undefined","job":5314382,"level":"error","msg":"Failed to remove network for build","network":"","project":987,"runner":"PsqsZYpLQ","time":"2024-06-06T07:40:50Z"}

The first error seems to be related to use_external_addr = true in the config. Changed to false.

And I noticed that Docker was not installed on the Runner and ubuntu was not part of the docker group. After fixing that, my job was executed. AMI is ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20240531, but I have no idea how to change it.

[kayman-mk]

• We mustn't use runner_worker_docker_machine_instance for configuration as it is tied to the docker+machine executor

[mmoutama09]

@kayman-mk The installation of Docker is now mandatory indeed; I've mentioned it in the usage.md file (along with adding the user in docker group).
I reused all the variables from runner_worker_docker_machine_instance to avoid the duplication of multiple variables. However, we could do it differently if we don't mind having multiple if conditions in a local block to determine which variable should be taken. What do you think?

[Tiduster]

@kayman-mk

And I noticed that Docker was not installed on the Runner and ubuntu was not part of the docker group. After fixing that, my job was executed. AMI is ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20240531, but I have no idea how to change it.

On our side we build a custom AMI from ubuntu and we add docker package manually. Docker autoscaler do not do this by default, so it require an AMI with docker engine to work.

We re-used runner_worker_docker_machine_ami_filter and runner_worker_docker_machine_ami_owners for this to no duplicate variables. We can create new variables if you prefer.

@mmoutama09 added some information about this in usage.md.

Best regards,

[mmoutama09]

@kayman-mk I've updated my code to separate docker-autoscaler from docker+machine.

To use the new docker-autoscaler we must provide an AMI with docker installed and the user used by autoscaler to connect to workers must be added to docker group.

The variable docker-registry-mirror is no longer provided as it is not in the runner autoscaler configuration, but we could add it directly to the AMI.

[kayman-mk]

Hmm, the need for a custom built image doesn't sound good to me at first hand. Any chance to use a pre-existing AMI instead? Or can we install Docker on the fly?

In case we want to host this AMI: Can you provide a built script (Packer?)?

[kayman-mk]

@Tiduster Could you please share the PAcker scripts to build the AMI? Would be a good idea to have them available and/or publish an AMI here.

[kayman-mk]

Why do we have the sg_ingresses introduced here? Just asking because there is no way to do the same for docker+machine

[kayman-mk]

@Tiduster, @mmoutama09 How do I set all the Docker options (runner_worker_docker_options) and the registry mirror?

EDIT: Docker works via old Docker options.

[kayman-mk]

Hm, tried again with version 1.0.0 of the plugin. Still have the connection issue described above.

Preparation failed: preparing environment: dial ssh: after retrying 30 times: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Checked the security groups and did a ssh from the agent to the worker. Same error. So at least no problem with creating the connection.. And I do not see any other errors in the log.

EDIT: Not sure how this is internally working. I tried sending a temporary key and login with that one. Still no success.

root@ip-192-64-74-36 bin]# aws ec2-instance-connect send-ssh-public-key --instance-id i-015020b95c394df73 --availability-zone "eu-central-1b" --instance-os-user ec2-user --ssh-public-key file:///root/.ssh/id_ed25519.pub --region eu-central-1
{
    "RequestId": "2c73837e-cf97-42f5-a415-0b0639a8b196",
    "Success": true
}
[root@ip-192-64-74-36 bin]# ssh [email protected]
The authenticity of host '192.64.35.158 (192.64.35.158)' can't be established.
ED25519 key fingerprint is SHA256:i8JpNRBtjgOCatirWNpfJgMbDQt9Yp5ZtgHHhMD0pvk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.64.35.158' (ED25519) to the list of known hosts.
[email protected]: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

AMI is amazon/al2023-ami-ecs-hvm-2023.0.20240723-kernel-6.1-x86_64

[Tiduster]

It uses EC2 connect in the background, as visible in the recommended IAM Policy: https://gitlab.com/gitlab-org/fleeting/plugins/aws#recommended-iam-policy

We didn't have this issue on our side, maybe something is missing in the module.

We can fix it monday :-) .

[Tiduster]

After more testing, the ECS official AMI is just not compatible.

We didn't dive too much, because as it's not working out of the box, using this AMI is useless in our use case :-( .
We assume there is something in the SSH default configuration or something that is preventing fleeting plugin to connect to the worker when this AMI is in use.

We DO NOT have this issue with a simple custom AMI pre-installed with docker, using Ubuntu or Amazon Linux 2023.

Unfortunately, baking a custom AMI is once again mandatory.

[kayman-mk]

@Tiduster @mmoutama09 Thanks for adding this function to the project. Great job!

[kayman-mk]

To be merged now. Before the release I will do a last check. Everything is working out of the box. Provisioning new machines seems to be 25% faster. I love it!

[Tiduster]

Thanks for all the fixes.
We will switch part of our production setup on this new version ASAP to detect any remaining issue.

Hopefully it has less bug than docker+machine ^^·