Bitbucket runner support

[DavidGOrtega]

• closes Support for Bitbucket Cloud Pipelines self-hosted runners #646
• Refactors runner a bit

[DavidGOrtega]

This is an ideal solution (that unfortunately does not work) without having to use docker, just the Java agent.
I keep it for the record just in case someday.

async startRunner(opts) {
    const { projectPath } = this;
    const { workdir, name, labels: labelsstr } = opts;
    
    const labels = labelsstr.split(',')
    const os = process.platform === 'darwin' ? 'macos' : 'linux';
    labels.push(os);

    try {
      const { uuid: accountId } = await this.request({ endpoint: `/user` });
      const { uuid: repoId } = await this.request({
        endpoint: `/repositories/${projectPath}`
      });
      const {
        uuid,
        oauth_client: { id, secret }
      } = await this.registerRunner({ name, labels });

      const destination = resolve(workdir, 'bb-runner.tar.gz');
      const ver = '1.322';
      const url = `https://product-downloads.atlassian.com/software/bitbucket/pipelines/atlassian-bitbucket-pipelines-runner-${ver}.tar.gz`;
      await download({ url, path: destination });
      await tar.extract({ file: destination, cwd: workdir });
      await exec(`chmod -R 777 ${workdir}`);

      if (os==='linux') {
        await exec(`
sudo tee /usr/bin/retry << 'EOF'
#!/bin/bash
exec env GIT_LFS_SKIP_SMUDGE=1 "$\{@:2}"
EOF'
`);
        await exec(`chmod 777 /usr/bin/retry`);
      }

      const command = `java -jar \
      -Dbitbucket.pipelines.runner.account.uuid=${accountId} \
      -Dbitbucket.pipelines.runner.repository.uuid=${repoId} \
      -Dbitbucket.pipelines.runner.uuid=${uuid} \
      -Dbitbucket.pipelines.runner.oauth.client.id=${id} \
      -Dbitbucket.pipelines.runner.oauth.client.secret=${secret} \
      -Dbitbucket.pipelines.runner.directory.working=${workdir}/temp \
      -Dbitbucket.pipelines.runner.runtime=${os}-bash \
      -Dbitbucket.pipelines.runner.docker.uri=unix:///var/run/docker.sock \
      -Dbitbucket.pipelines.runner.scheduled.state.update.initial.delay.seconds=0 \
      -Dbitbucket.pipelines.runner.scheduled.state.update.period.seconds=30 \
      -Dbitbucket.pipelines.runner.environment=PRODUCTION \
      -Dfile.encoding=UTF-8 \
      -Dsun.jnu.encoding=UTF-8 \
      ${workdir}/bin/runner.jar`
      
      return spawn(command, { shell: true });
    } catch (err) {
      throw new Error(`Failed preparing runner: ${err.message}`);
    }
  }

[casperdcl]

Is this ready for review?

• it's ok if Docker and/or GPUs aren't fully supported yet
• need a corresponding PR on https://cml.dev/doc/self-hosted-runners#allocating-cloud-compute-resources-with-cml (adding a Bitbucket tab with a workflow)
  • no need to do the whole 4-example-repo-cases thing (yet)

[DavidGOrtega]

@DavidGOrtega, if cml runner --single is not supported, should it print a fatal error when launched with that flag?

😮 single was supported for everyone as far as I remembered! I mean, not depending on the capability. GH originally did not support it on its own and we had a mechanism in place... Did we remove it?

[DavidGOrtega]

@0x2b3bfa0 now we are honouring single. We can indeed use this method for anyone instead of relying in native single.
We would fix

• runner error logs, destructure id of undefined #990

[0x2b3bfa0]

Note that, unlike on GitHub and GitLab, Bitbucket will fail immediately if there isn't any self-hosted runner able to satisfy the specified runs-on parameter.

[0x2b3bfa0]

@0x2b3bfa0 now we are honouring single. We can indeed use this method for anyone instead of relying in native single.

Note that runner-side cml runner --single might be inconsistent and take a second job before disposing itself like in actions/runner#510. Bitbucket may (or may not) have the same issue.

[0x2b3bfa0]

Generating Application Passwords

Use https://bitbucket.org/account/settings/app-passwords/new to generate a new application password.

cml runner

cml rerun-workflow

[0x2b3bfa0]

The following bitbucket-pipelines.yml file can be triggered manually from the Bitbucket Cloud user interface:

pipelines:
  custom:
    test:
    - step:
        image: node
        script:
        - curl https://releases.hashicorp.com/terraform/1.2.1/terraform_1.2.1_linux_amd64.zip | funzip > /bin/terraform && chmod 755 /bin/terraform
        - npx github:iterative/cml#bb-runner runner --cml-version=github:iterative/cml#bb-runner --cloud=aws --cloud-region=us-west-1 --cloud-type=g2.2xlarge --labels=example
    - step:
        runs-on: [self.hosted, example]
        image: alpine
        script:
        - echo hello world

Note
Use secured user-defined variables to pass cloud & forge credentials.

[0x2b3bfa0]

It works!™ Created #1015 collecting potential enhancements and iterative/cml.dev#243 for documentation.

[0x2b3bfa0]

@DavidGOrtega, any last will before merging?

[DavidGOrtega]

Note that, unlike on GitHub and GitLab, Bitbucket will fail immediately if there isn't any self-hosted runner able to satisfy the specified runs-on parameter.

I do not see what its your point. GH also failed in the past if we had no runner to execute a job. Is there anything we can do about it? Indeed the wait mechanism that we have in terraform covered to needs:

• know if something went wrong like a bad token
• be able to wait to have the runner in place so the next job will have the runner available and not fail in GH

Note that runner-side cml runner --single might be inconsistent and take a second job before disposing itself like in actions/runner#510. Bitbucket may (or may not) have the same issue.

No if... we remove the 10 seconds wait. Indeed we could control that issue in every runner forcing the kill of the process since we are the ones that cleanups the resources. Being polite we are having issues like #990
Im working on that 😉

[0x2b3bfa0]

I do not see what its your point. GH also failed in the past if we had no runner to execute a job. Is there anything we can do about it? Indeed the wait mechanism that we have in terraform covered to needs:

• know if something went wrong like a bad token
• be able to wait to have the runner in place so the next job will have the runner available and not fail in GH

My point is that iterative/terraform-provider-iterative#315 is still invalid, and we still need the readiness check for some of the reasons you mention plus this one.

[0x2b3bfa0]

No if... we remove the 10 seconds wait. Indeed we could control that issue in every runner forcing the kill of the process since we are the ones that cleanups the resources. Being polite we are having issues like #990. I'm working on that 😉

Great! While it could still (theoretically) take a second job in the time it takes us to parse the completion logs, that's unlikely to happen in practice. May work. 🤷🏼‍♂️