From 814968e220494473b29db0fa42ef3ca89a2f0a93 Mon Sep 17 00:00:00 2001 From: Paul Podgorsek Date: Fri, 30 Aug 2024 22:39:50 +0100 Subject: [PATCH] Issue #503 - Add CI pipeline example for Azure DevOps --- README.md | 260 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 167 insertions(+), 93 deletions(-) diff --git a/README.md b/README.md index bba1b4c..6abc54d 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,8 @@ * [Installing additional dependencies](#installing-additional-dependencies) * [Security consideration](#security-consideration) * [Continuous integration](#continuous-integration) + * [Azure DevOps pipeline](#ci-azure-devops) + * [Jenkins pipeline](#ci-jenkins) * [Defining a test run ID](#defining-a-test-run-id) * [Upload test reports to an AWS S3 bucket](#upload-test-reports-to-an-aws-s3-bucket) * [Testing this project](#testing-this-project) @@ -74,10 +76,12 @@ As stated by [the official GitHub project](https://github.com/robotframework/Sel This container can be run using the following command: - docker run \ - -v :/opt/robotframework/reports:Z \ - -v :/opt/robotframework/tests:Z \ - ppodgorsek/robot-framework: +```sh +docker run \ + -v :/opt/robotframework/reports:Z \ + -v :/opt/robotframework/tests:Z \ + ppodgorsek/robot-framework: +``` @@ -114,9 +118,11 @@ It is possible to use different directories to read tests from and to generate r It is possible to parallelise the execution of your test suites. Simply define the `ROBOT_THREADS` environment variable, for example: - docker run \ - -e ROBOT_THREADS=4 \ - ppodgorsek/robot-framework:latest +```sh +docker run \ + -e ROBOT_THREADS=4 \ + ppodgorsek/robot-framework:latest +``` By default, there is no parallelisation. @@ -126,10 +132,12 @@ By default, there is no parallelisation. When using parallelisation, it is possible to pass additional [pabot options](https://github.com/mkorpela/pabot#command-line-options), such as `--testlevelsplit`, `--argumentfile`, `--ordering`, etc. These can be passed by using the `PABOT_OPTIONS` environment variable, for example: - docker run \ - -e ROBOT_THREADS=4 \ - -e PABOT_OPTIONS="--testlevelsplit" \ - ppodgorsek/robot-framework:latest +```sh +docker run \ + -e ROBOT_THREADS=4 \ + -e PABOT_OPTIONS="--testlevelsplit" \ + ppodgorsek/robot-framework:latest +``` @@ -137,9 +145,11 @@ When using parallelisation, it is possible to pass additional [pabot options](ht RobotFramework supports many options such as `--exclude`, `--variable`, `--loglevel`, etc. These can be passed by using the `ROBOT_OPTIONS` environment variable, for example: - docker run \ - -e ROBOT_OPTIONS="--loglevel DEBUG" \ - ppodgorsek/robot-framework:latest +```sh +docker run \ + -e ROBOT_OPTIONS="--loglevel DEBUG" \ + ppodgorsek/robot-framework:latest +``` @@ -157,9 +167,11 @@ This project is meant to allow your tests to run anywhere. Sometimes that can be To set the timezone used inside the Docker image, you can set the `TZ` environment variable: - docker run \ - -e TZ=America/New_York \ - ppodgorsek/robot-framework:latest +```sh +docker run \ + -e TZ=America/New_York \ + ppodgorsek/robot-framework:latest +``` @@ -169,10 +181,12 @@ It is possible to install additional dependencies dynamically at runtime rather To do so, simply mount a text file containing the list of dependencies you would like to install using `pip`: (by default, this file is empty if not mounted) - docker run \ - -v :/opt/robotframework/pip-requirements.txt:Z \ - -v :/opt/robotframework/tests:Z \ - ppodgorsek/robot-framework:latest +```sh +docker run \ + -v :/opt/robotframework/pip-requirements.txt:Z \ + -v :/opt/robotframework/tests:Z \ + ppodgorsek/robot-framework:latest +``` The file must follow [Pip's official requirements file format](https://pip.pypa.io/en/stable/reference/requirements-file-format/). @@ -191,9 +205,11 @@ rpa==1.50.0 By default, containers are implicitly run using `--user=1000:1000`, please remember to adjust that command-line setting accordingly, for example: - docker run \ - --user=1001:1001 \ - ppodgorsek/robot-framework:latest +```sh +docker run \ + --user=1001:1001 \ + ppodgorsek/robot-framework:latest +``` Remember that that UID/GID should be allowed to access the mounted volumes in order to read the test suites and to write the output. @@ -208,42 +224,90 @@ This is a good security practice to make sure containers cannot perform unwanted ## Continuous integration + + +### Azure DevOps pipeline + +Azure DevOps provides a `Docker@2` task but it only allows a limited number of commands, it is therefore required to use a `script` instead: + +```yml +- job: FunctionalRegressionTests + displayName: Robot Framework tests + steps: + - script: | + set -x + + mkdir -p $(Build.Repository.LocalPath)/robot-framework-reports + chmod -R ugo+rwx $(Build.Repository.LocalPath)/robot-framework-reports + + # The ROBOT_OPTIONS allow to generate a specific output file that can be + # used to publish test results in the Azure DevOps pipeline run + # Remember to replace ${{ parameters.robotFrameworkFolder }} by the correct + # folder name in your repository + docker run --rm \ + --shm-size=1g \ + -v $(Build.Repository.LocalPath)/robot-framework-reports:/opt/robotframework/reports:Z \ + -v $(Build.Repository.LocalPath)/${{ parameters.robotFrameworkFolder }}:/opt/robotframework/tests:Z \ + -e ROBOT_OPTIONS="-x outputxunit.xml" \ + -t docker.io/ppodgorsek/robot-framework:latest + displayName: Run Robot Framework tests + + - task: PublishTestResults@2 + condition: always() + displayName: Publish Robot Framework test results + inputs: + testResultsFormat: 'JUnit' + testResultsFiles: '**/outputxunit.xml' + searchFolder: '$(Build.Repository.LocalPath)/robot-framework-reports' + failTaskOnFailedTests: true + failTaskOnFailureToPublishResults: true + failTaskOnMissingResultsFile: true +``` + + + +### Jenkins pipeline + It is possible to run the project from within a Jenkins pipeline by relying on the shell command line directly: - pipeline { - agent any - stages { - stage('Functional regression tests') { - steps { - sh "docker run --shm-size=1g -e BROWSER=firefox -v $WORKSPACE/robot-tests:/opt/robotframework/tests:Z -v $WORKSPACE/robot-reports:/opt/robotframework/reports:Z ppodgorsek/robot-framework:latest" - } - } - } +```yml +pipeline { + agent any + stages { + stage('Functional regression tests') { + steps { + sh "docker run --shm-size=1g -e BROWSER=firefox -v $WORKSPACE/robot-tests:/opt/robotframework/tests:Z -v $WORKSPACE/robot-reports:/opt/robotframework/reports:Z ppodgorsek/robot-framework:latest" + } } + } +} +``` The pipeline stage can also rely on a Docker agent, as shown in the example below: - pipeline { - agent none - stages { - stage('Functional regression tests') { - agent { docker { - image 'ppodgorsek/robot-framework:latest' - args '--shm-size=1g -u root' } - } - environment { - BROWSER = 'firefox' - ROBOT_TESTS_DIR = "$WORKSPACE/robot-tests" - ROBOT_REPORTS_DIR = "$WORKSPACE/robot-reports" - } - steps { - sh ''' - /opt/robotframework/bin/run-tests-in-virtual-screen.sh - ''' - } - } - } +```yml +pipeline { + agent none + stages { + stage('Functional regression tests') { + agent { docker { + image 'ppodgorsek/robot-framework:latest' + args '--shm-size=1g -u root' } + } + environment { + BROWSER = 'firefox' + ROBOT_TESTS_DIR = "$WORKSPACE/robot-tests" + ROBOT_REPORTS_DIR = "$WORKSPACE/robot-reports" + } + steps { + sh ''' + /opt/robotframework/bin/run-tests-in-virtual-screen.sh + ''' + } } + } +} +``` @@ -257,9 +321,11 @@ For that purpose, the `ROBOT_TEST_RUN_ID` variable was introduced: It can simply be passed during the execution, such as: - docker run \ - -e ROBOT_TEST_RUN_ID="feature/branch-name" \ - ppodgorsek/robot-framework:latest +```sh +docker run \ + -e ROBOT_TEST_RUN_ID="feature/branch-name" \ + ppodgorsek/robot-framework:latest +``` By default, the test run ID is empty. @@ -269,12 +335,14 @@ By default, the test run ID is empty. To upload the report of a test run to an S3 bucket, you need to define the following environment variables: - docker run \ - -e AWS_ACCESS_KEY_ID= \ - -e AWS_SECRET_ACCESS_KEY= \ - -e AWS_DEFAULT_REGION= \ - -e AWS_BUCKET_NAME= \ - ppodgorsek/robot-framework:latest +```sh +docker run \ + -e AWS_ACCESS_KEY_ID= \ + -e AWS_SECRET_ACCESS_KEY= \ + -e AWS_DEFAULT_REGION= \ + -e AWS_BUCKET_NAME= \ + ppodgorsek/robot-framework:latest +``` @@ -282,35 +350,39 @@ To upload the report of a test run to an S3 bucket, you need to define the follo Not convinced yet? Simple tests have been prepared in the `test/` folder, you can run them using the following commands: - # Using Chromium - docker run \ - -v `pwd`/reports:/opt/robotframework/reports:Z \ - -v `pwd`/test:/opt/robotframework/tests:Z \ - -e BROWSER=chrome \ - ppodgorsek/robot-framework:latest - - # Using Firefox - docker run \ - -v `pwd`/reports:/opt/robotframework/reports:Z \ - -v `pwd`/test:/opt/robotframework/tests:Z \ - -e BROWSER=firefox \ - ppodgorsek/robot-framework:latest +```sh +# Using Chromium +docker run \ + -v `pwd`/reports:/opt/robotframework/reports:Z \ + -v `pwd`/test:/opt/robotframework/tests:Z \ + -e BROWSER=chrome \ + ppodgorsek/robot-framework:latest + +# Using Firefox +docker run \ + -v `pwd`/reports:/opt/robotframework/reports:Z \ + -v `pwd`/test:/opt/robotframework/tests:Z \ + -e BROWSER=firefox \ + ppodgorsek/robot-framework:latest +``` For Windows users who use **PowerShell**, the commands are slightly different: - # Using Chromium - docker run \ - -v ${PWD}/reports:/opt/robotframework/reports:Z \ - -v ${PWD}/test:/opt/robotframework/tests:Z \ - -e BROWSER=chrome \ - ppodgorsek/robot-framework:latest - - # Using Firefox - docker run \ - -v ${PWD}/reports:/opt/robotframework/reports:Z \ - -v ${PWD}/test:/opt/robotframework/tests:Z \ - -e BROWSER=firefox \ - ppodgorsek/robot-framework:latest +```sh +# Using Chromium +docker run \ + -v ${PWD}/reports:/opt/robotframework/reports:Z \ + -v ${PWD}/test:/opt/robotframework/tests:Z \ + -e BROWSER=chrome \ + ppodgorsek/robot-framework:latest + +# Using Firefox +docker run \ + -v ${PWD}/reports:/opt/robotframework/reports:Z \ + -v ${PWD}/test:/opt/robotframework/tests:Z \ + -e BROWSER=firefox \ + ppodgorsek/robot-framework:latest +``` Screenshots of the results will be available in the `reports/` folder. @@ -384,11 +456,13 @@ As mentioned on the [Docker Hub](https://hub.docker.com/r/ppodgorsek/robot-frame As mentioned in the official documentation, [Podman](https://docs.podman.io/en/latest/markdown/podman-run.1.html#platform-os-arch) and [Docker](https://docs.docker.com/build/building/multi-platform/) provide a `--platform` option which selects a given application architecture, such as: - docker run \ - --platform linux/amd64 \ - -v :/opt/robotframework/reports:Z \ - -v :/opt/robotframework/tests:Z \ - ppodgorsek/robot-framework: +```sh +docker run \ + --platform linux/amd64 \ + -v :/opt/robotframework/reports:Z \ + -v :/opt/robotframework/tests:Z \ + ppodgorsek/robot-framework: +``` Please note: builds and automated tests of this project will remain performed on a `linux/amd64` architecture so such emulation might not work, depending on your device and operating system.