Add Github actions support for library CI testing (#2577)

This PR adds a re-usable `library` workflow for Github actions so that Sming libraries can be independently tested. It also tidies up the main ``ci.yml`` file and fixes a couple of minor related issues. The ``library.yml`` re-useable workflow is provided, which takes care of these tasks: - Checking in the library to test - Checking in the Sming framework - Installing build tools - Builds all applications within the library's ``samples`` directory, for all supported architectures - If a test application is provided then that should be located in a ``test`` directory. This is built for all architectures, and also executed for Host. Builds are handled using :source:`Tools/ci/library/Makefile`. See also https://docs.github.com/en/actions/using-workflows/reusing-workflows. To use this in a project, add a suitable workflow to the ``.github/workflows`` directory. Templates are provided in the ``.github/workflows/library`` directory. Here is the basic ``push`` scenario: ``` name: CI Push on: [push] jobs: build: uses: SmingHub/Sming/.github/workflows/library.yml@develop # Inputs are all optional, defaults are shown with: # Repository to fetch Sming from sming_repo: 'https://github.com/SmingHub/Sming' # Sming branch to run against sming_branch: 'develop' # Library alias alias: '' ``` The ``sming_repo`` and ``sming_branch`` inputs are provided if your library requires modifications to Sming which are not (yet) in the main repository. The ``alias`` input is required where the library repository name does not correspond with the working title. For example, the ``jerryscript`` library is in a repository called ``Sming-jerryscript``, so must be checked out using a different name. If Sming contains a library (or Component) with the same name then it will be overridden, with a warning ``Multiple matches found for Component 'jerryscript'`` in the build log. The ``ci-dispatch.yml`` example demonstrates manual triggering, which allows these inputs to be easily changed. See https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow. Note that the workflow must be available in the library's default branch, or it will not appear in the github web page.
SmingHub · Oct 31, 2022 · ebd8feb · ebd8feb
1 parent 0c01a88
commit ebd8feb
Show file tree

Hide file tree

Showing 8 changed files with 212 additions and 44 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -11,22 +11,21 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        arch: [Esp8266, Host, Esp32, Rp2040]
-        variant: [""]
         os: [ubuntu-latest, windows-latest]
+        variant: [esp8266, host, esp32, esp32s2, esp32c3, rp2040]
         include:
-        - arch: Esp32
-          variant: esp32s2
-          os: ubuntu-latest
-        - arch: Esp32
-          variant: esp32s2
-          os: windows-latest
-        - arch: Esp32
-          variant: esp32c3
-          os: ubuntu-latest
-        - arch: Esp32
-          variant: esp32c3
-          os: windows-latest
+          - variant: esp8266
+            arch: Esp8266
+          - variant: host
+            arch: Host
+          - variant: esp32
+            arch: Esp32
+          - variant: esp32s2
+            arch: Esp32
+          - variant: esp32c3
+            arch: Esp32
+          - variant: rp2040
+            arch: Rp2040
 
     concurrency: 
       group: ${{ github.workflow }}-${{ github.ref }}-${{ toJson(matrix) }}
@@ -42,52 +41,36 @@ jobs:
     - name: Checkout code
       uses: actions/checkout@v3
 
-    - name: Setup SMING_HOME for Ubuntu
-      if: ${{ matrix.os ==  'ubuntu-latest' }}
-      run: |
-        echo "CI_BUILD_DIR=$GITHUB_WORKSPACE" >> $GITHUB_ENV
-        echo "SMING_HOME=$GITHUB_WORKSPACE/Sming" >> $GITHUB_ENV
-
-    - name: Setup SMING_HOME for Windows
-      if: ${{ matrix.os ==  'windows-latest' }}
+    - name: Configure environment
+      shell: pwsh
       run: |
-        echo ("CI_BUILD_DIR=" + $env:GITHUB_WORKSPACE) >> $env:GITHUB_ENV
-        $env:SMING_HOME = Join-Path $env:GITHUB_WORKSPACE "Sming"
-        echo ("SMING_HOME=" + $env:SMING_HOME) >> $env:GITHUB_ENV
+        "CI_BUILD_DIR=" + (Resolve-Path ".").path >> $env:GITHUB_ENV
+        "SMING_HOME=" + (Resolve-Path "Sming").path >> $env:GITHUB_ENV
+        "SMING_ARCH=${{ matrix.arch }}" >> $env:GITHUB_ENV
+        "SMING_SOC=${{ matrix.variant }}" >> $env:GITHUB_ENV
 
-    - name: Install Sming Framework on Ubuntu
+    - name: Install build tools for Ubuntu
       if: ${{ matrix.os ==  'ubuntu-latest' }}
-      env:
-        SMING_ARCH: ${{matrix.arch}}
-        SMING_SOC: ${{matrix.variant}}
       run: |
         Tools/ci/install.sh
 
-    - name: Install Sming Framework on Windows   
+    - name: Install build tools for Windows   
       if: ${{ matrix.os ==  'windows-latest' }}
-      env: 
-        SMING_ARCH: ${{matrix.arch}}
-        SMING_SOC: ${{matrix.variant}}
       run: |
-         Tools/ci/setenv.ps1
+         . Tools/ci/setenv.ps1
          Tools/ci/install.cmd
 
-    - name: Build and Test for ${{matrix.arch}} on Ubuntu
+    - name: Build and test for ${{matrix.arch}} on Ubuntu
       env: 
-        SMING_ARCH: ${{matrix.arch}}
-        SMING_SOC: ${{matrix.variant}}
         CLANG_FORMAT: clang-format-8
       if: ${{ matrix.os == 'ubuntu-latest' }}
       run: |
         source $SMING_HOME/../Tools/export.sh
         $CLANG_FORMAT --version
         Tools/ci/build.sh
 
-    - name: Build and Test for ${{matrix.arch}} on Windows
-      env: 
-        SMING_ARCH: ${{matrix.arch}}
-        SMING_SOC: ${{matrix.variant}}
+    - name: Build and test for ${{matrix.arch}} on Windows
       if: ${{ matrix.os == 'windows-latest' }}
       run: |
-        Tools/ci/setenv.ps1
+        . Tools/ci/setenv.ps1
         Tools/ci/build.cmd
diff --git a/.github/workflows/library.yml b/.github/workflows/library.yml
@@ -0,0 +1,97 @@
+name: Continuous Integration (CI)
+
+on:
+  workflow_call:
+    inputs:
+      sming_repo:
+        description: 'Full URL for Sming repository'
+        default: 'https://github.com/SmingHub/Sming'
+        type: string
+      sming_branch:
+        description: 'Sming branch to run against'
+        default: 'develop'
+        type: string
+      alias:
+        description: 'Library alias'
+        default: ''
+        type: string
+
+jobs:
+  build:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        variant: [esp8266, host, esp32, esp32s2, esp32c3, rp2040]
+        include:
+          - variant: esp8266
+            arch: Esp8266
+          - variant: host
+            arch: Host
+          - variant: esp32
+            arch: Esp32
+          - variant: esp32s2
+            arch: Esp32
+          - variant: esp32c3
+            arch: Esp32
+          - variant: rp2040
+            arch: Rp2040
+
+    concurrency: 
+      group: ${{ github.workflow }}-${{ github.ref }}-${{ toJson(matrix) }}
+      cancel-in-progress: true
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Fix autocrlf setting
+      run: |
+        git config --global --add core.autocrlf input
+
+    - name: Checkout code
+      uses: actions/checkout@v3
+
+    - name: Create library alias
+      if: ${{ inputs.alias }}
+      shell: pwsh
+      run: |
+        New-Item -ItemType SymbolicLink -Path "../${{ inputs.alias }}" -Target (Resolve-Path ".").path
+
+    - name: Checkout sming
+      run: |
+        git clone ${{ inputs.sming_repo }} -b ${{ inputs.sming_branch }} --depth 1 ../../sming
+
+    - name: Configure environment
+      shell: pwsh
+      run: |
+        "SMING_HOME=" + (Resolve-Path "../../sming/Sming").path >> $env:GITHUB_ENV
+        "COMPONENT_SEARCH_DIRS=" + (Resolve-Path "..").path >> $env:GITHUB_ENV
+        "CI_MAKEFILE=" + (Resolve-Path "../../sming/Tools/ci/library/Makefile") >> $env:GITHUB_ENV
+        "SMING_ARCH=${{ matrix.arch }}" >> $env:GITHUB_ENV
+        "SMING_SOC=${{ matrix.variant }}" >> $env:GITHUB_ENV
+
+    - name: Install build tools for Ubuntu
+      if: ${{ matrix.os ==  'ubuntu-latest' }}
+      run: |
+        . $SMING_HOME/../Tools/export.sh
+        $SMING_HOME/../Tools/ci/install.sh $SMING_ARCH
+
+    - name: Install build tools for Windows   
+      if: ${{ matrix.os ==  'windows-latest' }}
+      run: |
+        . "$env:SMING_HOME/../Tools/ci/setenv.ps1"
+        . "$env:SMING_HOME/../Tools/ci/install.cmd"
+
+    - name: Build and Test for ${{matrix.arch}} on Ubuntu
+      env:
+        CLANG_FORMAT: clang-format-8
+      if: ${{ matrix.os == 'ubuntu-latest' }}
+      run: |
+        . $SMING_HOME/../Tools/export.sh
+        make -j$(nproc) -f $CI_MAKEFILE
+
+    - name: Build and Test for ${{matrix.arch}} on Windows
+      if: ${{ matrix.os == 'windows-latest' }}
+      run: |
+        . "$env:SMING_HOME/../Tools/ci/setenv.ps1"
+        make -j $env:NUMBER_OF_PROCESSORS -f $env:CI_MAKEFILE
diff --git a/.github/workflows/library/ci-dispatch.yml b/.github/workflows/library/ci-dispatch.yml
@@ -0,0 +1,18 @@
+name: CI Dispatch
+
+on:
+  workflow_dispatch:
+    inputs:
+      sming_repo:
+        description: 'Full URL for Sming repository'
+        default: 'https://github.com/SmingHub/Sming'
+      sming_branch:
+        description: 'Sming branch to run against'
+        default: 'develop'
+
+jobs:
+  build:
+    uses: SmingHub/Sming/.github/workflows/library.yml@develop
+    with:
+      sming_repo: ${{ inputs.sming_repo }}
+      sming_branch: ${{ inputs.sming_branch }}
diff --git a/.github/workflows/library/ci-push.yml b/.github/workflows/library/ci-push.yml
@@ -0,0 +1,7 @@
+name: CI Push
+
+on: [push]
+
+jobs:
+  build:
+    uses: SmingHub/Sming/.github/workflows/library.yml@develop
diff --git a/Sming/Components/Storage/Tools/hwconfig/common.py b/Sming/Components/Storage/Tools/hwconfig/common.py
@@ -24,7 +24,7 @@ def critical(msg):
 
 def fixpath(path):
     """Paths in Windows can get a little weird """
-    if len(path) > 2 and path[1] != ':' and platform.system() == 'Windows' and path[2] == '/':
+    if path[0] == '/' and platform.system() == 'Windows':
         return path[1] + ':' + path[2:]
     return path
 

diff --git a/Tools/ci/install.sh b/Tools/ci/install.sh
@@ -24,6 +24,6 @@ else
     INSTALL_OPTS="fonts"
 fi
 
-"$SMING_HOME/../Tools/install.sh" ${SMING_ARCH,,} $INSTALL_OPTS
+"$SMING_HOME/../Tools/install.sh" $SMING_ARCH $INSTALL_OPTS
 
 fi
diff --git a/Tools/install.sh b/Tools/install.sh
@@ -17,6 +17,7 @@ err=0
 FONT_PACKAGES="fonts-ubuntu fonts-noto-mono xfonts-base fonts-urw-base35 fonts-droid-fallback"
 
 for opt in "$@"; do
+    opt=$(echo "$opt" | tr '[:upper:]' '[:lower:]')
     case $opt in
         all)
             inst_host=1

diff --git a/docs/source/information/develop/ci.rst b/docs/source/information/develop/ci.rst
@@ -44,6 +44,68 @@ where practical powershell core is used as this runs on either.
 Library CI support
 ------------------
 
+Sming libraries may be separately built and tested whether or not they are included as part of
+the Sming repository (or a fork).
+
+There are two mechanisms avaiable.
+
+GitHub Actions
+~~~~~~~~~~~~~~
+
+The ``library.yml`` re-useable workflow is provided, which takes care of these tasks:
+
+- Checking in the library to test
+- Checking in the Sming framework
+- Installing build tools
+- Builds all applications within the library's ``samples`` directory, for all supported architectures
+- If a test application is provided then that should be located in a ``test`` directory.
+  This is built for all architectures, and also executed for Host.
+
+Builds are handled using :source:`Tools/ci/library/Makefile`.
+
+See also https://docs.github.com/en/actions/using-workflows/reusing-workflows.
+
+To use this in a project, add a suitable workflow to the ``.github/workflows`` directory.
+Templates are provided in the ``.github/workflows/library`` directory.
+
+Here is the basic ``push`` scenario:
+
+.. code-block:: yaml
+
+   name: CI Push
+   on: [push]
+   jobs:
+     build:
+       uses: SmingHub/Sming/.github/workflows/library.yml@develop
+       # Inputs are all optional, defaults are shown
+       with:
+         # Repository to fetch Sming from
+         sming_repo: 'https://github.com/SmingHub/Sming'
+         # Sming branch to run against
+         sming_branch: 'develop'
+         # Library alias
+         alias: ''
+
+The ``sming_repo`` and ``sming_branch`` inputs are provided if your library requires modifications
+to Sming which are not (yet) in the main repository.
+
+The ``alias`` input is required where the library repository name does not correspond with
+the working title.
+For example, the ``jerryscript`` library is in a repository called ``Sming-jerryscript``,
+so must be checked out using a different name.
+If Sming contains a library (or Component) with the same name then it will be overridden,
+with a warning ``Multiple matches found for Component 'jerryscript'`` in the build log.
+
+The ``ci-dispatch.yml`` example demonstrates manual triggering, which allows these inputs to be easily changed.
+See https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow.
+
+Note that the workflow must be available in the library's default branch, or it will
+not appear in the github web page.
+
+
+Appveyor
+~~~~~~~~
+
 Appveyor may be configured to test a Sming library separately. Steps to enable:
 
 Add project to appveyor account