[FEATURE] Enhance the CI Workflow to skip the Unmodified Architectures

[lupyuen]

Is your feature request related to a problem? Please describe.

• NuttX Devs need to wait (2 hours) for the CI Build to complete across all Architectures (Arm32, Arm64, RISC-V, Xtensa), even though they're modifying a Single Architecture
• We're using too many GitHub Runners and Build Minutes, exceeding the ASF Policy for GitHub Actions
• Our usage of GitHub Runners is going up ($12K per month), we need to stay within the ASF Budget for GitHub Runners ($8.2K per month)
• What if CI could build only the Modified Architecture?
• Right now most of our CI Builds are taking 2 hours 15 mins. Can we complete the build within 1 hour, when we Create / Modify a Simple PR?

Describe the solution you'd like

• We propose a Partial Solution, based on the Arch and Board Labels (recently added to CI)
• We target only the Simple PRs: One Arch Label + One Board Label + One Size Label, like "Arch: risc-v, Board: risc-v, Size: XS"
• If "Arch: arm" is the only non-size label, then we build only arm-01, arm-02, ...
• Same for "Board: arm"
• If Arch and Board Labels are both present: They must be the same
• Similar rules for RISC-V, Simulator, x86_64 and Xtensa
• Simple PR + Docs is still considered a Simple PR (so devs won't be penalised for adding docs)

Fetch the Arch Labels

This is how we fetch the Arch Labels, and identify as Arm, Arm64, RISC-V, Xtensa:

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L32-L104

      # Get the Arch for the PR: arm, arm64, risc-v, xtensa, ...
      - name: Get arch
        id: get-arch
        run: |        

          # If PR is Not Created or Modified: Build all targets
          pr=${{github.event.pull_request.number}}
          if [[ "$pr" == "" ]]; then
            echo "Not a Created or Modified PR, will build all targets"
            exit
          fi

          # Ignore the Label "Area: Documentation", because it won't affect the Build Targets
          query='.labels | map(select(.name != "Area: Documentation")) | '
          select_name='.[].name'
          select_length='length'

          # Get the Labels for the PR: "Arch: risc-v \n Board: risc-v \n Size: XS"
          # If GitHub CLI Fails: Build all targets
          labels=$(gh pr view $pr --repo $GITHUB_REPOSITORY --json labels --jq $query$select_name || echo "")
          numlabels=$(gh pr view $pr --repo $GITHUB_REPOSITORY --json labels --jq $query$select_length || echo "")
          echo "numlabels=$numlabels" | tee -a $GITHUB_OUTPUT

          # Identify the Size, Arch and Board Labels
          if [[ "$labels" == *"Size: "* ]]; then
            echo 'labels_contain_size=1' | tee -a $GITHUB_OUTPUT
          fi
          if [[ "$labels" == *"Arch: "* ]]; then
            echo 'labels_contain_arch=1' | tee -a $GITHUB_OUTPUT
          fi
          if [[ "$labels" == *"Board: "* ]]; then
            echo 'labels_contain_board=1' | tee -a $GITHUB_OUTPUT
          fi

          # Get the Arch Label
          if [[ "$labels" == *"Arch: arm64"* ]]; then
            echo 'arch_contains_arm64=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: arm"* ]]; then
            echo 'arch_contains_arm=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: risc-v"* ]]; then
            echo 'arch_contains_riscv=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: simulator"* ]]; then
            echo 'arch_contains_sim=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: x86_64"* ]]; then
            echo 'arch_contains_x86_64=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Arch: xtensa"* ]]; then
            echo 'arch_contains_xtensa=1' | tee -a $GITHUB_OUTPUT
          fi

          # Get the Board Label
          if [[ "$labels" == *"Board: arm64"* ]]; then
            echo 'board_contains_arm64=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: arm"* ]]; then
            echo 'board_contains_arm=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: risc-v"* ]]; then
            echo 'board_contains_riscv=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: simulator"* ]]; then
            echo 'board_contains_sim=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: x86_64"* ]]; then
            echo 'board_contains_x86_64=1' | tee -a $GITHUB_OUTPUT
          elif [[ "$labels" == *"Board: xtensa"* ]]; then
            echo 'board_contains_xtensa=1' | tee -a $GITHUB_OUTPUT
          fi

        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Why || echo ""? That's because if the GitHub CLI gh fails for any reason, we will build all targets. This ensures that our CI Workflow won't get disrupted due to errors in GitHub CLI.

Handle Only Simple PRs

We handle only Simple PRs: One Arch Label + One Board Label + One Size Label, like "Arch: risc-v, Board: risc-v, Size: XS". If it's not a Simple PR: We build everything.

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L127-L169

          # inputs.boards is a JSON Array: ["arm-01", "risc-v-01", "xtensa-01", ...]
          # We compact and remove the newlines
          boards=$( echo '${{ inputs.boards }}' | jq --compact-output ".")
          numboards=$( echo "$boards" | jq "length" )

          # We consider only Simple PRs with:
          # Arch + Size Labels Only
          # Board + Size Labels Only
          # Arch + Board + Size Labels Only
          if [[ "$labels_contain_size" != "1" ]]; then
            echo "Size Label Missing, will build all targets"
            quit=1
          elif [[ "$numlabels" == "2" && "$labels_contain_arch" == "1" ]]; then
            echo "Arch + Size Labels Only"
          elif [[ "$numlabels" == "2" && "$labels_contain_board" == "1" ]]; then
            echo "Board + Size Labels Only"
          elif [[ "$numlabels" == "3" && "$labels_contain_arch" == "1"  && "$labels_contain_board" == "1" ]]; then
            # Arch and Board must be the same
            if [[
              "$arch_contains_arm" != "$board_contains_arm" ||
              "$arch_contains_arm64" != "$board_contains_arm64" ||
              "$arch_contains_riscv" != "$board_contains_riscv" ||
              "$arch_contains_sim" != "$board_contains_sim" ||
              "$arch_contains_x86_64" != "$board_contains_x86_64" ||
              "$arch_contains_xtensa" != "$board_contains_xtensa"
            ]]; then
              echo "Arch and Board are not the same, will build all targets"
              quit=1
            else
              echo "Arch + Board + Size Labels Only"
            fi
          else
            echo "Not a Simple PR, will build all targets"
            quit=1
          fi

          # If Not a Simple PR: Build all targets
          if [[ "$quit" == "1" ]]; then
            echo "selected_builds=$boards" | tee -a $GITHUB_OUTPUT
            exit
          fi

For Arm Arch: Identify the Non-Arm Builds

Suppose the PR says "Arch: arm" or "Board: arm". We filter out the builds that should be skipped (RISC-V, Xtensa, etc):

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L169-L234

          # For every board
          for (( i=0; i<numboards; i++ ))
          do
            # Fetch the board
            board=$( echo "$boards" | jq ".[$i]" )
            skip_build=0
            
            # For "Arch / Board: arm": Build arm-01, arm-02, ...
            if [[ "$arch_contains_arm" == "1" || "$board_contains_arm" == "1" ]]; then
              if [[ "$board" != *"arm"* ]]; then
                skip_build=1
              fi
            # Omitted: Arm64, RISC-V, Simulator x86_64, Xtensa
            ...
            # For Other Arch: Allow the build
            else
              echo Build by default: $board
            fi
  
            # Add the board to the selected builds
            if [[ "$skip_build" == "0" ]]; then
              echo Add $board to selected_builds
              if [[ "$selected_builds" == "" ]]; then
                selected_builds=$board
              else
                selected_builds=$selected_builds,$board
              fi
            fi
          done

          # Return the selected builds as JSON Array
          # If Selected Builds is empty: Skip all builds
          echo "selected_builds=[$selected_builds]" | tee -a $GITHUB_OUTPUT
          if [[ "$selected_builds" == "" ]]; then
            echo "skip_all_builds=1" | tee -a $GITHUB_OUTPUT
          fi

Skip The Non-Arm Builds

Earlier we saw the code in arch.yml (Reusable Workflow) that identifies the builds to be skipped. The code above is called by build.yml (Build Workflow) which will actually skip the builds:

https://github.com/apache/nuttx/blob/master/.github/workflows/build.yml#L119-L148

  # Select the Linux Builds based on PR Arch Label
  Linux-Arch:
    uses: apache/nuttx/.github/workflows/arch.yml@master
    needs: Fetch-Source
    with:
      os: Linux
      boards: |
        [
          "arm-01", "other", "risc-v-01", "sim-01", "xtensa-01",
          "arm-02", "risc-v-02", "sim-02", "xtensa-02",
          "arm-03", "arm-04", "arm-05", "arm-06", "arm-07", "arm-08", "arm-09", "arm-10", "arm-11", "arm-12", "arm-13"
        ]

  # Run the selected Linux Builds
  Linux:
    needs: Linux-Arch
    if: ${{ needs.Linux-Arch.outputs.skip_all_builds != '1' }}
    runs-on: ubuntu-latest
    env:
      DOCKER_BUILDKIT: 1

    strategy:
      max-parallel: 12
      matrix:
        boards: ${{ fromJSON(needs.Linux-Arch.outputs.selected_builds) }}

    steps:
      ## Omitted: Run cibuild.sh on Linux

Why needs: Fetch-Source? That's because the PR Labeler runs concurrently in the background. When we add Fetch-Source as a Job Dependency, we give the PR Labeler sufficient time to run (1 min), before we read the PR Label in arch.yml.

Same for RISC-V, Simulator, x86_64 and Xtensa Builds

We do the same for RISC-V, Simulator, x86_64 and Xtensa:

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L105-L129

            # For "Arch / Board: arm64": Build other
            elif [[ "$arch_contains_arm64" == "1" || "$board_contains_arm64" == "1" ]]; then
              if [[ "$board" != *"other"* ]]; then
                skip_build=1
              fi

            # For "Arch / Board: risc-v": Build risc-v-01, risc-v-02
            elif [[ "$arch_contains_riscv" == "1" || "$board_contains_riscv" == "1" ]]; then
              if [[ "$board" != *"risc-v"* ]]; then
                skip_build=1
              fi
  
            # For "Arch / Board: simulator": Build sim-01, sim-02
            elif [[ "$arch_contains_sim" == "1" || "$board_contains_sim" == "1" ]]; then
              if [[ "$board" != *"sim"* ]]; then
                skip_build=1
              fi

            # For "Arch / Board: x86_64": Build other
            elif [[ "$arch_contains_x86_64" == "1" || "$board_contains_x86_64" == "1" ]]; then
              if [[ "$board" != *"other"* ]]; then
                skip_build=1
              fi
  
            # For "Arch / Board: xtensa": Build xtensa-01, xtensa-02
            elif [[ "$arch_contains_xtensa" == "1" || "$board_contains_xtensa" == "1" ]]; then
              if [[ "$board" != *"xtensa"* ]]; then
                skip_build=1
              fi

Skip the macOS and Windows Builds

For these Simple PRs (One Arch Label + One Size Label), we skip the macOS and Windows builds (macos, macos/sim-*, msys2) since these builds are costly:

(macos and macos/sim-* builds will take 2 hours to complete due to the queueing for macOS Runners)

https://github.com/apache/nuttx/blob/master/.github/workflows/build.yml#L194-L281

  # Select the macOS Builds based on PR Arch Label
  macOS-Arch:
    uses: apache/nuttx/.github/workflows/arch.yml@master
    needs: Fetch-Source
    with:
      os: Linux
      boards: |
        ["macos", "sim-01", "sim-02"]

  # Run the selected macOS Builds
  macOS:
    permissions:
      contents: none
    runs-on: macos-13
    needs: macOS-Arch
    if: ${{ needs.macOS-Arch.outputs.skip_all_builds != '1' }}
    strategy:
      max-parallel: 2
      matrix:
        boards: ${{ fromJSON(needs.macOS-Arch.outputs.selected_builds) }}
    steps:
      ## Omitted: Run cibuild.sh on macOS
      ...
  # Select the msys2 Builds based on PR Arch Label
  msys2-Arch:
    uses: apache/nuttx/.github/workflows/arch.yml@master
    needs: Fetch-Source
    with:
      os: Linux
      boards: |
        ["msys2"]

  # Run the selected msys2 Builds
  msys2:
    needs: msys2-Arch
    if: ${{ needs.msys2-Arch.outputs.skip_all_builds != '1' }}
    runs-on: windows-latest
    strategy:
      fail-fast: false
      max-parallel: 1
      matrix:
        boards: ${{ fromJSON(needs.msys2-Arch.outputs.selected_builds) }}

    defaults:
      run:
        shell: msys2 {0}
    steps:
      ## Omitted: Run cibuild.sh on msys2

skip_all_builds will be set to 1 for Simple PRs on macOS and msys2.

(Except for "Arch: Simulator", which will enable the macOS Builds for sim-01 and sim-02)

Ignore the Documentation

NuttX Devs shouldn't be penalised for adding docs! That's why we ignore the label "Area: Documentation", so that Simple PR + Docs is still a Simple PR (which will skip the unnecessary builds).

https://github.com/apache/nuttx/blob/master/.github/workflows/arch.yml#L44-L55

          # Ignore the Label "Area: Documentation", because it won't affect the Build Targets
          query='.labels | map(select(.name != "Area: Documentation")) | '
          select_name='.[].name'
          select_length='length'

          # Get the Labels for the PR: "Arch: risc-v \n Board: risc-v \n Size: XS"
          # If GitHub CLI Fails: Build all targets
          labels=$(gh pr view $pr --repo $GITHUB_REPOSITORY --json labels --jq $query$select_name || echo "")
          numlabels=$(gh pr view $pr --repo $GITHUB_REPOSITORY --json labels --jq $query$select_length || echo "")
          echo "numlabels=$numlabels" | tee -a $GITHUB_OUTPUT

Sync the CI Workflow from nuttx repo to nuttx-apps

Remember to sync build.yml and arch.yml from nuttx repo to nuttx-apps! apache/nuttx-apps#2676

build.yml refers to arch.yml (for the build rules). So when we sync build.yml from nuttx to nuttx-apps, we won't need to remove the references to arch.yml.

We could make nuttx-apps/build.yml point to the nuttx/arch.yml. But that would make the CI fragile: Changes to nuttx/arch.yml might cause nuttx-apps/build.yml to break.

Yep arch.yml is totally not needed in nuttx-apps. I have difficulty keeping nuttx/build.yml and nuttx-apps/build.yml in sync, that's why I simply copied over arch.yml as-is. In future we could extend arch.yml with Build Rules that are specific to nuttx-apps?

If we decide to remove nuttx-apps/arch.yml: This means that we need to rewrite the build.yml logic from this:

  # Select the Linux Builds based on PR Arch Label
  Linux-Arch:
    uses: apache/nuttx-apps/.github/workflows/arch.yml@master
    needs: Fetch-Source
    with:
      boards: |
        [
          "arm-01", "other", "risc-v-01", "sim-01", "xtensa-01", ...
        ]

  # Run the selected Linux Builds
  Linux:
    needs: Linux-Arch
    if: ${{ needs.Linux-Arch.outputs.skip_all_builds != '1' }}
    strategy:
      matrix:
        boards: ${{ fromJSON(needs.Linux-Arch.outputs.selected_builds) }}

Back to this:

  Linux:
    needs: Fetch-Source
    strategy:
      matrix:
        boards: [arm-01, arm-02, arm-03, arm-04, arm-05, arm-06, arm-07, arm-08, arm-09, arm-10, arm-11, arm-12, arm-13, other, risc-v-01, risc-v-02, sim-01, sim-02, xtensa-01, xtensa-02]

Testing

(Note: The timings here are obsolete)

When we test our updated CI Workflow, we see that the irrelevant builds are skipped in seconds. Click "Show All Jobs" to reveal the timings:

• CI Build for Arm32 PR

  (Completed in 2 hours, roughly 15 mins faster than before. Bottleneck is arm-05, which takes 2 hours, we should split into smaller jobs)

• CI Build for Arm64 PR

  (Completed in 51 mins yay!)

• CI Build for RISC-V PR

  (Completed in 1 hour 50 mins. Bottleneck is riscv-01 and riscv-02, at 1 hour 47 mins each, we should split into smaller jobs)

• CI Build for Xtensa PR

  (Completed in 1 hour 17 mins. Bottleneck is xtensa-01 and xtensa-02, at 1 hour 15 mins each)

• CI Build for Other PRs and Non-PR (All Targets)

  (Same as the present CI)

• When PRs are Merged: All Targets are recompiled (2 hours 13 mins), no changes from the Present CI. Merge Arm32 PR

• Previously the irrelevant builds were skipped in seconds. Now the irrelevant builds (e.g. arm-01) are totally omitted for Simple PRs (e.g. "Arch: risc-v")

  

Actual Performance

We recorded the CI Build Performance based on Real-World PRs:

• For Arm32: Simple PRs will build in 2 hours (previously also 2 hours)

• For Arm64: Simple PRs will build in 49 mins (previously 2 hours 11 mins)

• For RISC-V: Simple PRs will build in 1 hour 45 mins (previously also 1 hour 45 mins)

• For Xtensa: Simple PRs will build in 1 hour 17 mins (previously 2 hours 11 mins)

• For x86_64: Simple PRs will build in 48 mins (previously 2 hours 13 mins)

• For Simulator: Simple PRs will build in 1 hour 32 mins (previously 2 hours 12 mins)

• OK no big wow yet. We need to break arm-05, riscv-01 and riscv-02 into multiple smaller jobs. Then things will really zoom! (See the Build Job Details)

  Move the RP2040 jobs from arm-05 to arm-06, then add arm-14. Add jobs riscv-03 to riscv-06.

  (Update: All Done! Check the PRs below)

• We already see a 27% Reduction in GitHub Runner Hours! From 15 Runner Hours down to 11 Runner Hours per Arm32 Build.

• Split the Board Labels according to Arch, like "Board: arm". So "Board: arm" should build the exact same way as "Arch: arm". Same for "Board: arm, Arch: arm". Update the Build Rules to use the Board Labels

  (Update: All Done! Check the PRs below)

• Split the others job into arm64 and x86_64

  (Update: All Done! Check the PRs below)

TODO: Reorg and rename the CI Build Jobs, for better performance and easier maintenance. But how?

• I have a hunch that CI works better when we pack the jobs into One-Hour Time Slices
• Kinda like packing yummy goodies into Bento Boxes, making sure they don't overflow the Time Boxes :-)
• We should probably shift the Riskiest / Most Failure Prone builds into the First Build Job. So we can Fail Faster (in case of problems), and skip the rest of the jobs
• Recently we see many builds for Arm32 Goldfish. Can we limit the builds to the Goldfish Boards only? To identify Goldfish PRs, we can label the PRs like this: "Arch: arm, SubArch: goldfish" and/or "Board: arm, SubBoard: goldfish"
• How will we filter out the Build Jobs (e.g. arm-01) that should be built for a SubBoard (e.g. stm32)? Maybe like this

Hi @raiden00pl and @simbit18: Do you think this will work? And keep Our Devs and ASF happy? Thanks!

Describe alternatives you've considered

No response

Verification

• I have verified before submitting the report.

[simbit18]

Hi @lupyuen
Nice system to skip jobs!!!! :)

I think that a further margin of reduction of the generalized time of the jobs is possible with these simple solutions:

• use of an storage folder for packages repeated several times in the configurations present in the job (git and curl in the NuttX and NuttX-Apps repositories example in PR CI: Improvement to speed up compilation and reduce download errors. #13301 )
• addition of other configurations in the .dat files (testlist folder) with cmake + ninja to compile NuttX.

[lupyuen]

Thanks @simbit18! Your suggestions are excellent, I think we can implement them carefully after we have rolled out the solution above. I still have lots to learn about the Storage Folder and CMake :-)

[raiden00pl]

Great ! That's what I had in mind when creating new labels.
The only problem I see is that I don't quite understand how github chooses the order of workflows in a parallel build. This approach will only work if the labeler is called before the build. Is this always ensured ?

TODO: Should we have Board Labels by Arch also?

I think so. Another approach is to add "Platform" label which detect more precisely the arch used (like "Platform: STM32"), but then the problem of having too many labels may occur. Or we can add "Platform" labels for some of the most popular chips that are most frequently modified or they use CI the most.

[lupyuen]

The only problem I see is that I don't quite understand how github chooses the order of workflows in a parallel build. This approach will only work if the labeler is called before the build. Is this always ensured ?

Thanks @raiden00pl! Yep I wonder about the order of workflows. If we look at a Typical CI Run:

• First thing that runs is build.yml > Fetch-Source, which will take 1 minute
• Then labeler.yml > Labeler starts concurrently to set the Labels
• Up next is build.yml > Linux (arm-01). Which will start after Fetch-Source and read the Labels
• So we are OK (I think) as long as Labeler doesn't exceed 1 minute

We see some improvements: Arm32 builds now complete in 2 hours (instead of 2 hours 15 mins). And we save on runners for RISC-V, Xtensa, Sim.

What's Next:

• I'll extend build.yml to handle Arch: simulator (as requested by Xiao Xiang) and Arch: x86_64 (because it appears often)
• Skip macos, macos/sim-*, msys2 for Simple PRs (1 Arch Label + 1 Size Label)
• Which will require the Get Arch logic to be refactored into a Reusable Workflow
• Hopefully we can then complete the CI Build within 1 hour! (And make ASF happy)

Update: All Done! Check the PRs below