Searching a way to allow specific matrix entries to fail

[weiznich]

Environment

Repo: diesel.rs/diesel

Issue Description

We have setup azure pipelines as automatic CI service for diesel.rs. That works mostly fine.
Now it is specific to the rust ecosystem that there are several compiler version, notably a stable version, a beta version and a nightly version. The later is updated every night a may not work. To detect such broken version projects are encouraged to run their CI also with the nightly version of the rust compiler, but setup the CI in such a way that failing builds with a nightly compiler are ignored. Basically I've found no way to do this with azure pipelines.

[moswald]

This may be possible with our yaml builds. @ericsciple do you know if this can be done?

[ericsciple]

@vtbassmatt fyi

@weiznich Unfortunately it is not allowed today. The current solution would be to model it as a separate job.

[vtbassmatt]

@ericsciple how do you prevent a task failure from bubbling up to the job (or job failure from bubbling up to the pipeline)? I could update the docs with a pattern for implementing allow-failure, but I don't actually know how to do it 😀

[ericsciple]

@vtbassmatt continueOnError is a task-level control and job-level control. However, when set on the job I believe it applies to all configurations.

My initial thoughts are: that behavior makes sense, especially as we think about deployment phases coming soon. If we need additional control, we might want an additional option somewhere else.

This is a similar problem to condition. The property actually exists at the phase-level, not on each job configuration.

A workaround would be to use templates instead of matrix.

Rather than adding additional controls, it makes me wonder whether the correct long term solution here is something like templates without actually requiring you to declare a separate file.

[chriskrycho]

This is going to end up being a serious pain point for effective adoption in both the Rust and Ember.js communities, where it's common to run against the current stable release, the current beta release, and often also the current unstable/nightly/master channel. (We're doing it against beta on ember-cli-typescript.) In that case, it's perfectly acceptable for the beta channel to fail, and it shouldn't report that the whole build failed as a result. Unfortunately, because it's at a job or task level, it's not possible to handle with matrixes as far as I can see (and as your comment suggests).

[vtbassmatt]

Thanks @chriskrycho - good to hear reinforcement of how important this is. If you want to point me at an example pipeline, I might be able to suggest a workaround. (And to be clear, it'll be a workaround, not the solution we want to deliver.)

[chriskrycho]

@vtbassmatt thanks! Here is our config for the ember-cli-typescript project:

• top-level config lines of interest
• template

[vtbassmatt]

Gotcha. Here's how I'd work around the limitation. In the template, add a parameter allowedToFail defaulted to false. In the template, update the lines that run tests:

  - ${{ if not(eq(parameters.allowedToFail, 'true')) }}:
    - script: |
        ${{ parameters.emberTestCommand }}
      displayName: ${{ parameters.emberTestDisplayName }}
  - ${{ if eq(parameters.allowedToFail, 'true') }}:
    - script: |
        ${{ parameters.emberTestCommand }}
        exit 0
      displayName: ${{ parameters.emberTestDisplayName }}

Then in your top-level pipeline, you'll have to have separate matrixes for the allowed-to-fail legs:

  - job: ember_cli_versions
    displayName: 'ember-cli'
    dependsOn: linux_fixed
    pool:
      vmImage: 'ubuntu-16.04'
    strategy:
      matrix:
        release:
          eCliVersion: latest
    steps:
      - template: .azure/ci-template.yml
        parameters:
          emberCliVersion: $(eCliVersion)
  - job: ember_cli_versions_allowed_failures
    displayName: 'ember-cli [allowed failures]'
    dependsOn: linux_fixed
    pool:
      vmImage: 'ubuntu-16.04'
    strategy:
      matrix:
        beta:
          eCliVersion: beta
    steps:
      - template: .azure/ci-template.yml
        parameters:
          emberCliVersion: $(eCliVersion)
          allowedToFail: 'true'

[mike-north]

    - script: |
        ${{ parameters.emberTestCommand }}
        exit 0
      displayName: ${{ parameters.emberTestDisplayName }}

With this use of exit 0, it seems like a reviewer would have to actually dig into the test logs of each passing pull request, in order to ascertain whether a given code change introduced a build failure on these allowed-to-fail jobs.

It's still very important to surface feedback about these failures in a first class way.

An example implementation might involve using GitHub's CheckRun feature to report failures with a "neutral" or "warning" state in the pull request "checks" page, while still marking the Status as green.

Example of Neutral Status

Currently, Azure Pipelines does not make full use of this feature

Azure Pipeline Warnings

By adding this capability, developers could do something like

script: <test-command> || echo -e "\043#vso[task.logissue type=warning;] Yikes!!"

to effectively turn failures into warnings, while having them surface in the GitHub UI just like they do in the Pipelines UI

[vtbassmatt]

@mike-north fair point, I didn't think about that. You can actually inject an error and still not fail the leg, which might be what's needed.

[moswald]

@vtbassmatt I assigned you to this issue while the correct functionality is being ironed out.

[pllim]

I am in the Python world and looking into transitioning a similar functionality from Travis CI. I think this issue covers it, but if not, please let me know whether to open a new one or look at a different one. Thanks!

xref spacetelescope/synphot_refactor#194 and astropy/astropy#8445

[letmaik]

@vtbassmatt Any updates? We would also need that for some projects.

[vtbassmatt]

Nothing to report, sorry.

[Zac-HD]

@vtbassmatt - any updates to share?

I'm adding Python 3.9 to our build in HypothesisWorks/hypothesis#2445, and I'd love to have an allowed-to-fail matrix entry for 3.10 / nightly too.

[vtbassmatt]

@Zac-HD sorry, still nothing. It hasn't been forgotten, but we've had a number of other priorities. Does one of the above workarounds help at all?

[Zac-HD]

No worries, I'll look into setting up a separate job for it, or might just get around to moving to GitHub Actions as I have for smaller projects 😄

[pllim]

@Zac-HD , if you have "allowed to fail" working for Actions, I would like the recipe. Thank you!

[Zac-HD]

@pllim I'll be experimenting with https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idcontinue-on-error and will let you know how it goes.

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days