API errors not failing GH validation

[iloveeclipse]

          > But why they are both "green" ? Shouldn't they have red crosses?

Because platform.ui has chosen to make them green:

eclipse.platform.ui/Jenkinsfile

Line 33 in c1c681a

recordIssues(publishAllIssues:false, ignoreQualityGate:true,

Is there a new version available? If yes, could you please update aggregator to use it?

There was some issues with deadlocks but no one cared enough to analyses the root cause so an older API-Tools version was used in the meanwhile. I have now created a PR to configure the changed behavior now but it is not yet released:

• Add swt verification build eclipse-tycho/tycho#4295

Originally posted by @laeubi in #2476 (comment)

[iloveeclipse]

@laeubi : I'm not sure which part of the mentioned line is responsible for showing "green" vs "red" Github checks shown on GH UI.
Is this publishAllIssues:false or ignoreQualityGate:true or both or something else?

[laeubi]

Is this publishAllIssues:false or ignoreQualityGate:true or both or something else?

• publishAllIssues = should all (including existing) be shown in Github as code annotations
• ignoreQualityGate = If true, then the result of the quality gate is ignored when selecting a reference build. This option is disabled by default, so a failing quality gate will be passed from build to build until the original reason for the failure has been resolved.

[iloveeclipse]

@laeubi : thanks, but sorry, I still don't understand which setting causes the failed API check to be shown as green instead as red? Could you please simply fix whatever you think is broken and push a PR that would "enable" github validation to fail on failed API check?

[laeubi]

I don't know what is the desired outcome, when I started with this I let the check fail / become "red" whenever something new was detected, then several iterations / changes where performed e.g. now we use "DELTA" what mean if you solve one problem you are allowed to add one new without the checks to become red.

As long as warnings / reported problems are only seen as something that gets into peoples way to merge there PR as soon as possible instead of something we need to fix/improve I honestly don't see any "right" settings.

This is for example how PDE uses the checks what seem to work (for the PDE project) since a while:
https://github.com/eclipse-pde/eclipse.pde/blob/f3ba0239012344e49638f28431ff4340c1eeeb47/Jenkinsfile#L42

[iloveeclipse]

I don't know what is the desired outcome

API errors are not warnings, they should always fail the build!

[laeubi]

API errors are not warnings, they should always fail the build!

As said this is likely due to a bug that the build do not fails immediately. But still these are (additionally) reported as build warnings / errors from the maven parser.

[jukzi]

API errors are not warnings, they should always fail the build!

API errors on master should not fail the jenkins build (otherwise it would not be found as baseline) but mark it unstable. They should fail the GitHub validation of PRs tough.

[jukzi]

i.e. it can be build, but should not be submitted

[sratz]

i.e. it can be build, but should not be submitted

Agreed, there should only be one red X to look at, not two (the actual build and the dedicated validation) with the same issues.

[iloveeclipse]

Guys, there should be at least one red X, currently everything is green, and that is the main problem.

Whoever understands what is the root cause (tycho bug or wrong jenkins file), please provide fix for that.

Note, if this is Jenkins file to blame, it seem to be very similar on all other platform repos except PDE, so it should be fixed overall.

[jukzi]

@iloveeclipse maybe you can start by demonstrating the error? My two attempts failed.

[iloveeclipse]

@iloveeclipse maybe you can start by demonstrating the error? My two attempts failed.

See #2476 (comment) and all conversation above.

[jukzi]

See #2476 (comment) and all conversation above.

I explained that one in #2476 (comment) ... not an Issue just a bad coincidence which is hard to get rid of with the tools we have.