Taskcluster status checks not finishing and blocking PRs

[foolip]

In #14088 (comment) @qiuzhong reported that Taskcluster isn't finishing and that they have seen this often recently. I made Taskcluster required just after I merged #14096, so that's why this has been happening only recently.

In #13194 (comment) we wondered what would happen, and now we know.

[foolip]

@jugglinmike have you heard back from the Taskcluster folks about this problem? Do you have up-to-date stats about how many PRs are affected?

[foolip]

The status will be "Expected — Waiting for status to be reported" when this occurs.

[foolip]

After consulting with @jgraham on IRC I've made the check not required while this is being resolved, and will assign to James.

[jugglinmike]

@jugglinmike have you heard back from the Taskcluster folks about this problem?

No, but you can keep an eye on the bug I filed to get the latest.

Do you have up-to-date stats about how many PRs are affected?

Sure. To keep things organized, I've reported the new stats in the original issue. I've also posted the new failures on that Taskcluster ticket.

[foolip]

Thanks @jugglinmike!

[foolip]

@jgraham can you help us poke for progress on this?

[lukebjerring]

Ping @jgraham any updates on this?

[foolip]

https://bugzilla.mozilla.org/show_bug.cgi?id=1507254 has been resolved now. @jugglinmike can you get stats for the past week, to see if we can make stability blocking again?

[jugglinmike]

To give us a little more confidence, I'll verify all commits over the past 2 weeks (the bug was reported as "fixed" on 2018-11-28):

$ node . --between 2018-11-28:2018-12-11 --discrepancies --format markdown

Good news! In that period, Taskcluster has reported results for every commit that TravisCI has reported results.

All results

pull request URL	pull request author	commit	TravisCI: Chrome	Taskcluster: Chrome	TravisCI: Firefox	Taskcluster: Firefox
gh-14430	autofoolip	2a8c9f3	PASS	PASS	ERROR	FAIL
gh-14424	lgrahl	c115aba	ERROR	FAIL	ERROR	FAIL
gh-14424	lgrahl	a01a43f	ERROR	FAIL	ERROR	FAIL
gh-14422	chromium-wpt-export-bot	4b1e031	ERROR	PASS	ERROR	PASS
gh-14419	youennf	c2e1ce7	ERROR	PASS	ERROR	FAIL
gh-14418	youennf	d9d2911	ERROR	PASS	ERROR	PASS
gh-14417	youennf	c7baefa	ERROR	FAIL	ERROR	FAIL
gh-14417	youennf	b8e143a	ERROR	FAIL	ERROR	FAIL
gh-14415	andreastt	0c61d8d	ERROR	FAIL	PASS	PASS
gh-14410	chromium-wpt-export-bot	ef1a5c8	PASS	FAIL	PASS	PASS
gh-14409	qiuzhong	fa019ce	ERROR	FAIL	ERROR	FAIL
gh-14409	qiuzhong	78aa08d	ERROR	FAIL	ERROR	FAIL
gh-14409	qiuzhong	78c0a07	ERROR	FAIL	ERROR	FAIL
gh-14405	chromium-wpt-export-bot	e630525	ERROR	PASS	ERROR	FAIL
gh-14400	koto	b8b57d1	ERROR	FAIL	ERROR	PASS
gh-14395	Ms2ger	4e388a4	ERROR	FAIL	ERROR	FAIL
gh-14386	chromium-wpt-export-bot	bafb027	PASS	PASS	ERROR	FAIL
gh-14385	chromium-wpt-export-bot	168fa7c	ERROR	PASS	ERROR	PASS
gh-14370	autofoolip	71880c7	PASS	PASS	ERROR	FAIL
gh-14367	ewilligers	a18a0bb	ERROR	PASS	PASS	PASS
gh-14364	jugglinmike	b17d27f	ERROR	FAIL	ERROR	PASS
gh-14363	chromium-wpt-export-bot	4365720	PASS	PASS	PASS	FAIL
gh-14362	chromium-wpt-export-bot	69114e4	PASS	FAIL	PASS	PASS
gh-14345	SummerLW	f0b5ee0		PASS		PASS
gh-14339	chromium-wpt-export-bot	3950f9f	PASS	PASS	ERROR	PASS
gh-14334	MurakamiShinyu	781fd3d	ERROR	PASS	ERROR	PASS
gh-14319	chromium-wpt-export-bot	50fd7a0	PASS	FAIL	PASS	FAIL
gh-14317	youennf	c6db943	PASS	FAIL	PASS	PASS
gh-14314	darsh7807	c95f792		PASS		PASS
gh-14300	foolip	f90c12d	PASS	FAIL	PASS	PASS
gh-14300	foolip	6bf4be2	PASS	FAIL	PASS	PASS
gh-14300	foolip	8f0c1c8	PASS	FAIL	PASS	FAIL
gh-14300	foolip	52e8198	PASS	PASS	PASS	FAIL
gh-14300	foolip	1fc51b4	PASS	FAIL	PASS	FAIL
gh-14300	foolip	18d6b1e	PASS	FAIL	PASS	FAIL
gh-14297	marcoscaceres	94fd139	ERROR	FAIL	ERROR	FAIL
gh-14293	chromium-wpt-export-bot	cc2a9d4	PASS	FAIL	PASS	PASS
gh-14288	jyasskin	48bb042	PASS	PASS	ERROR	PASS
gh-14285	chromium-wpt-export-bot	be36d60	ERROR	PASS	ERROR	PASS
gh-14282	chromium-wpt-export-bot	cd6694f	PASS	PASS	PASS	FAIL
gh-14278	annevk	072ffc9	PASS	FAIL	PASS	PASS
gh-14276	autofoolip	bb8785f	PASS	FAIL	PASS	PASS
gh-14273	qiuzhong	9aaf8ca	PASS	FAIL	ERROR	FAIL

Summary:
- Total pull requests:         162
- Total commits:               244
- Tested commits:              218
- Contested commits (Chrome):  10
- Contested commits (Firefox): 8
- Contested commits (both):    25

(For anyone interested in reproducing this data, the script I'm using is available here.)

[foolip]

Excellent! I've made Taskcluster a required check for master again and so closing this issue.