Test Safari Technology Preview

[cdumez]

I believe the dashboard is testing Safari 10.0. A lot of WPT improvements went into Safari 10.1 and it is very unfortunate those are not reflected on the dashboard.

[gsnedders]

Currently, Safari and Edge are tested on Sauce, and are limited by the versions available there (see also #110). I think there was some talk about trying to use BrowserStack instead, but I don't know what came of that.

[jgraham]

https://wiki.saucelabs.com/display/DOCS/Platform+Configurator#/ suggests that Sauce doesn't support 10.1.

Ideally we would have access to dedicated mac hardware for this kind of thing, since Mac based cloud providers are rather niche.

[foolip]

Ideally, we should be testing using Safari Technology Preview, but we'd need dedicated hardware for it, since neither Sauce nor BrowserStack have it. (BrowserStack has a "Tech Preview", but it's very old.)

[RByers]

@cdumez, WWYT about Apple running the tests on hardware you own (using Safari TP and a process anyone else could easily replicate) and submitting the results to the dashboard? We're looking into a setup like that for Edge too.

[foolip]

Now Safari 11 is released, so I added that to the title.

[foolip]

This is blocked on one of the following happening:

• Sauce provides newer Safari
• BrowserStack does it
• We (Google) set up and maintain our our Mac hardware
• You (Apple) sets up hardware, we figure out how to post results back

[foolip]

Repeating some of what I said on #109:

We've done some planning for Q4 now, and unfortunately this issue does not seem tractable right now. It's possible that BrowserStack or Sauce will get more recent versions of Safari, but ETA is unknown. We'll revisit this for Q1.

@youennf @cdumez FYI, please shout if this seems wrong and there is a way we can do it faster.

[foolip]

BrowserStack now has Safari 11 on macOS High Sierra.

[foolip]

@mattl, could you look into what amount of effort would be needed to use BrowserStack instead of Sauce?

[gsnedders]

https://wiki.saucelabs.com/display/DOCS/2017/10/20/Announcing+support+for+Safari+11+in+Mac+OS+10.12 says they now have Safari 11.

[gsnedders]

(Note that after #186 lands we probably want some issue to track running Safari TP/WebKit Build Archive builds.)

[gsnedders]

Changed title per above, and reopening.

[foolip]

web-platform-tests/wpt#8979 is pretty promising. If any kind people at Apple want to take a look, that might help towards this issue as well in the end.

[foolip]

There is one run of Safari TP that's been uploaded so far:
https://wpt.fyi/results/?sha=eea0b54014&label=experimental

https://wpt.fyi/results/?product=safari[experimental] also shows this, but will show the latest one if another is uploaded.

@jugglinmike, have runs been failing since then, or might a new run suddenly appear?

[jugglinmike]

Both! First, it was due to the regression in WPT. Our one worker has been backed up with impossible-to-complete builds, and I only just canceled them last night. Now, it appears as though it's wandered offline. That's a new problem for this worker, so I'll have to dig a bit before I can say what's gone wrong, exactly.

[jugglinmike]

We implemented this on a branch and proved it on a staging server. We're still using a dedicated Buildbot "master" on a separate Amazon EC2 instance, but the codebases have been merged, and the results are being uploaded to the results receiver.

Here are the relevant commits in master:

02ea8fc...929a0e2

I'm no macOS administrator, so I'm reluctant to call this "stable." That said, we've been successfully uploading results since August 19, so I think it's fair to close this issue, now. I'm happy to triage any new issues as they arrive.