[wptrunner] Automating Safari configuration

[jugglinmike]

We document some configuration that users are advised to apply when testing Safari. Those instructions have recently grown a bit, and we're currently working to add the new step to the automation scripts in the "results-collection" project. In that code review, @foolip suggested automating the process in the WPT CLI itself.

Conceptually, this is similar to how the WPT CLI modifies the behavior of Chrome and Firefox. However, a few technical differences in Safari may discourage similar functionality:

• The configuration is system-wide. Changes to the browser under test are also changes to the browser that contributors may be using for accessing the web generally. We could try to restore the original configuration at the end of each command-line invocation, but process crashes would leave the browser in a "configured for WPT" state.
• The configuration is loaded when the browser starts. If the user has a window open (perhaps reading the WPT documentation), then the configuration change will not take effect.

We might be able to address these concerns through command-line prompts (i.e. "Disable Safari's popup blocking? [Y/n]" and "Restart browser to apply changes? [Y/n]"). We're already prompting the user to authorize a WebDriver server at this stage, after all.

I don't know if/how this kind of behavior fits in to the larger design of the CLI (or if there are technical alternatives that obviate any of these problems). That's why I'm raising it here!

[gsnedders]

So one thing we can definitely do is check the defaults state and raise a warning if we don't have the ones we're looking for.

We can restore the original configuration unless wptrunner itself crashes, right? Which isn't ideal, but it should at least be relatively rare? It may still be worth putting behind a flag, though…

[jugglinmike]

So one thing we can definitely do is check the defaults state and raise a
warning if we don't have the ones we're looking for.

Alerting the user seems like a good idea, but "raise a warning" could be
interpreted in a couple ways. Do you mean write warning text to standard out,
or raise an exception which halts execution? Or something else?

We can restore the original configuration unless wptrunner itself crashes,
right? Which isn't ideal, but it should at least be relatively rare? It may
still be worth putting behind a flag, though…

I don't know how to quantify the frequency, but I'd be very sympathetic to the
person with the misconfigured browser no matter how rare. An opt-in would make
this less problematic. I have doubts about implementing a feature with rough
edges, though.

[foolip]

I like the prompts idea, and we'd pass --yes in CI.

Trying to restore the original settings will only work until the first time that someone aborts ./wpt run at a bad time, or if it crashes. We could try to put the restoring in the very outermost finally block possible, but I suspect it would still go wrong eventually.

[gsnedders]

Do you mean write warning text to standard out, or raise an exception which halts execution? Or something else?

I meant something non-fatal, but maybe it should be fatal by default.