rule out w3c-test.org

[plehegar]

We shouldn't have links in spec headers linking to w3c-test.org this is un unstable server and shouldn't cannot be relied on.
@sideshowbarker , feel to pitch if you think otherwise.

[vivienlacourba]

a comment I made to @plehegar

"Test suite" links are using either w3c-test.org or wpt.fyi (trying
earlier I got some timeouts on the first one).
See 2 CR published on 22 May:
∘ https://www.w3.org/TR/2018/CR-pointerevents2-20180522/
∘ https://www.w3.org/TR/2018/CR-payment-request-20180522/

[sideshowbarker]

We shouldn't have links in spec headers linking to w3c-test.org this is un unstable server and shouldn't cannot be relied on.
@sideshowbarker , feel to pitch if you think otherwise.

100% agreed

[tabatkins]

It's useful to have some server I can link to to see the live test running, so people don't have to figure out how to stand up their own wpt instance just to look at tests. If w3c-test.org isn't good for that, we should get something set up for that somewhere else.

[sideshowbarker]

It's useful to have some server I can link to to see the live test running, so people don't have to figure out how to stand up their own wpt instance just to look at tests. If w3c-test.org isn't good for that, we should get something set up for that somewhere else.

Anybody can set up an alternative instance — it’s pretty straightforward: git clone the wpt repo onto some server for domain my-wptserve.something (with TLS set up through, e.g., letsencrypt), edit the wptserve config file (e.g., to set the ports to port 80 and 443), and start up wptserve the same way you normally would in your local working directory. Then you can navigate to http://my-wptserve.something and https://my-wptserve.something and have access to all the tests.

It’s not that w3c-test.org isn't good for looking at tests — it is and of course that’s what it’s intended for. But it’s not officially maintained by the W3C systeams team and anyway there are quirks (bugs, really) in wptserve that cause it to wedge itself up pretty regularly when it’s run under load for hours or days at a time as a persistent process. That makes it hard for us to guarantee the availability of w3c-test.org as a stable service that people can rely on. Publishing specs that link to it — at least without some kind of very prominent warning to let people know it’s unstable — risks giving people the impression that it’s more stable and reliable than it is.

[tabatkins]

Anybody can set up an alternative instance — it’s pretty straightforward:

This is "straightforward", in that there's a published list of steps, but it's also extremely non-trivial and far too much work to expect the average person to do who just wants to review some tests. So having a stood-up instance of wptserve to link to is still of very high value, I think.

I'm not going to plaster a disclaimer about the live test link onto every single test line in the new Bikeshed feature (it would be ridiculous), and putting said disclaimer anywhere else in the spec would just mean it won't get read (so it would be useless), so I don't have a good solution for advertising the stability of the live test viewer.

Thus my statement that if w3c-test.org isn't reliable enough to link to, we should make sure we have something that is. (And I guess fix the bugs in wptserve that wedge it.)

[sideshowbarker]

Thus my statement that if w3c-test.org isn't reliable enough to link to, we should make sure we have something that is. (And I guess fix the bugs in wptserve that wedge it.)

So that’s the thing: Anybody setting up an instance of wptserve that runs persistently under load in the way that the w3c-test.org instance does is going to run into the same problems that has.

So far nobody’s stepped forward to put the time needed into investigating the wptserve issues. If somebody can commit to doing that, then I’m happy to give them shell access to the w3c-test.org server so they can get on there and investigate. Until that happens we’re going to remain in the same state we have been for the literally several years now that we’ve known this is an issue and been discussing it — and w3c-test.org is going to remain unstable.

So personally I think what would be far better instead at this point is that we move the hosting to some shared server that a number of us have shell access to — and quit calling it w3c-test.org but instead call it wpt-test.org or something (and make w3c-test.org just redirect to it). Then we could work together on debugging and fixing the problems needed to make it stable.

[jugglinmike]

It would be good to get a sense for how unstable the current deployment really is. @sideshowbarker can you speak to this anecdotally? Do you think that we'd get an accurate picture by cURL'ing http://w3c-test.org on a fixed interval for a couple of weeks? Or is the uptime too irregular for that to be representative?

[jugglinmike]

Looks like @foolip already set up some monitoring. I wrote a script to summarize that data (below); here are some quick calculations:

From 2017-10-12 to 2018-06-26
  Down 766 times (2.98 events/day)
  Down for 279 hours (95.48% availability)
  Mean downtime: 21.86 minutes
  Standard deviation: 8883
From 2018-06-20 to 2018-06-26
  Down 15 times (2.25 events/day)
  Down for 2 hours (98.60% availability)
  Mean downtime: 8.93 minutes
  Standard deviation: 191

From 2018-05-27 to 2018-06-26
  Response time
    Mean: 0.75 seconds
    Standard deviation: 1.22

Script used to generate that summary

from __future__ import division

from datetime import date
import json
import time
import urllib2

HOURS = 60 * 60
DAYS = HOURS * 24
YESTERDAY = time.time() - DAYS
LAST_WEEK = time.time() - 7 * DAYS

def get_json(url):
    request = urllib2.Request(url,
            headers={'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0'})
    response = urllib2.urlopen(request)
    return json.load(response)

def calculate(series):
    total = reduce(lambda x, y: x + y, series)
    mean = (total / len(series))
    variance = reduce(lambda x, y: x + (y - mean) ** 2, series) / len(series)
    return {
      'total': total,
      'mean': mean,
      'variance': variance
    }

def summarize_availability(availability):
    start = availability[-1]['Start_Unix']
    end = availability[0]['End_Unix']
    down_durations = []

    for event in availability:
        if event['Status'] == 'Up':
            continue
        down_durations.append(event['End_Unix'] - event['Start_Unix'])

    total_time = end - start
    stats = calculate(down_durations)

    print 'From %s to %s' % (
        date.fromtimestamp(start).isoformat(),
        date.fromtimestamp(end).isoformat()
    )
    print '  Down %d times (%.2f events/day)' % (
        len(down_durations), len(down_durations) / (total_time / DAYS)
    )
    print '  Down for %d hours (%.2f%% availability)' % (
        stats['total'] / HOURS, 100 * (total_time - stats['total']) / total_time
    )
    print '  Mean downtime: %.2f minutes' % (stats['mean'] / 60)
    print '  Standard deviation: %d' % (stats['variance'] ** 0.5)

def summarize_response_times(respone_times):
    start = response_times[0]['0']/1000
    end = response_times[-1]['0']/1000
    stats = calculate(map(lambda event: event['1'], response_times))
    print 'From %s to %s' % (
        date.fromtimestamp(start).isoformat(),
        date.fromtimestamp(end).isoformat()
    )
    print '  Response time'
    print '    Mean: %.2f seconds' % stats['mean']
    print '    Standard deviation: %.2f' % (stats['variance'] ** 0.5)

response_times = get_json('https://app.statuscake.com/Workfloor/Influx.php?Ref=Public&Type=Chart&TestID=2526304&All=true&TestType=HTTP&_=1530046510252')['All']
availability = get_json('https://app.statuscake.com/Workfloor/Get.Status.Perioids.php?PublicID=kbUT7hiAxw&VID=2526304')

summarize_availability(availability)
summarize_availability(
    filter(lambda event: event['Start_Unix'] > LAST_WEEK, availability)
)

print ''

summarize_response_times(response_times)

[jugglinmike]

@sideshowbarker my co-worker @bmac is up for the task! His handle in IRC is also "bmac"; do you think you could share access credentials with him?

[bmac]

Confirming @jugglinmike's assertion that I'm going to start dedicating some time investigating improving wptserve's reliability under load as a persistent process.

[sideshowbarker]

Thanks @jugglinmike and howdy @bmac! I’ve provisioned an account for you on the w3c-test.org host. Access to it is ssh-only, so to finish getting you set up, I’ll need your ssh public key.

[bmac]

Thanks @sideshowbarker. I sent you an email with my ssh public key.