Introduce results collection with Buildbot #522

jugglinmike · 2018-03-21T01:44:52Z

Although it's preferable to submit changes in piecemeal fashion, this change to
the infrastructure is difficult to break down in a meaningful way. I plan to
walk @rwaldron through the patch in person tomorrow. I'm happy to do the same
for any interested party using some video conferencing/desktop sharing system.
In addition, the new system is currently deployed so
that we have a better sense of how it actually functions.

jugglinmike · 2018-03-21T16:35:42Z

The trial I ran last night surfaced a number of issues with the final
"notification" step:

The bucket name option is ignored (The temporary "wptd2" bucket is hard-coded
into the script. Also note that the bucket in use has to be CORS-enabled;
instructions for doing this are available in the Google Cloud Storage
documentation)
The public web URL of the summary file should be specified instead of the
gs:// URL
The script should accept any HTTP response in the 2xx range (The current
script expects a 201, but I'm not sure why this is.)

There are a few other changes that need to be made to integrate with the
current system, but we should revisit them as we de-couple the results
collection and WPT Dashboard itself:

The SHA sum representing the git revision of WPT should be truncated to 10
characters in order to render correctly in the web UI (moving forward, we
should use fully-qualified SHAs with the expectation that consumers will
adjust this value according to their needs)
The results should be organized in terms of an identifier from the
browser.json file. This duplicates the structured information of that file
but only imperfectly. For instance, the operating system version" is not
always included.

I'll be addressing these issues with new commits to this pull request shortly.

rwaldron

A few minor nits. Some are serious, others are not. I'll let you determine which are which ;)

rwaldron · 2018-03-21T19:22:39Z

run/README.md

+3. Open a terminal and navigate to the directory containing this text file
+4. Run the following command:
+
+       `ansible-playbook -i inventory/production playbook.yml


Missing closing backtick (either add, or remove both and rely on indentation)

rwaldron · 2018-03-21T19:25:30Z

run/provisioning/inventory/production

+31356437386434326330643736653262636238643261616330343432326432626262376533646133
+64613530623536373936663836383430666536356537326331663430363663306639313664363834
+35363035353636616166303262346661353137363139626662663332356438366330343737633535
+336439363936303136353532343039343034


I suspect there might be an overallocation of the number 3 in this data... can you confirm?

You're right. I had a few 3s left over from an old project, and I thought it would be okay to use them here

rwaldron · 2018-03-21T19:40:22Z

run/provisioning/roles/web-browsers/tasks/main.yml

+    - emacs-nox
+    - firefox
+    - git
+    - google-chrome-stable


Note: we acknowledge that inclusion of google-chrome-stable and firefox are temporary measures

It's true, and it wouldn't hurt to document that intention with a comment, either

rwaldron · 2018-03-21T19:41:35Z

run/provisioning/vars/vault

+39323032613636356336363863386234353163396635666430386266303634643531623135396138
+65363333376463643933616265666239386339636338383665393339643139396530393930326264
+31383632393163333030656535323036316565336633373964613461303363636262623933366235
+3564303937653961333139356261653963363934316362363131


This one is substantially shorter than the previous—are you sure that's correct?

I assumed so, but now that you ask, I'm not so sure.

rwaldron · 2018-03-21T19:46:31Z

run/runner.py

-        gs_http_results_url=None,
-        summary_filename=None,
-        summary_http_url=None,
-    ):


This sad-face emoji was strangely appropriate following all of those arguments for one function.

A sure sign of technical debt. Note that this patch, fresh as it is, has its own problems--see the main function of upload-wpt-results.py.

rwaldron · 2018-03-21T19:51:20Z

run/src/master/master.cfg

+####### CHANGESOURCES
+
+# the 'change_source' setting tells the buildmaster how it should find out
+# about source code changes.  Here we point to the buildbot version of a python hello-world project.


At some point these comments should be updated (wherever applicable) to reflect reality

Done and done

rwaldron · 2018-03-21T19:57:13Z

run/src/master/master.cfg

+                      workdir='gitpoller-workdir',
+                      branch='master',
+                      pollAtLaunch=True,
+                      pollinterval=300)


pollinterval is deprecated. Update to pollInterval

You're like a bloodhound for this stuff. See also buildbot/buildbot#4011

rwaldron · 2018-03-21T20:02:47Z

run/src/master/master.cfg

+    chunked_steps.append(WPTChunkedStep(schedulerNames=['chunked'],
+                                        platform=spec,
+                                        doStepIf=partial(filter_build, speed),
+                                        total_chunks=100))


todo: make a total_chunks variable

rwaldron · 2018-03-21T20:08:10Z

run/src/master/master.cfg

+        # failed builds are manually re-tried via the web interface.
+        steps.Git(repourl='git://github.com/w3c/web-platform-tests'),
+        steps.SetPropertyFromCommand(name='Retrieve commit date of latest revision',
+                                     command=['git', 'log', '-1', '--format=%cd', '--date=iso-strict'],


Please add a comment explaining why this code is capturing the date of the commit now, instead of at the end.

rwaldron · 2018-03-21T20:31:55Z

util/provision-run/vars/vault

-64326632313065306237336339373361616130366139653530396534363063353535616530303964
-33626264633034383132623361353762363939633032663364633938323630316131626363376566
-36353337363438346537386335306535343765373565373430366338653033366336393665323534
-383839633333646530356434626530623530


I think this is the rest of the code that's missing from the second file I reviewed? IDK...?

It might be. Honestly, I have a hard time keeping track of these really long number

rwaldron · 2018-03-26T14:42:01Z

As far as I can tell, the travis failure is unrelated to this changeset.

rwaldron · 2018-03-27T14:20:35Z

run/src/master/master.cfg

@@ -215,7 +215,7 @@ upload_factory = util.BuildFactory([
                                 '--os-version', util.Property('os_version'),
                                 '--wpt-revision', util.Property('revision'),
                                 '--wpt-revision-date', util.Property('revision_date'),
-                                 '--bucket-name', 'wptd2',
+                                 '--bucket-name', 'wptd',


jugglinmike · 2018-03-27T15:49:00Z

Thanks, @rwaldron. I've squashed most of the commits, but I'm retaining a few that have to do with file re-organization. I think it makes sense to keep these distinct because they would only add noise to the commit that introduces the new system. Since the intermediary commits do not describe valid states of the code base, this branch should be merged with a merge commit.

The original version of this branch is available here:

master...bocoup:buildbot-prebase

rwaldron · 2018-03-27T16:27:03Z

@jugglinmike thanks!

jugglinmike added 2 commits March 20, 2018 18:35

Re-locate provisioning scripts

e3a14ad

Remove original implementation

39dd38c

boazsender added the review label Mar 21, 2018

rwaldron suggested changes Mar 21, 2018

View reviewed changes

jugglinmike force-pushed the buildbot branch from fd96734 to 9dfb09a Compare March 22, 2018 16:21

rwaldron approved these changes Mar 26, 2018

View reviewed changes

rwaldron reviewed Mar 27, 2018

View reviewed changes

Implement results collection via Buildbot

0a89050

jugglinmike force-pushed the buildbot branch from 6ba559f to 0a89050 Compare March 27, 2018 15:44

rwaldron merged commit 0a89050 into web-platform-tests:master Mar 27, 2018

boazsender removed the review label Mar 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce results collection with Buildbot #522

Introduce results collection with Buildbot #522

jugglinmike commented Mar 21, 2018

jugglinmike commented Mar 21, 2018

rwaldron left a comment

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 23, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 21, 2018

jugglinmike Mar 21, 2018

rwaldron Mar 23, 2018

rwaldron commented Mar 26, 2018

rwaldron Mar 27, 2018

jugglinmike commented Mar 27, 2018

rwaldron commented Mar 27, 2018

Introduce results collection with Buildbot #522

Introduce results collection with Buildbot #522

Conversation

jugglinmike commented Mar 21, 2018

jugglinmike commented Mar 21, 2018

rwaldron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwaldron commented Mar 26, 2018

Choose a reason for hiding this comment

jugglinmike commented Mar 27, 2018

rwaldron commented Mar 27, 2018