Custom elements can miss the upgrade when it is defined in an import in a frame (flaky tests since M35 roll)

[leafpetersen]

This bug is to track the failing tests in the M35 roll.

The following tests are failing in M35 and have been disabled. Todo: fix them.

pkg/third_party/angular_tests/browser_test/core_dom/shadow_root_options

[leafpetersen]

pkg/third_party/angular_tests/browser_test/core_dom/shadow_root_options
html/form_data_test/functional
html/indexeddb_5_test
html/custom/attribute_changed_callback_test/unsupported_on_polyfill
html/websql_test/functional
co19/LayoutTests/fast/dom/HTMLObjectElement/set-type-to-null-crash_t01
co19/LayoutTests/fast/dom/HTMLOutputElement/dom-settable-token-list_t01
co19/LayoutTests/fast/dom/MutationObserver/database-callback-delivery_t01
co19/LayoutTests/fast/dom/MutationObserver/observe-attributes_t01
co19/LayoutTests/fast/dom/NodeIterator/NodeIterator-basic_t01
co19/WebPlatformTest/dom/nodes/Node-textContent_t01
co19/WebPlatformTest/shadow-dom/elements-and-dom-objects/shadowroot-object/shadowroot-attributes/test-005_t01
co19/WebPlatformTest/shadow-dom/elements-and-dom-objects/shadowroot-object/shadowroot-attributes/test-006_t01
co19/WebPlatformTest/shadow-dom/elements-and-dom-objects/the-content-html-element/test-004_t01
co19/WebPlatformTest/shadow-dom/elements-and-dom-objects/the-content-html-element/test-004_t02
co19/WebPlatformTest/shadow-dom/elements-and-dom-objects/the-shadow-html-element/test-003_t01
co19/WebPlatformTest/shadow-dom/events/event-dispatch/test-002_t01
co19/WebPlatformTest/shadow-dom/elements-and-dom-objects/the-shadow-html-element/test-003_t01
co19/WebPlatformTest/shadow-dom/events/retargeting-relatedtarget/test-001_t01
co19/WebPlatformTest/shadow-dom/events/retargeting-relatedtarget/test-002_t01
co19/WebPlatformTest/shadow-dom/shadow-trees/upper-boundary-encapsulation/test-009_t01
pkg/polymer/example/component/news/test/news_index_test
pkg/polymer/test/attr_deserialize_test
pkg/polymer/test/attr_mustache_test
pkg/polymer/test/bind_test
pkg/polymer/test/entered_view_test
pkg/polymer/test/event_handlers_test
pkg/polymer/test/event_path_test
pkg/polymer/test/events_test
pkg/polymer/test/instance_attrs_test
pkg/polymer/test/js_interop_test
pkg/polymer/test/nested_binding_test
pkg/polymer/test/noscript_test
pkg/polymer/test/prop_attr_bind_reflection_test
pkg/polymer/test/prop_attr_reflection_test

[leafpetersen]

co19/LayoutTests/fast/dom/MutationObserver/database-callback-delivery_t01
co19/WebPlatformTest/dom/EventTarget/addEventListener_A01_t01
co19/WebPlatformTest/dom/EventTarget/removeEventListener_A01_t01
html/websql_test/functional

Fixed in r36591

[jmesserly]

fyi -- every polymer test is marked flaky so it's hiding real failures :|
i'm not seeing any flakes in 36 with my changes. removed in https://codereview.chromium.org/307793002

[vsmenon]

Thanks, John! Are there any remaining regressions on polymer tests?

[jmesserly]

ugh. the failures don't happen locally but happen on the buildbot :(

[leafpetersen]

Been a while since I tried this, but I was able to reproduce these locally at one point, but only by running the full test suite. Running any individual test would never result in a failure for me.

[floitschG]

cc @ricowind.

[whesse]

A large number of these all pass locally, so they may be passing with chrome 37. Enabling many of them, and running on ContentShellOnAndroid as well.

[ricowind]

ugh. the failures don't happen locally but happen on the buildbot :(
Flaky stuff like this is hard to debug, here are a few notes:
The browser controller will run many tests in the same browser, so if, by some chance, the browser (here dartium) has some unintentional state between loading new pages in the iframe this can easily flake.
There will be differences between running this on the bots and locally:
Locally you will run simultaneously in 32 browsers (assuming normal z620), i.e., the chance of two of these polymer tests getting scheduled on the same browser is smaller
On the bot we run with 4 simultaneously -> higher chance

You can try to simulate by using -j4 or even -j1, but that is not a guarantee that it will reproduce.

[whesse]

Yes, the flaky failures continue to happen on the buildbot, even after rolling to chrome 37. I will put the flaky tests back in the status file. Here is a list of which tests failed on which platforms, in a single run at revision 38829:

win-inc

FAILED: none-dartium release_ia32 pkg/polymer/test/attr_deserialize_test
FAILED: none-dartium release_ia32 pkg/polymer/test/attr_mustache_test
FAILED: none-dartium release_ia32 pkg/polymer/test/bind_test
FAILED: none-dartium release_ia32 pkg/polymer/test/computed_properties_test
FAILED: none-dartium release_ia32 pkg/polymer/test/custom_event_test
FAILED: none-dartium release_ia32 pkg/polymer/test/event_handlers_test
FAILED: none-dartium release_ia32 pkg/polymer/test/event_path_test
FAILED: none-dartium release_ia32 pkg/polymer/test/events_test
FAILED: none-dartium release_ia32 pkg/polymer/test/instance_attrs_test
FAILED: none-dartium release_ia32 pkg/polymer/test/js_custom_event_test
FAILED: none-dartium release_ia32 pkg/polymer/test/nested_binding_test
FAILED: none-dartium release_ia32 pkg/polymer/test/noscript_test
FAILED: none-dartium release_ia32 pkg/polymer/test/prop_attr_bind_reflection_test
FAILED: none-dartium release_ia32 pkg/polymer/test/publish_attributes_test
FAILED: none-dartium release_ia32 pkg/polymer/test/publish_inherited_properties_test
FAILED: none-dartium release_ia32 pkg/polymer/test/register_test
FAILED: none-dartium release_ia32 pkg/polymer/test/take_attributes_test
FAILED: none-dartium release_ia32 pkg/polymer/test/two_way_bind_test
FAILED: none-dartium release_ia32 pkg/web_components/test/interop_test

mac-inc
FAILED: none-dartium release_ia32 pkg/polymer/test/attr_deserialize_test
FAILED: none-dartium release_ia32 pkg/polymer/test/attr_mustache_test
FAILED: none-dartium release_ia32 pkg/polymer/test/bind_test
FAILED: none-dartium release_ia32 pkg/polymer/test/computed_properties_test
FAILED: none-dartium release_ia32 pkg/polymer/test/custom_event_test
FAILED: none-dartium release_ia32 pkg/polymer/test/entered_view_test
FAILED: none-dartium release_ia32 pkg/polymer/test/event_handlers_test

linux-inc
FAILED: none-dartium release_ia32 pkg/polymer/test/attr_mustache_test
FAILED: none-dartium release_ia32 pkg/polymer/test/js_custom_event_test
FAILED: none-dartium release_ia32 pkg/polymer/test/nested_binding_test
FAILED: none-dartium release_ia32 pkg/polymer/test/noscript_test
FAILED: none-dartium release_ia32 pkg/polymer/test/register_test
FAILED: none-dartium release_ia32 pkg/polymer/test/take_attributes_test

mac-checked
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/attr_deserialize_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/entered_view_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/js_custom_event_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/prop_attr_bind_reflec

linux-checked
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/attr_mustache_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/js_custom_event_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/nested_binding_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/noscript_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/register_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/take_attributes_test

windows-checked
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/attr_deserialize_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/attr_mustache_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/bind_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/computed_properties_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/custom_event_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/entered_view_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/event_path_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/events_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/instance_attrs_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/js_custom_event_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/js_interop_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/nested_binding_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/noscript_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/prop_attr_bind_reflection_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/prop_attr_reflection_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/publish_attributes_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/publish_inherited_properties_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/take_attributes_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/template_distribute_dynamic_test
FAILED: none-dartium-checked release_ia32 pkg/polymer/test/two_way_bind_test
FAILED: none-dartium-checked release_ia32 pkg/web_components/test/interop_test

[whesse]

cc @jmesserly.
cc @sigmundch.

[sigmundch]

We actually investigated this for a while. Leaf was able to reproduce locally if you run all of the html tests in dartium locally (that's enough load sometimes to trigger a flake), and with Pete they narrowed down a bit what was the source of the problem. Later on Vijay discovered that this issue is highlighted by the fact that we run tests within an iframe, and they came up with a way to reliably reproduce the failure using 2 iframes.

This is now being tracked as a chromium bug here:
https://code.google.com/p/chromium/issues/detail?id=384516

---

cc @blois.
cc @vsmenon.

[sigmundch]

Changed the title to: "Custom elements can miss the upgrade when it is defined in an import in a frame (flaky tests since M35 roll)".

[sigmundch]

FYI - the chromium bug was just marked as fixed, so we might see some of these flakes get fixed whenever that rolls into Dartium