investigate flaky internet/test-dns-idna2008 test #25870

Trott · 2019-02-01T14:52:42Z

test/internet/test-dns-idna2008.js (added in four days ago when #25679 landed) failed in CI a few hours ago:

https://ci.nodejs.org/job/node-test-commit-custom-suites/855/default/console

Host: test-rackspace-ubuntu1604-x64-2

00:01:48 not ok 9 internet/test-dns-idna2008
00:01:48   ---
00:01:48   duration_ms: 1.14
00:01:48   severity: fail
00:01:48   exitcode: 1
00:01:48   stack: |-
00:01:48     (node:4568) ExperimentalWarning: The dns.promises API is experimental
00:01:48     assert.js:753
00:01:48         throw newErr;
00:01:48         ^
00:01:48     
00:01:48     AssertionError [ERR_ASSERTION]: ifError got unwanted exception: queryA ESERVFAIL straße.de
00:01:48         at QueryReqWrap.dns.resolve4.mustCall (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/internet/test-dns-idna2008.js:27:10)
00:01:48         at QueryReqWrap.callback (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:368:15)
00:01:48         at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:10)
00:01:48         at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:19)
00:01:48   ...

It had previously succeeded in CI (see https://ci.nodejs.org/job/node-test-commit-custom-suites/852/default/console).

I guess ESERVFAIL is a DNS problem likely unrelated to the test itself? Although other DNS tests in internet ran just fine, so maybe it is in fact specific to this test?

@bnoordhuis

The text was updated successfully, but these errors were encountered:

bnoordhuis · 2019-02-01T23:14:34Z

It's been a one-time failure so far? ESERVFAIL is reported by the DNS server; could be a temporary service outage.

Trott · 2019-02-02T00:45:15Z

It's been a one-time failure so far? ESERVFAIL is reported by the DNS server; could be a temporary service outage.

I had the temporary-hiccup idea too, but other DNS tests in the internet suite didn't fail. Still, the implication I'm getting from your comment--that there's probably not much point in chasing it down unless we see it more than once--is no doubt correct. Maybe leave this open for a bit and close it after a week or so if we don't see it again?

Trott · 2019-02-02T14:02:35Z

Failed again today:

https://ci.nodejs.org/job/node-test-commit-custom-suites/856/default/console

test-rackspace-ubuntu1604-x64-1

00:16:53 not ok 9 internet/test-dns-idna2008
00:16:53   ---
00:16:53   duration_ms: 1.14
00:16:53   severity: fail
00:16:53   exitcode: 1
00:16:53   stack: |-
00:16:53     (node:22610) ExperimentalWarning: The dns.promises API is experimental
00:16:53     assert.js:753
00:16:53         throw newErr;
00:16:53         ^
00:16:54     
00:16:54     AssertionError [ERR_ASSERTION]: ifError got unwanted exception: queryA ESERVFAIL straße.de
00:16:54         at QueryReqWrap.dns.resolve4.mustCall (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/internet/test-dns-idna2008.js:27:10)
00:16:54         at QueryReqWrap.callback (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:367:15)
00:16:54         at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:10)
00:16:54         at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:19)
00:16:54   ...

bnoordhuis · 2019-02-03T09:19:47Z

I'll take a look but I wonder if it's an environmental issue (i.e., network issue) because it never fails for me locally. What about you?

Trott · 2019-02-03T10:38:17Z

I'll take a look but I wonder if it's an environmental issue (i.e., network issue) because it never fails for me locally. What about you?

I've not had it fail locally either. It passed in last night's run on test-rackspace-ubuntu1604-x64-2.

To get a baseline (or to see if it passes 7 times, and then the firewall kicks in and it fails the remaining 93 times or whatever), here's 100 runs in CI:

https://ci.nodejs.org/job/node-stress-single-test/2147/

Trott · 2019-02-03T10:47:20Z

To get a baseline (or to see if it passes 7 times, and then the firewall kicks in and it fails the remaining 93 times or whatever), here's 100 runs in CI:

https://ci.nodejs.org/job/node-stress-single-test/2147/

No failures in stress test on test-rackspace-ubuntu1604-x64-2.

Re-running, but with all internet tests, but only 10 consecutive runs:

https://ci.nodejs.org/job/node-stress-single-test/2148/

Trott · 2019-02-03T11:05:20Z

Re-running, but with all internet tests, but only 10 consecutive runs:

https://ci.nodejs.org/job/node-stress-single-test/2148/

Also successful on test-rackspace-ubuntu1604-x64-2.
¯\(ツ)/¯

Trott · 2019-03-03T22:04:31Z

Happened in CI again today.

https://ci.nodejs.org/job/node-test-commit-custom-suites/897/default/console

test-rackspace-ubuntu1604-x64-1

00:05:33 not ok 9 internet/test-dns-idna2008
00:05:33   ---
00:05:33   duration_ms: 25.277
00:05:33   severity: fail
00:05:33   exitcode: 1
00:05:33   stack: |-
00:05:33     (node:18514) ExperimentalWarning: The dns.promises API is experimental
00:05:33     assert.js:742
00:05:33         throw newErr;
00:05:33         ^
00:05:33     
00:05:33     AssertionError [ERR_ASSERTION]: ifError got unwanted exception: queryA ESERVFAIL straße.de
00:05:33         at QueryReqWrap.dns.resolve4.mustCall (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/internet/test-dns-idna2008.js:27:10)
00:05:33         at QueryReqWrap.callback (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:367:15)
00:05:33         at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:10)
00:05:33         at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:19)
00:05:33   ...

Trott · 2019-03-04T14:56:16Z

Again today:

https://ci.nodejs.org/job/node-test-commit-custom-suites/898/default/console

test-rackspace-ubuntu1604-x64-1

00:04:15 not ok 9 internet/test-dns-idna2008
00:04:15   ---
00:04:15   duration_ms: 35.304
00:04:15   severity: fail
00:04:16   exitcode: 1
00:04:16   stack: |-
00:04:16     (node:25878) ExperimentalWarning: The dns.promises API is experimental
00:04:16     /home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:675
00:04:16     const crashOnUnhandledRejection = (err) => { throw err; };
00:04:16                                                  ^
00:04:16     
00:04:16     Error: queryA ESERVFAIL straße.de
00:04:16         at QueryReqWrap.onresolve [as oncomplete] (internal/dns/promises.js:167:17)
00:04:16   ...

Trott · 2019-03-05T14:34:19Z

It failed again today. So it failed each of the last three runs. (The test is run once a day in CI.) Interesting that when it fails, it fails reliably. And when it doesn't fail, it doesn't fail at all.

https://ci.nodejs.org/job/node-test-commit-custom-suites/899/default/console

test-rackspace-ubuntu1604-x64-1

00:04:16 not ok 9 internet/test-dns-idna2008
00:04:16   ---
00:04:16   duration_ms: 30.292
00:04:16   severity: fail
00:04:16   exitcode: 1
00:04:16   stack: |-
00:04:16     (node:22481) ExperimentalWarning: The dns.promises API is experimental
00:04:16     /home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:675
00:04:16     const crashOnUnhandledRejection = (err) => { throw err; };
00:04:16                                                  ^
00:04:16     
00:04:16     Error: queryA ESERVFAIL straße.de
00:04:16         at QueryReqWrap.onresolve [as oncomplete] (internal/dns/promises.js:167:17)
00:04:16   ...

Trott · 2019-03-05T14:44:50Z

@nodejs/build Thoughts on this one?

Trott · 2019-03-05T14:50:55Z

Ooh, this time around things are different from when initially reported. The test is taking 30 seconds or so before failing. Probably some throttling going on somewhere, yeah? I'm logged on and will try running the lookups/resolves in series rather than in parallel and see if that does anything...

Trott · 2019-03-05T14:52:37Z

Nope, still fails even with just one query and still takes about a long time to fail....

Trott · 2019-03-05T14:53:25Z

(Although it failed with EAGAIN instead of ESERVFAIL....)

Trott · 2019-03-06T14:35:00Z

And again:

https://ci.nodejs.org/job/node-test-commit-custom-suites/900/default/

test-rackspace-ubuntu1604-x64-1

00:19:37 not ok 9 internet/test-dns-idna2008
00:19:37   ---
00:19:37   duration_ms: 30.288
00:19:37   severity: fail
00:19:37   exitcode: 1
00:19:37   stack: |-
00:19:37     (node:1531) ExperimentalWarning: The dns.promises API is experimental
00:19:37     /home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:675
00:19:37     const crashOnUnhandledRejection = (err) => { throw err; };
00:19:37                                                  ^
00:19:37     
00:19:37     Error: queryA ESERVFAIL straße.de
00:19:37         at QueryReqWrap.onresolve [as oncomplete] (internal/dns/promises.js:167:17)
00:19:37   ...

Trott · 2019-03-06T14:46:02Z

FWIW, a reboot does not appear to have fixed it.

richardlau · 2019-03-08T17:15:42Z

@Trott I've (temporarily) unpinned this issue as we can only have three pinned and I think the current test breakage on master (#26528) should definitely be pinned. We can swap then back when master is unbroken.

refack · 2019-03-08T18:16:19Z

So... node-daily-master is unbroke.
But... this test will still be flaky if the system's DNS server fail to resolve straße.de.
.
.
.
I just had an idea: pass the dns.lookup test if the error is Error: queryA ESERVFAIL straße.de since IIUC it mean node passed the correct query to the OS.

* use known well-behaved DNS server * force pass on ESERVFAIL PR-URL: nodejs#26473 Fixes: nodejs#25870 Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

refack · 2019-03-10T14:50:48Z

https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/5173/testReport/junit/(root)/test/internet_test_dns_idna2008/, but but but, it passed CI!?

fix of fix in #26570

Fixes a bug I introduced in 9613221 PR-URL: nodejs#26570 Refs: nodejs#25870 Refs: nodejs#26473 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]>

* use known well-behaved DNS server * force pass on ESERVFAIL PR-URL: #26473 Fixes: #25870 Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

Fixes a bug I introduced in 9613221 PR-URL: #26570 Refs: #25870 Refs: #26473 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]>

* use known well-behaved DNS server * force pass on ESERVFAIL PR-URL: #26473 Fixes: #25870 Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

Fixes a bug I introduced in 9613221 PR-URL: #26570 Refs: #25870 Refs: #26473 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]>

Fixes a bug I introduced in 9613221 PR-URL: nodejs#26570 Refs: nodejs#25870 Refs: nodejs#26473 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]>

Fixes a bug I introduced in 9613221 PR-URL: #26570 Refs: #25870 Refs: #26473 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]>

The test should pass if ESERVFAIL is the result. Refs: nodejs#25870 (comment)

The test should pass if ESERVFAIL is the result. Refs: nodejs#25870 (comment) PR-URL: nodejs#27208 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>

Trott added dns Issues and PRs related to the dns subsystem. flaky-test Issues and PRs related to the tests with unstable failures on the CI. labels Feb 1, 2019

Trott pinned this issue Mar 3, 2019

refack mentioned this issue Mar 6, 2019

test: de-flake test-dns-idna2008.js #26473

Merged

3 tasks

refack self-assigned this Mar 7, 2019

richardlau unpinned this issue Mar 8, 2019

refack closed this as completed in #26473 Mar 9, 2019

refack mentioned this issue Mar 10, 2019

test: fix syntax error in test-dns-idna2008.js when failing #26570

Merged

3 tasks

refack removed their assignment Mar 11, 2019

Trott added a commit to Trott/io.js that referenced this issue Apr 13, 2019

test: fix test-dns-idna2008.js

4fd2912

The test should pass if ESERVFAIL is the result. Refs: nodejs#25870 (comment)

Trott mentioned this issue Apr 13, 2019

test: fix test-dns-idna2008.js #27208

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

investigate flaky internet/test-dns-idna2008 test #25870

investigate flaky internet/test-dns-idna2008 test #25870

Trott commented Feb 1, 2019

bnoordhuis commented Feb 1, 2019

Trott commented Feb 2, 2019

Trott commented Feb 2, 2019

bnoordhuis commented Feb 3, 2019

Trott commented Feb 3, 2019

Trott commented Feb 3, 2019

Trott commented Feb 3, 2019

Trott commented Mar 3, 2019

Trott commented Mar 4, 2019

Trott commented Mar 5, 2019

Trott commented Mar 5, 2019

Trott commented Mar 5, 2019

Trott commented Mar 5, 2019

Trott commented Mar 5, 2019 •

edited

Loading

Trott commented Mar 6, 2019

Trott commented Mar 6, 2019

richardlau commented Mar 8, 2019 •

edited

Loading

refack commented Mar 8, 2019

refack commented Mar 10, 2019

investigate flaky internet/test-dns-idna2008 test #25870

investigate flaky internet/test-dns-idna2008 test #25870

Comments

Trott commented Feb 1, 2019

bnoordhuis commented Feb 1, 2019

Trott commented Feb 2, 2019

Trott commented Feb 2, 2019

bnoordhuis commented Feb 3, 2019

Trott commented Feb 3, 2019

Trott commented Feb 3, 2019

Trott commented Feb 3, 2019

Trott commented Mar 3, 2019

Trott commented Mar 4, 2019

Trott commented Mar 5, 2019

Trott commented Mar 5, 2019

Trott commented Mar 5, 2019

Trott commented Mar 5, 2019

Trott commented Mar 5, 2019 • edited Loading

Trott commented Mar 6, 2019

Trott commented Mar 6, 2019

richardlau commented Mar 8, 2019 • edited Loading

refack commented Mar 8, 2019

refack commented Mar 10, 2019

Trott commented Mar 5, 2019 •

edited

Loading

richardlau commented Mar 8, 2019 •

edited

Loading