DNS services are not resolving properly #5613

smarterclayton · 2015-11-02T22:13:17Z

The change on Sept 15th changed how services resolved in the absence
of search paths, which resulted in very long times to resolve DNS in
some cases.

This reduces search times to a few ms.

Fixes #5154

@liggitt

smarterclayton · 2015-11-02T22:15:59Z

[test]

smarterclayton · 2015-11-02T22:18:37Z

Fixes #5154

liggitt · 2015-11-02T22:23:10Z

Add tests to end to end to make sure short names still resolve? We currently have this:

MASTER_SERVICE_IP="$(dig @${API_HOST} "kubernetes.default.svc.cluster.local." +short A | head -n 1)"

Also test these?

kubernetes.default.svc
kubernetes.default
kubernetes

liggitt · 2015-11-02T22:24:02Z

actually, not sure shortnames will work unless we have search paths set up

smarterclayton · 2015-11-02T22:24:11Z

dns_test is pretty good. It's also very tolerant. I'd like to wait until
we have the SCC stuff to enable the upstream e2e.

On Mon, Nov 2, 2015 at 5:23 PM, Jordan Liggitt [email protected]
wrote:

Add tests to end to end to make sure short names still resolve? We
currently have this:

MASTER_SERVICE_IP="$(dig @${API_HOST} "kubernetes.default.svc.cluster.local." +short A | head -n 1)"

Also test these?

kubernetes.default.svc
kubernetes.default
kubernetes

—
Reply to this email directly or view it on GitHub
#5613 (comment).

smarterclayton · 2015-11-03T02:09:33Z

v1.0.6:

$ dig @localhost _endpoints.kubernetes.default.svc.cluster.local

; <<>> DiG 9.9.6-P1-RedHat-9.9.6-8.P1.fc21 <<>> @localhost _endpoints.kubernetes.default.svc.cluster.local
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14752
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;_endpoints.kubernetes.default.svc.cluster.local. IN A

;; ANSWER SECTION:
_endpoints.kubernetes.default.svc.cluster.local. 30 IN A 10.0.2.15

;; Query time: 86 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Nov 03 02:11:01 UTC 2015
;; MSG SIZE  rcvd: 81

[vagrant@openshiftdev origin]$ dig @localhost _endpoints.kubernetes.default.svc.cluster.local SRV

; <<>> DiG 9.9.6-P1-RedHat-9.9.6-8.P1.fc21 <<>> @localhost _endpoints.kubernetes.default.svc.cluster.local SRV
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39167
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;_endpoints.kubernetes.default.svc.cluster.local. IN SRV

;; ANSWER SECTION:
_endpoints.kubernetes.default.svc.cluster.local. 30 IN SRV 10 50 8443 unknown-port-8443.e1.kubernetes.

;; ADDITIONAL SECTION:
unknown-port-8443.e1.kubernetes. 30 IN  A   10.0.2.15

;; Query time: 253 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Nov 03 02:11:08 UTC 2015
;; MSG SIZE  rcvd: 163
[vagrant@openshiftdev origin]$ dig @localhost kubernetes.default.svc.cluster.local

; <<>> DiG 9.9.6-P1-RedHat-9.9.6-8.P1.fc21 <<>> @localhost kubernetes.default.svc.cluster.local
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27725
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;kubernetes.default.svc.cluster.local. IN A

;; ANSWER SECTION:
kubernetes.default.svc.cluster.local. 30 IN A   172.30.0.1

;; Query time: 133 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Nov 03 02:12:26 UTC 2015
;; MSG SIZE  rcvd: 70

[vagrant@openshiftdev origin]$ dig @localhost kubernetes.default.svc.cluster.local SRV

; <<>> DiG 9.9.6-P1-RedHat-9.9.6-8.P1.fc21 <<>> @localhost kubernetes.default.svc.cluster.local SRV
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56804
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;kubernetes.default.svc.cluster.local. IN SRV

;; ANSWER SECTION:
kubernetes.default.svc.cluster.local. 30 IN SRV 10 50 443 unknown-port-443.portal.kubernetes.

;; ADDITIONAL SECTION:
unknown-port-443.portal.kubernetes. 30 IN A 172.30.0.1

;; Query time: 113 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Nov 03 02:12:29 UTC 2015
;; MSG SIZE  rcvd: 158

liggitt · 2015-11-03T04:23:10Z

pkg/dns/serviceresolver.go

@@ -72,8 +76,30 @@ func (b *ServiceResolver) Records(name string, exact bool) ([]msg.Service, error
 	if len(segments) == 0 {
 		return nil, nil
 	}
+	glog.V(4).Infof("Answering query %s:%t", dnsName, exact)
+	switch base := segments[0]; base {
+	case "pod":


where did this come from?

kube added it

pointer to their stuff so we can be sure we're consistent?

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/README.md

liggitt · 2015-11-03T04:57:55Z

test/integration/dns_test.go

 					Port:   2346,
 				},
 				{
-					Target: "other.e1.headless2.",
+					Target: headless2IPHash + "._other._tcp.headless2.default.svc.cluster.local.",


citation needed

The change on Sept 15th changed how services resolved in the absence of search paths, which resulted in very long times to resolve DNS in some cases. Change the SRV record style to match upstream Kube (hash of pod ip) Add support for the "pod" range <IP>.namespace.pod.cluster.local resolves to <IP>. Update tests.

openshift-bot · 2015-11-03T07:33:06Z

Evaluated for origin test up to eb5c744

openshift-bot · 2015-11-03T08:32:52Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6696/)

smarterclayton · 2015-11-03T14:52:15Z

Any other comments?

smarterclayton · 2015-11-03T15:16:29Z

SRV records changed from _port._proto.SVCNAME to _port._proto.SVCNAME.NAMESPACE.svc.cluster.local (should resolve the same, but faster)

smarterclayton · 2015-11-03T15:18:30Z

Endpoint SRV records change from _port.EP<NUM>.SVCNAME to ENDPOINTIPHASH._port._proto.SVCNAME.NAMESPACE.svc.cluster.local

smarterclayton · 2015-11-03T15:19:18Z

[merge]

openshift-bot · 2015-11-03T15:25:08Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/6696/) (Image: devenv-rhel7_2637)

openshift-bot · 2015-11-03T15:25:08Z

Evaluated for origin merge up to eb5c744

Merged by openshift-bot

smarterclayton added component/master area/reliability priority/P1 labels Nov 2, 2015

smarterclayton mentioned this pull request Nov 2, 2015

Repeated skydns log: incomplete CNAME chain: rcode is not equal to success #5154

Closed

smarterclayton force-pushed the dns_broken branch from d5122fe to 975adec Compare November 2, 2015 22:36

smarterclayton force-pushed the dns_broken branch from 975adec to 506fca8 Compare November 3, 2015 03:02

liggitt reviewed Nov 3, 2015
View reviewed changes

smarterclayton force-pushed the dns_broken branch from 506fca8 to f3fa8fc Compare November 3, 2015 04:42

liggitt reviewed Nov 3, 2015
View reviewed changes

smarterclayton force-pushed the dns_broken branch from f3fa8fc to eb5c744 Compare November 3, 2015 05:58

danmcp assigned smarterclayton Nov 3, 2015

smarterclayton added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 3, 2015

openshift-bot pushed a commit that referenced this pull request Nov 3, 2015

Merge pull request #5613 from smarterclayton/dns_broken

1245d42

Merged by openshift-bot

openshift-bot merged commit 1245d42 into openshift:master Nov 3, 2015

smarterclayton added this to the 1.1.0 milestone Nov 8, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNS services are not resolving properly #5613

DNS services are not resolving properly #5613

smarterclayton commented Nov 2, 2015

smarterclayton commented Nov 2, 2015

smarterclayton commented Nov 2, 2015

liggitt commented Nov 2, 2015

liggitt commented Nov 2, 2015

smarterclayton commented Nov 2, 2015

smarterclayton commented Nov 3, 2015

liggitt Nov 3, 2015

smarterclayton Nov 3, 2015

liggitt Nov 3, 2015

smarterclayton Nov 3, 2015

liggitt Nov 3, 2015

openshift-bot commented Nov 3, 2015

openshift-bot commented Nov 3, 2015

smarterclayton commented Nov 3, 2015

smarterclayton commented Nov 3, 2015

smarterclayton commented Nov 3, 2015

smarterclayton commented Nov 3, 2015

openshift-bot commented Nov 3, 2015

openshift-bot commented Nov 3, 2015

DNS services are not resolving properly #5613

DNS services are not resolving properly #5613

Conversation

smarterclayton commented Nov 2, 2015

smarterclayton commented Nov 2, 2015

smarterclayton commented Nov 2, 2015

liggitt commented Nov 2, 2015

liggitt commented Nov 2, 2015

smarterclayton commented Nov 2, 2015

smarterclayton commented Nov 3, 2015

liggitt Nov 3, 2015

Choose a reason for hiding this comment

smarterclayton Nov 3, 2015

Choose a reason for hiding this comment

liggitt Nov 3, 2015

Choose a reason for hiding this comment

smarterclayton Nov 3, 2015

Choose a reason for hiding this comment

liggitt Nov 3, 2015

Choose a reason for hiding this comment

openshift-bot commented Nov 3, 2015

openshift-bot commented Nov 3, 2015

smarterclayton commented Nov 3, 2015

smarterclayton commented Nov 3, 2015

smarterclayton commented Nov 3, 2015

smarterclayton commented Nov 3, 2015

openshift-bot commented Nov 3, 2015

openshift-bot commented Nov 3, 2015