downward API extended test flake #12629

smarterclayton · 2017-01-23T22:56:00Z

Uh oh:

Extended.[k8s.io] Downward API volume should update labels on modification [Conformance]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/10550/testReport/junit/(root)/Extended/_k8s_io__Downward_API_volume_should_update_labels_on_modification__Conformance_/

/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:124
Timed out after 120.001s.
Expected
    <string>: content of file "/etc/labels": key1="value1"
    key2="value2"
    ...
    content of file "/etc/labels": key1="value1"
    key2="value2"
    
to contain substring
    <string>: key3="value3"
    
/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:123

Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:12.227062384 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {default-scheduler } Scheduled: Successfully assigned labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0 to 172.18.4.242
Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:17.219264543 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {kubelet 172.18.4.242} Pulled: Container image "gcr.io/google_containers/mounttest:0.7" already present on machine
Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:17.945694933 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {kubelet 172.18.4.242} Created: Created container with docker id eae9aa4c3f70; Security:[seccomp=unconfined]
Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:19.558858783 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {kubelet 172.18.4.242} Started: Started container with docker id eae9aa4c3f70
Jan 23 15:18:26.827: INFO: At 2017-01-23 15:16:26 -0500 EST - event for labelsupdatec8dca6b8-e1a8-11e6-a908-0e53c2054dd0: {kubelet 172.18.4.242} FailedMount: MountVolume.SetUp failed for volume "kubernetes.io/downward-api/c8ddb323-e1a8-11e6-9825-0e53c2054dd0-podinfo" (spec.Name: "podinfo") pod "c8ddb323-e1a8-11e6-9825-0e53c2054dd0" (UID: "c8ddb323-e1a8-11e6-9825-0e53c2054dd0") with: remove /mnt/openshift-xfs-vol-dir/pods/c8ddb323-e1a8-11e6-9825-0e53c2054dd0/volumes/kubernetes.io~downward-api/podinfo/resolv.conf: device or resource busy

The text was updated successfully, but these errors were encountered:

smarterclayton · 2017-01-23T22:56:20Z

@pmorie @ncdc have never seen this before

smarterclayton · 2017-01-23T22:56:36Z

High priority in case this is a severe issue under the covers

ncdc · 2017-01-24T20:59:37Z

This failed because the kubelet wasn't able to write the updated downward api file after the pod update was issued. The device or resource busy error trying to remove the old data is the problem. Not sure why it hit that error.

derekwaynecarr · 2017-02-02T19:05:04Z

given andy's comment, do people agree with a lower priority?

smarterclayton · 2017-02-02T20:16:40Z

Should the downward API retry? Does it already?

ncdc · 2017-02-06T15:03:44Z

It does retry, but I don't think retrying is specific to downward api. It's just the kubelet and the volume code trying to set up a volume mount.

smarterclayton · 2017-02-08T01:10:49Z

Ha! https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/5303

ncdc · 2017-02-08T17:02:46Z

@derekwaynecarr @sjenning @pmorie it's probably worth digging into this when someone has some free time to see what is holding the file open (assuming that's the reason).

derekwaynecarr · 2017-02-10T19:18:13Z

@pmorie -- can you look at this?

pmorie · 2017-02-13T20:54:38Z

This one flakes occasionally upstream as well, and has been for quite a while - moving to p2.

derekwaynecarr · 2017-02-13T21:01:13Z

i assume this issue: kubernetes/kubernetes#37456 (as of 11/26)

0xmichalis · 2017-12-05T08:36:29Z

Different failure, same test

/tmp/openshift/init/rpm/BUILD/origin-3.8.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:125
Timed out after 120.000s.
Expected
    <string>: content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    content of file "/etc/labels": key1="value1"
    key2="value2"
    
to contain substring
    <string>: key3="value3"
    
/tmp/openshift/init/rpm/BUILD/origin-3.8.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:124

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17589/test_pull_request_origin_extended_conformance_install/3493/

openshift-bot · 2018-03-05T11:32:35Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

sdodson · 2018-03-05T15:38:10Z

/remove-lifecycle stale

openshift-bot · 2018-06-03T18:40:25Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2018-07-03T18:41:45Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2018-08-02T19:59:13Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

smarterclayton added kind/test-flake Categorizes issue or PR as related to test flakes. priority/P1 labels Jan 23, 2017

pweil- assigned pmorie Jan 24, 2017

pweil- added component/kubernetes component/storage labels Jan 24, 2017

pmorie added priority/P2 and removed priority/P1 labels Feb 13, 2017

gabemontero mentioned this issue Mar 27, 2017

address redundant line if new-app error output #13541

Merged

mtnbikenc mentioned this issue Apr 13, 2017

Disable swap space on nodes at install and upgrade openshift/openshift-ansible#3884

Merged

0xmichalis mentioned this issue Dec 5, 2017

Automated cherry-pick of #17577 on release-3.8 #17589

Merged

php-coder mentioned this issue Dec 12, 2017

Improve the process of pod updates by preferring non-mutating SCCs and reducing pod mutations #16934

Merged

vrutkovs mentioned this issue Jan 5, 2018

Fix flake8 errors in utils/test openshift/openshift-ansible#6488

Merged

sosiouxme mentioned this issue Jan 22, 2018

[sig-storage] Downward API volume should update labels on modification [Conformance] [Suite:openshift/conformance/parallel] [Suite:k8s] 3m21s #17882

Closed

tsmetana mentioned this issue Feb 12, 2018

UPSTREAM: 58685: Fill size attribute for the OpenStack V3 API volumes #18237

Merged

openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 5, 2018

openshift-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 5, 2018

openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 3, 2018

openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 3, 2018

openshift-ci-robot closed this as completed Aug 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

downward API extended test flake #12629

downward API extended test flake #12629

smarterclayton commented Jan 23, 2017

smarterclayton commented Jan 23, 2017

smarterclayton commented Jan 23, 2017

ncdc commented Jan 24, 2017 •

edited

Loading

derekwaynecarr commented Feb 2, 2017

smarterclayton commented Feb 2, 2017

ncdc commented Feb 6, 2017

smarterclayton commented Feb 8, 2017

ncdc commented Feb 8, 2017

derekwaynecarr commented Feb 10, 2017

pmorie commented Feb 13, 2017

derekwaynecarr commented Feb 13, 2017

0xmichalis commented Dec 5, 2017 •

edited

Loading

openshift-bot commented Mar 5, 2018

sdodson commented Mar 5, 2018

openshift-bot commented Jun 3, 2018

openshift-bot commented Jul 3, 2018

openshift-bot commented Aug 2, 2018

downward API extended test flake #12629

downward API extended test flake #12629

Comments

smarterclayton commented Jan 23, 2017

smarterclayton commented Jan 23, 2017

smarterclayton commented Jan 23, 2017

ncdc commented Jan 24, 2017 • edited Loading

derekwaynecarr commented Feb 2, 2017

smarterclayton commented Feb 2, 2017

ncdc commented Feb 6, 2017

smarterclayton commented Feb 8, 2017

ncdc commented Feb 8, 2017

derekwaynecarr commented Feb 10, 2017

pmorie commented Feb 13, 2017

derekwaynecarr commented Feb 13, 2017

0xmichalis commented Dec 5, 2017 • edited Loading

openshift-bot commented Mar 5, 2018

sdodson commented Mar 5, 2018

openshift-bot commented Jun 3, 2018

openshift-bot commented Jul 3, 2018

openshift-bot commented Aug 2, 2018

ncdc commented Jan 24, 2017 •

edited

Loading

0xmichalis commented Dec 5, 2017 •

edited

Loading