[FCOS] Pull MCD image and extract binary if it doesn't exists

[vrutkovs]

This removes condition to run this after first boot, cause some UPI
installs may want to have other modification right after the
first boot.

Required to make vSphere UPI tests pass, as these reboot machine to set static IP thus ignition-firstboot-complete.service is not active and MCD image cannot be pulled.

/cc @LorbusChris

[cgwalters]

Required to make vSphere UPI tests pass, as these reboot machine to set static IP thus ignition-firstboot-complete.service is not active and MCD image cannot be pulled.

Hm, well we should probably reopen
openshift/installer#2554

[cgwalters]

Oh I bet the problem you're hitting is coreos/ignition-dracut@598bbdd hasn't shipped to FCOS stable yet?

[cgwalters]

Fine with me for now, though let's remember to drop this once you rebase to a newer FCOS.
/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cgwalters,vrutkovs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[vrutkovs]

Weird, seems machine-os-content is generating a broken oscontainer?

Nov 20 17:25:36 ip-10-0-6-115 bootkube.sh[1083]: I1120 17:25:36.292285 3206 rpm-ostree.go:247] No com.coreos.ostree-commit label found in metadata! Inspecting... Nov 20 17:25:36 ip-10-0-6-115 bootkube.sh[1083]: I1120 17:25:36.292430 3206 rpm-ostree.go:364] Running captured: ostree refs --repo /var/lib/containers/storage/overlay/d6c8f4c284c2e9ed94bf40f952b8acd82f9fecc6cdafdef13f3aa0829df04b01/merged/srv/repo Nov 20 17:25:36 ip-10-0-6-115 bootkube.sh[1083]: error: /var/lib/containers/storage/overlay/d6c8f4c284c2e9ed94bf40f952b8acd82f9fecc6cdafdef13f3aa0829df04b01/merged/srv/repo: opendir(/var/lib/containers/storage/overlay/d6c8f4c284c2e9ed94bf40f952b8acd82f9fecc6cdafdef13f3aa0829df04b01/merged/srv/repo): No such file or directory Nov 20 17:25:36 ip-10-0-6-115 bootkube.sh[1083]: error: error running ostree refs --repo /var/lib/containers/storage/overlay/d6c8f4c284c2e9ed94bf40f952b8acd82f9fecc6cdafdef13f3aa0829df04b01/merged/srv/repo: : exit status 1

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[vrutkovs]

/hold

[cgwalters]

Interesting...I think this may be related to the high UIDs saved in the image?

https://paste.fedoraproject.org/paste/w9kqQCa5OtzbYFZLrv5OOA

# Host, podman unprivileged

walters@quicksilver ~> podman pull registry.svc.ci.openshift.org/fcos/machine-os-content:4.3
Trying to pull registry.svc.ci.openshift.org/fcos/machine-os-content:4.3...
Getting image source signatures
Copying blob 91e66579173e done
Copying blob 4f4fb700ef54 done
Copying config 1ca9270195 done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 1000240000:1000240000 for /srv/repo): lchown /srv/repo: invalid argument
Error: error pulling image "registry.svc.ci.openshift.org/fcos/machine-os-content:4.3": unable to pull registry.svc.ci.openshift.org/fcos/machine-os-content:4.3: unable to pull image: Error committing the finished image: error adding layer with blob "sha256:91e66579173e2bbd1fa2aea41484902f421186a45821d720ef94e7de80397639": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 1000240000:1000240000 for /srv/repo): lchown /srv/repo: invalid argument
walters@quicksilver ~>

# RHCOS VM, privileged podman

[root@coreos merged]# ls -al srv/repo/
total 16
drwxr-sr-x.   7 1000240000 1000240000  102 Nov 20 16:03 .
drwxr-xr-x.   3 core       core         18 Nov 20 16:00 ..
-rw-------.   1 1000240000 1000240000    0 Nov 20 14:45 .lock
-rw-r--r--.   1 1000240000 1000240000   38 Nov 20 14:45 config
drwxr-sr-x.   2 1000240000 1000240000    6 Nov 20 14:45 extensions
drwxr-sr-x. 258 1000240000 1000240000 8192 Nov 20 14:45 objects
drwxr-sr-x.   5 1000240000 1000240000   49 Nov 20 14:45 refs
drwxr-sr-x.   2 1000240000 1000240000    6 Nov 20 14:45 state
drwxr-sr-x.   3 1000240000 1000240000   19 Nov 20 16:03 tmp

I think we really want the builder to RunAsUser: 0 so that the saved files are stored as 0.

[vrutkovs]

Could be related, however the pull succeeds:

Nov 21 13:13:21 ip-10-0-7-29 bootkube.sh[1084]: I1121 13:13:21.580257    3234 run.go:16] Running: podman pull -q registry.svc.ci.openshift.org/ci-op-7xcp2x9q/stable@sha256:3ca5bea9af9cd7579ff2f459e40001e61058abe653cd412ca47431a126bd81c1
Nov 21 13:13:29 ip-10-0-7-29 podman[3386]: 2019-11-21 13:13:29.066935762 +0000 UTC m=+7.461054845 image pull  
Nov 21 13:13:29 ip-10-0-7-29 bootkube.sh[1084]: edb21247b82d33765c25d0c0f700886e56ea55d00df97d9a2ef274ff5db31144

[vrutkovs]

interestingly enough this doesn't happen on installer PR, so there's probably something broken machine-os-content

[LorbusChris]

hm, weird. I'll kick e2e again, maybe something has changed over the past few days..

/retest

[vrutkovs]

/retest

[LorbusChris]

@vrutkovs just as a note, we need to make sure this goes into the rebase over in #1289 as well :)

[LorbusChris]

/retest

[vrutkovs]

/retest

[vrutkovs]

/retest

[vrutkovs]

openshift/release#6181 was breaking os-content, it got merged and tests should eventually pass

[vrutkovs]

/hold cancel