Latest CRI-O does not work for CC

[chavafg]

I tried to update CRI-O to unblock our CI [1], but unfortunately I got this error from the runtime:
Logs can be found on [2].

# time="2017-09-13 21:46:04.038222562Z" level=debug msg="running conmon: /home/runner/workspace/src/github.com/kubernetes-incubator/cri-o/conmon/conmon" args=[-c 9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8 -u 9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8 -r /usr/local/bin/cc-runtime -b /tmp/tmp.ACHvv06bWw/crio-run/aufs-containers/9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8/userdata -p /tmp/tmp.ACHvv06bWw/crio-run/aufs-containers/9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8/userdata/pidfile -l /var/log/crio/pods/9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8/9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8.log --exit-dir /var/run/crio/exits] 
# time="2017-09-13 21:46:05.000801369Z" level=debug msg="Received container pid: -1" 
# time="2017-09-13 21:46:05.000828124Z" level=debug msg="Container creation error: cgroupsPath "/Burstable/pod_123-456/crio-9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8" is absolute, cgroup mount MUST exist
# " 
# time="2017-09-13 21:46:06.514058420Z" level=info msg="Got pod network {Name:podsandbox1 Namespace:redhat.test.crio ID:9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8 NetNS:/var/run/netns/k8s_podsandbox1_redhat.test.crio_redhat-test-crio_1-fff0d780 PortMappings:[]}" 
# time="2017-09-13 21:46:06.514090805Z" level=info msg="About to del CNI network crionet (type=bridge)" 
# Creating the pod sandbox failed: rpc error: code = 2 desc = container create failed: cgroupsPath "/Burstable/pod_123-456/crio-9d120eb5a6efb1b45740fc6b65d90adb466c8d597e9ef0253516eeb93aad42c8" is absolute, cgroup mount MUST exist
# rm: cannot remove ‘/tmp/tmp.ACHvv06bWw/crio/aufs’: Device or resource busy

CRI-O Log does not show anything.

[1] clearcontainers/tests#531
[2] http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/clear-containers-tests-azure-ubuntu-16-04/111/consoleText
https://semaphoreci.com/clearcontainers/tests/branches/pull-request-531/builds/1

[amshinde]

An initial look shows that the error is from our own runtime:

runtime/oci.go

Line 232 in 97aa6e6

return "", fmt.Errorf("cgroupsPath %q is absolute, cgroup mount MUST exist",

A cgroupsPath ("/Burstable/pod_123-456/crio-22ca6911d8830bead552e27c75065406a377155e2d3094e66fb1ce9832ef3f54") was passed in the spec file, but no cgroup mount was found in the list of mounts, hence the error. Could be crio is no longer passing cgroup mounts, but I need to look at this further.
The cgroup mount would typically look like

{
            "destination": "/sys/fs/cgroup",
            "type": "cgroup",
            "source": "cgroup",
            "options": [
                "nosuid",
                "noexec",
                "nodev",
                "relatime",
                "ro"
            ]
        }

@sameo Do we need to error out in case no cgroup mounts are found in the first place?
We could either not create the cgroup files altogether (which was being done for fixing a docker leak iirc)
or assume the path as "/sys/fs/cgroup" and proceed, similar to what is being done at

runtime/oci.go

Line 216 in 97aa6e6

return filepath.Join(cgroupsDirPath, resource, ociSpec.Linux.CgroupsPath), nil

[sameo]

@amshinde a couple of comments:

• According to the OCI spec, an absolute cgroup path should be interpreted as relative to the system cgroup mount point. So not having a cgroup mount point is a valid case.

• I don't think CRI-O ever passed a cgroup mount for pods, but only for pod containers.

So this:

	if !cgroupMountFound {
		if isPod {
			return "", fmt.Errorf("cgroupsPath %q is absolute, cgroup mount MUST exist",
				ociSpec.Linux.CgroupsPath)
		}

		// In case of container (CRI-O), if the mount point is not
		// provided, we assume this is a relative path.
		return filepath.Join(cgroupsDirPath, resource, ociSpec.Linux.CgroupsPath), nil
	}

is incorrect and should be:

	if !cgroupMountFound {
	        // According to the OCI spec, we should interpret absolute path as relative to the system
                //  cgroup mount point when there is no cgroup mount in the config.json      
		return filepath.Join(cgroupsDirPath, resource, ociSpec.Linux.CgroupsPath), nil
	}

[sameo]

cc @runcom