Podman loses env-var values on commit

[kofalt]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I'm using the nvidia-container-runtime-hook, and I notice that podman is losing environment variable information when I try to commit images.

Since the hook uses these to operate, you can't run any CUDA image you commit. I tried to use the -c ENV flag as a workaround, but I think that may have the same problem.

# Before
"CUDA_PKG_VERSION=9-1=9.1.85-1",
"NVIDIA_REQUIRE_CUDA=cuda>=9.1",

# After
"CUDA_PKG_VERSION=9-1",
"NVIDIA_REQUIRE_CUDA=cuda>",

Steps to reproduce the issue:

1. Run this:

podman run --name reproduce-bug -it docker.io/nvidia/cuda:9.1-devel true

podman commit reproduce-bug new-image

2. Observe that environment variable information is lost:

podman inspect docker.io/nvidia/cuda:9.1-devel | jq -S '.[].Config.Env' 
[
  "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "CUDA_VERSION=9.1.85",
  "CUDA_PKG_VERSION=9-1=9.1.85-1",
  "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
  "NVIDIA_VISIBLE_DEVICES=all",
  "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
  "NVIDIA_REQUIRE_CUDA=cuda>=9.1",
  "NCCL_VERSION=2.2.12",
  "LIBRARY_PATH=/usr/local/cuda/lib64/stubs"
]

podman inspect new-image | jq -S '.[].Config.Env' 
[
  "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "TERM=xterm",
  "HOSTNAME=",
  "container=podman",
  "CUDA_VERSION=9.1.85",
  "CUDA_PKG_VERSION=9-1",
  "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
  "LIBRARY_PATH=/usr/local/cuda/lib64/stubs",
  "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
  "NVIDIA_VISIBLE_DEVICES=all",
  "NVIDIA_REQUIRE_CUDA=cuda>",
  "NCCL_VERSION=2.2.12"
]

3. This means you can't run your new image:

podman run -it  new-image bash

Error: container create failed: time="2019-05-16T05:03:57-05:00" level=warning msg="signal: killed"
time="2019-05-16T05:03:57-05:00" level=error msg="container_linux.go:346: starting container process caused \"process_linux.go:446: container init caused \\\"process_linux.go:429: running prestart hook 0 caused \\\\\\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda> --pid=28668 /var/lib/containers/storage/overlay/46aa6148dbc8d37673bb1167009035007e009c71aa3275bc072bc2f0118023d5/merged]\\\\\\\\nnvidia-container-cli: requirement error: invalid expression\\\\\\\\n\\\\\\\"\\\"\"\n"
container_linux.go:346: starting container process caused "process_linux.go:446: container init caused \"process_linux.go:429: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda> --pid=28668 /var/lib/containers/storage/overlay/46aa6148dbc8d37673bb1167009035007e009c71aa3275bc072bc2f0118023d5/merged]\\\\nnvidia-container-cli: requirement error: invalid expression\\\\n\\\"\""
: internal libpod error

Output of podman version:

Version:            1.3.0-dev
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.3.0-dev
host:
  BuildahVersion: 1.8-dev
  Conmon:
    package: 'conmon: /usr/libexec/crio/conmon'
    path: /usr/libexec/crio/conmon
    version: 'conmon version , commit: '
  Distribution:
    distribution: linuxmint
    version: "19.1"
  MemFree: 2020462592
  MemTotal: 16714416128
  OCIRuntime:
    package: 'cri-o-runc: /usr/bin/runc'
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 16664227840
  SwapTotal: 17075007488
  arch: amd64
  cpus: 8
  hostname: blade
  kernel: 4.15.0-50-generic
  os: linux
  rootless: false
  uptime: 18h 10m 52.55s (Approximately 0.75 days)
insecure registries:
  registries: []
registries:
  registries: []
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 15
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 20
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical

[TomSweeneyRedHat]

I'm going to guess that we're splitting environment variables at an equal sign (=) somewhere and accidentally dropping stuff if an equal sign is part of the value in the environment variable.

[TomSweeneyRedHat]

Thanks for the issue @kofalt !