Feature: improve debugging of COPY failures

[cgthayer]

Description

Runing docker build does not provide enough detail for debugging COPY failures. Requesting a feature to help debugging.

For example, this is a common issue as noted by 200+ upvotes on stackoverflow: https://stackoverflow.com/a/36028733/1865272

Steps to reproduce the issue:

mkdir tmp
cd tmp
echo '**/bin' > .dockerignore
mkdir bin
date > bin/date.out
echo 'FROM alpine
COPY bin/date.out foo' > Dockerfile
docker build .

Describe the results you received:
The output is:

Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM alpine
 ---> a24bb4013296
Step 2/2 : COPY bin/date.out foo
COPY failed: stat /var/lib/docker/tmp/docker-builder394713048/bin/date.out: no such file or directory

Note that the output from -D doesn't help either.

Describe the results you expected:

If copy fails to find anything, I'd like it to check without the ignore rules and emit something like:

"Warning: Your .dockerignore prevented these files from copying: bin/date.out"

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:46 2020
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13-beta2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.14
  Git commit:       ff3fbc9d55
  Built:            Mon Aug  3 15:06:50 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false
 Plugins:
  ecs: Docker ECS (Docker Inc., v1.0.0-beta.5)
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)

Server:
 Containers: 13
  Running: 1
  Paused: 0
  Stopped: 12
 Images: 1303
 Server Version: 19.03.13-beta2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.104-microsoft-standard
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 24.87GiB
 Name: docker-desktop
 ID: MWAP:FMYO:IGBY:JVHV:IY5P:LUJF:BUYE:ADYV:35TP:U3GW:TR2A:KV63
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 51
  Goroutines: 56
  System Time: 2020-08-27T00:52:06.6468469Z
  EventsListeners: 4
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

Windows 10 running WSL2 with Ubuntu 18.04

[thaJeztah]

Thanks for reporting; I recently made some minor changes to the "classic" builder to somewhat improve the error message (see moby/moby#41327); with those changes (not yet in a release) the same error would look like;

COPY failed: file not found in build context or excluded by .dockerignore: stat bin/date.out: file does not exist

So with that change, the obscure "/var/lib/docker/tmp/docker-builder394713048/" path (which is the temp-directory on the daemon side) is omitted, and the error makes a mention of .dockerignore (which could help users find the problem).

Unfortunately, detecting if the file is missing because it's excluded (through .dockerignore) or missing is not possible. I also opened a ticket in the BuildKit repository (BuildKit is the next generation builder, which can be enabled by setting DOCKER_BUILDKIT=1); moby/buildkit#1647

BuildKit will need a similar fix, but from the discussion on that ticket, it might be possible to detect that a file/directory was excluded by .dockerignore, so that a more useful message can be presented.

That said; there's other things to improve around .dockerignore; I collected some issues in an epic in moby/moby#40319.

[adeniyistephen]

@thaJeztah is this issue in DockerCli or would be in moby? if yes I would really love to work on it, Thanks.

[thaJeztah]

I think this will be in buildkit; moby/buildkit#1647

[adeniyistephen]

Are inputs and pointers available for it? if yes I would like to work on it.