[autoscaler] Ray submit with docker doesn't rsync the script to the container

[vakker]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian GNU/Linux 9.8 (stretch) on GCP
• Ray installed from (source or binary): source (5bfcfa8)
• Ray version: master - 5bfcfa8 (0.7.0.dev0)
• Python version: 3.7
• Exact command to reproduce:

ray up -y minimal.yaml  # with a docker image of the master build of Ray
ray submit --docker minimal.yaml hello_ray.py

Describe the problem

If I run submit without --docker then the worker can execute the specified script from ~, but when --docker is set, it throws:

python: can't open file '/root/hello_ray.py': [Errno 2] No such file or directory

When the docker run is called in the initialization phase, there doesn't seem to be any attached volumes, so I guess a shared location between the host and the container should resolve this, but there could be better ways as well.

Source code / logs

I can include full logs if necessary.

[richardliaw]

Sorry for the late reply.

A good fix here would be to use docker cp if using submit (and implicitly rsync): https://docs.docker.com/engine/reference/commandline/cp/

I can probably get to this in a couple days, but if you're interested in a contributing a fix, I'd be more than glad to help fix up the PR and get it merged.

[vakker]

Thanks for looking into this. Sorry, I could do it probably next week, but definitely not in the next few days...

[richardliaw]

Oops, this should be addressed in #4403.

[1Reinier]

So what ended up being the fix for this?

[preeth1]

I am also looking for the fix to this.