/bin/bash: .command.sh: No such file or directory

[huguesfontenelle]

Edit:

Resolution
The error happens when the run folder is not in /Users. docker-machine only mounts /Users (see comment)

---

Original post:

All examples from https://github.com/nextflow-io/examples run fine, but not when using -with-docker.

I'm using OSX 10.9.5.

$ nextflow run helloworld.nf -with-docker
N E X T F L O W  ~  version 0.15.3
Launching helloworld.nf
[warm up] executor > local
[96/e577c9] Submitted process > printHello (2)
[c9/81db50] Submitted process > printHello (1)
[d5/c9b4a2] Submitted process > printHello (3)
[9e/b6af1f] Submitted process > printHello (4)
Error executing process > 'printHello (3)'

Caused by:
  Process 'printHello (3)' terminated with an error exit status

Command executed:

  echo bonjour

Command exit status:
  1

Command output:
  (empty)

Command error:
  /bin/bash: .command.sh: No such file or directory

Work dir:
  /DEVEL/CLONES/nextflow-io/examples/work/d5/c9b4a21f3b58d4c97e933ae31f6bb5

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named: '.command.sh'

WARN: Killing pending processes (3)

I installed Docker through brew cask:

$ brew cask info dockertoolbox
dockertoolbox: 1.8.2a
Docker Toolbox
https://www.docker.com/toolbox
/opt/homebrew-cask/Caskroom/dockertoolbox/1.8.2a (178M)
https://github.com/caskroom/homebrew-cask/blob/master/Casks/dockertoolbox.rb
==> Contents
  DockerToolbox-1.8.2a.pkg (pkg)

Some additional info:

$ docker info
Containers: 3
Images: 49
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 55
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.2 (TCL 6.4); master : aba6192 - Thu Sep 10 20:58:17 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: default
ID: HWDD:YHXG:V7J3:CAKH:ZHU2:AQ2N:A6JM:XKEO:F5SI:44KH:S7OW:DDMR
Debug mode (server): true
File Descriptors: 11
Goroutines: 18
System Time: 2015-09-22T13:38:29.984827117Z
EventsListeners: 0
Init SHA1: 
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

Note that there has been a very recent version for Docker that runs on MacOS without resorting to boot2docker (see how-to-use-docker-on-os-x-the-missing-guide), perhaps this is useful info.

[mfoll]

I'm using 10.10.5 and it's working ok with docker installed using the recent version you mention:

bash-3.2$ nextflow run helloworld -with-docker nextflow/examples
N E X T F L O W  ~  version 0.15.3
Launching 'nextflow-io/helloworld' - revision: 35f898dfe5 [master]
[warm up] executor > local
[4c/6141cb] Submitted process > sayHello (5)
[92/6a7ec7] Submitted process > sayHello (1)
[a1/51b7dc] Submitted process > sayHello (2)
[ad/455ccd] Submitted process > sayHello (4)
[a1/44468f] Submitted process > sayHello (3)
Hello world!
Γεια σου world!
Bojour world!
Ciao world!
Hola world!

And some info:

bash-3.2$ docker -info
flag provided but not defined: -info
See 'docker --help'.
bash-3.2$ docker info
Containers: 2
Images: 33
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 37
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: default
ID: KITL:ILHZ:DZWK:MXUP:TS2N:7RKW:6OAU:SGA6:S2WL:2RII:4LFW:D7FD
Debug mode (server): true
File Descriptors: 11
Goroutines: 21
System Time: 2015-09-22T13:54:45.87807142Z
EventsListeners: 0
Init SHA1: 
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

[pditommaso]

It looks to me, that your docker installation is not mounting the hosting (Mac) file system.

You can verify that running running this command:

docker run -v $PWD:$PWD -w $PWD nextflow/examples bash -c 'ls -la'

It should list the content of the directory where you have executed it.

[huguesfontenelle]

$ mkdir boo
$ cd boo/
$ docker run -v $PWD:$PWD -w $PWD nextflow/examples bash -c 'ls -la'
total 4
drwxr-xr-x 2 root root   40 Sep 22 16:15 .
drwxr-xr-x 3 root root 4096 Sep 22 16:15 ..

What I supposed to see?

My apologies if this is a Docker issue and not a Nextflow one, I'm new to both, but help is greatly appreciated.

[pditommaso]

Wait, this makes sense because you have just created a new empty dir.

Try this:

$ rm -rf boo 
$ mkdir boo && cd boo/
$ touch hello 
$ docker run -v $PWD:$PWD -w $PWD nextflow/examples bash -c 'ls -la'

It should list hello

[huguesfontenelle]

It did not list hello :-(
So you're right about something :-)

... that my docker installation is not mounting the hosting (Mac) file system.
(And I have no idea why or how to solve this)

[pditommaso]

Unfortunately I have no experience with dockertoolbox. I'm (still using) boot2docker to run Docker on mac.

You can try to get help on the Docker forum. Alternatively you can uninstall the dockertoolbox and install docker and boot2docker in your mac.

Note however that Docker is not mandatory to use Nextflow. The above example can be execute without using it just entering this command:

$ nextflow run helloworld.nf

Docker is only required for the examples using Blast and T-Coffee if you do not have them installed in your computer.

[huguesfontenelle]

Thanks a lot.
Yes I'm running nextflow all fine, we've it in production in my medical genetics centre. Just that I have a Mac and the cluster is RedHat, so I thought that -with-docker would be perfect for testing locally.

I followed the "tip", changed dir to the working dir and ran bash .command.run which gave me a .command.sh: No such file or directory
Strange.

 $ nextflow run helloworld.nf -with-docker
N E X T F L O W  ~  version 0.15.3
Launching helloworld.nf
[warm up] executor > local
[50/dcb858] Submitted process > printHello (3)
[e4/bcc87d] Submitted process > printHello (2)
[3e/cce791] Submitted process > printHello (1)
[7f/d278e8] Submitted process > printHello (4)
Error executing process > 'printHello (1)'

Caused by:
  Process 'printHello (1)' terminated with an error exit status

Command executed:

  echo hello

Command exit status:
  1

Command output:
  (empty)

Command error:
  /bin/bash: .command.sh: No such file or directory

Work dir:
  /DEVEL/CLONES/nextflow-io/examples/work/3e/cce7913a431fdeecebc7f41daf58bc

Tip: you can replicate the issue by changing to the process work dir and entering the command: 'bash .command.run'

WARN: Killing pending processes (3)

$ cd /DEVEL/CLONES/nextflow-io/examples/work/3e/cce7913a431fdeecebc7f41 

$ bash .command.run
mkfifo: /DEVEL/CLONES/nextflow-io/examples/work/3e/cce7913a431fdeecebc7f41daf58bc/.command.po: File exists
mkfifo: /DEVEL/CLONES/nextflow-io/examples/work/3e/cce7913a431fdeecebc7f41daf58bc/.command.pe: File exists
/bin/bash: /DEVEL/CLONES/nextflow-io/examples/work/3e/cce7913a431fdeecebc7f41daf58bc/.command.sh: No such file or directory

$ ls -la
total 40
drwxr-xr-x  11 huguesfo  164037   374 Sep 22 22:04 .
drwxr-xr-x   4 huguesfo  164037   136 Sep 22 22:04 ..
-rw-r--r--   1 huguesfo  164037     0 Sep 22 22:05 .command.begin
-rw-r--r--   1 huguesfo  164037    59 Sep 22 22:04 .command.env
-rw-r--r--   1 huguesfo  164037   124 Sep 22 22:05 .command.err
-rw-r--r--   1 huguesfo  164037     0 Sep 22 22:05 .command.out
prw-r--r--   1 huguesfo  164037     0 Sep 22 22:05 .command.pe
prw-r--r--   1 huguesfo  164037     0 Sep 22 22:05 .command.po
-rw-r--r--   1 huguesfo  164037  1866 Sep 22 22:04 .command.run
-rw-r--r--   1 huguesfo  164037    27 Sep 22 22:04 .command.sh
-rw-r--r--   1 huguesfo  164037     1 Sep 22 22:05 .exitcode

[pditommaso]

This confirms that the problem is Docker, because when you executes .command.run it tries to run the task script i.e. .command.sh using Docker.

Since your Docker installation is not mounting the hosting file system it is unable to access that script.

I suppose that is an issue with the dockertoolbox that is quite a new tool. I would suggest to use boot2docker instead.

[pditommaso]

I'm closing this as it doesn't look to be a nextflow problem. Feel free to comment below if you need further help.

[emi80]

Hi Hugues,

I see from your first message your Nextflow work folder is /DEVEL/CLONES/nextflow-io/examples/work. I think that, by default, docker-machine only mounts /Users into the virtual machine. Try to run the Nextflow examples in a folder under /Users (your $HOME for example) and it should work.

If you use the Virtualbox provider you can easily check/change that by just opening Virtualbox and going to the SharedFolder tab under the Docker virtual machine settings.

Hope it helps!

[pditommaso]

You got it! Surely that's the problem.

[huguesfontenelle]

@emi80 you nailed it! Thank you so much and everyone else for helping out! @pditommaso I can't wait digging deeper into Nextflow!

In the meantime I had downgraded to VirtualBox 4.3.30r and Docker 1.8.2, but of course that did not help.

[gijzelaerr]

this happens with any remote configuration setup using docker-machine. I've setup a remote swarm controlled with docker machine on a cluster and was experimenting with nextflow, but getting scripts in and out (if i understand correctly) don't work. I think this will become a more used use case in scientific computing - firing jobs to a remote machine or cluster. getting data in and out of a remote docker container is a bit of an unsolved problem afaik, maybe a volume driver is the way to go.

[Nitin123-4]

I am still facing the same issue;

bash work/39/94a36d502ce72117d15d7ee42cfda6/.command.run

work/39/94a36d502ce72117d15d7ee42cfda6/.command.sh: No such file or directory
But this file is there :

Maybe : Since your Docker installation is not mounting the hosting file system it is unable to access that script.

Is there any solution for this?

[TomYipOracle]

I'm also seeing the error about .command.run not found. Inspected the container using this command:
docker run -i -t quay.io/nextflow/bash /bin/bash

and also inspected my local file system, and I cannot find anything like '.command.*'

This is what I tried and the error received:

$ nextflow run nextflow-io/hello
N E X T F L O W ~ version 21.04.3
Launching nextflow-io/hello [extravagant_ardinghelli] - revision: ec11eb0ec7 [master]
executor > k8s (4)
executor > k8s (4)
[d0/3884fc] process > sayHello (1) [ 25%] 1 of 4, failed: 1
Error executing process > 'sayHello (1)'

Caused by:
Process sayHello (1) terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

echo 'Bonjour world!'

Command exit status:

Command output:
(empty)

Command wrapper:
/bin/bash: .command.run: No such file or directory

Work dir:
/Users/tyip/nextflow/work/d0/3884fc356ce3a04cdff6a3b27ff4c5

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

I also tried another sample project, and got the same problem in the above.
$ nextflow run nextflow-io/rnaseq-nf

I did see pod created and terminated when I run the command, and I assume it's running with the nextflow.config that is stored in the project specified. Please bear with me if I'm missing something obvious as I'm fairly new to Nextflow, Kubernetes and Docker :)