Watch code and automatically rebuild when something changes

[bfirsh]

Compose could watch your code and automatically kick off builds when something changes. This would mean:

• Code can be reloaded automatically regardless of language. This would also make code reloading work on compiled languages if that was part of the build process!
• Volumes are no longer needed in development
• Production server can be used in development (e.g. gunicorn can be used to serve Django apps in development)

It could either be an option to up (docker-compose up --watch), an option in the config file specifying directories to watch/ignore, or perhaps a separate command (docker-compose watch).

Thanks to @samalba for the suggestion!

[samalba]

👍

[marksteve]

👍

[sherzberg]

+1. If this is to be part of fig up, how about an opt in flag?

web:
  watch: 
   - *.py
   - *.css
   - *.coffee

[cameron]

👍

Where this gets documented, it's probably worth noting the following pattern that helps avoiding unnecessary package manager (pip/npm/bower/et al) action:

# adding your language deps before adding the rest of your source....
ADD requirements.txt /requirements.txt
RUN pip install -r requirements.txt

# ... will prevent changes to source incurring a re-install of language deps
ADD . /src

[KyleAMathews]

💯

[luebken]

+1

[mdaffin]

👍

[tptee]

👍

[davidpaulhunt]

It's been a few months, and I couldn't tell from the merge referenced above, was a "watch" ability integrated?

[aanand]

No, fig watch has not been implemented.

[aanand]

It's still on the table though - see ROADMAP.md

[bfirsh]

An idea from @fxdgear: instead of rebuilding, it could also be useful to just restart the container when the code changes. E.g. when you've mounted your code with a volume.

[aanand]

Now that we have a --no-build flag, perhaps the logic should be:

$ fig up                      # current behaviour
$ fig up --watch              # watch and rebuild + restart
$ fig up --watch --no-build   # watch and restart

That said, if we go down the #693 path, we'll have to revisit this.

[amjibaly]

👍

For this to be really awesome, there's got to be a way for docker build to accept or pass through some type of credentials without storing it in the image. I'm thinking private github modules but it really applies to any environment.

The following is a NodeJS-based example using my current version of "fig up --watch" which is just a nodemon-based script:

I run fig up --watch and the first time npm install runs because the package.json hasn't been cached yet. This needs to fetch private modules from github so it passes those through (maybe this is setup in my ~/.figconfig or something similar?) My .dockerignore has node_modules in it so those are never included in the build context.

Now, I still want to do things like npm link where I can change a common module and fig up --watch knows to reload. I currently do this by adding a service to my fig.yml and mounting it to my common "code" volume. I can't think of a better way to do this at the moment.

[docteurklein]

what about the right tool for the right job ?

wouldn't guard fit perfectly here ?

guard :shell do
  watch(/Dockerfile$/) { `fig build` }
  watch(/fig.yml$/) { `fig up -d` }
end

Anyway, it would be a nice addition.

[ches]

@amjibaly:

For this to be really awesome, there's got to be a way for docker build to accept or pass through some type of credentials without storing it in the image. I'm thinking private github modules but it really applies to any environment.
...
I run fig up --watch and the first time npm install runs because the package.json hasn't been cached yet. This needs to fetch private modules from github so it passes those through (maybe this is setup in my ~/.figconfig or something similar?)

This sounds out of scope for fig or docker build to me. Like in this example, if you're referencing private Git repos as NPM dependencies, you could use the [email protected]:me/myrepo.git style of remote and then Git would just defer to SSH for key authorization without NPM, docker build, or fig needing to know anything about it. Most SCMs work similarly with SSH, or at worst other underlying tools should likely check an environment variable or config file of their own. But maybe there's something else you had in mind that I'm not considering?

[amjibaly]

@ches I didn't realize docker build already supports this. I just gave it a shot and it doesn't seem to work, can you point me to the documentation? Here's the error I see:

npm ERR! git clone [email protected]:myorg/myproj.git Cloning into bare repository '/root/.npm/_git-remotes/git-github-com-myorg-myproj-git-9e8969c5'...
npm ERR! git clone [email protected]:myorg/myproj.git error: cannot run ssh: No such file or directory
npm ERR! git clone [email protected]:myorg/myproj.git fatal: unable to fork

In package.json I have:

"dependencies": { "myproj": "git+ssh://[email protected]:myorg/myproj.git#master" }

[dnephin]

@amjibaly looks like you need to install ssh in your container

@docteurklein I tend to agree, there are existing tools which would handle this pretty easily (the python watchdog package includes a cli called watchmedo which would also work).

[amjibaly]

@dnephin I installed ssh, now I'm getting this error:

Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.

Running npm install locally works of course.

[docteurklein]

@amjibaly you're totally off-topic, no?

[amjibaly]

@docteurklein yeah my bad. FYI @ches it turns out that docker build does not support forwarding ssh creds so my original point still stands: moby/moby#6396

[wiktor-obrebski]

wouldn't it be too slow to work with everyday programming process? rebuild take time, restart too.

[Starefossen]

Not entirely sure if this should be something Docker Compose should do! I for one have several other build steps in between of noticing a file change and ultimately restarting the application; such as compiling, transpiring, concatenation etc.

Restarting the application when a file changes is something that should be handled by developer tools inside of the container. I'd rather see better support for multiple Docker Compose environment configurations in order to better support a developer configuration which are different from a prod configuration to accommodate for restart on file-change.

[nathanleclaire]

I've actually done an implementation of a feature like this at one point (in figgo) and I'm -1 on fig watch. Even with a tiny little app it's just slow enough to be annoying, I don't think I would use it. It'd be easy enough to write a little program to wrap fig and incorporate this functionality that I don't think we'd stand much to gain by supporting it.

[sebglon]

+1

[pie6k]

I'd say it's not stale

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[ndeloof]

compose now offers option to run builds using the docker CLI builder, which can be configured to use BuildKit. As the later doesn't require to send the whole docker context to daemon before a build start, and offers advanced caching capabilities, it seems a good candidate so service images can be rebuilt on a regular basis.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[SuperSandro2000]

spam for stale bot

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[tonymj76]

i followed this thread from 2014 now to 2020 to find a clue on how to auto build microservices (Go) binary in dev when a file is change. am getting tired of running two many terminal so am looking into docker to see how it can help my life
Also looking at https://github.com/cortesi/modd

[Lukortech]

6 years into the request and I still cannot easily rebuild my docker without creating freaking volumes.
What is this.

[facundopadilla]

6 años después de la solicitud y todavía no puedo reconstruir fácilmente mi ventana acoplable sin crear malditos volúmenes.
Qué es esto.

2021 lol

[DarthGoon]

I found a way to do this, that I like.

I'm using inotifywait to watch the file system, and nohup to start the processes; In a dev tooling "layer" in my linux docker container.

I put this in my Dockerfile-

# DEV TOOLING
RUN apt-get update && apt-get install -y inotify-tools
COPY code_reloader.sh /
RUN chmod +x /code_reloader.sh
ENTRYPOINT ["/code_reloader.sh"]
#############

This is code_reloader.sh-

#!/bin/bash
set -x

reload() {
  nohup python /opt/manage.py runserver 0.0.0.0:5000 > /nohup.out &
  DEV_SERVER_PID=$!
  PROCESS_RELOADING=0
}

reload

inotifywait -m -r -e modify /opt | while read CHANGE_EVENT;
do
  if [ $PROCESS_RELOADING -ne 1 ];
    then
      PROCESS_RELOADING=1
      echo "#### $CHANGE_EVENT DETECTED, Reloading PID:$DEV_SERVER_PID... ####"
      kill $DEV_SERVER_PID
      reload
    fi
done

The nohup command in reload() is what would be the ENTRYPOINT in the production container's Dockerfile. I happen to be running a python service in this example.
BE SURE TO redirect the nohup output file if it's in the same directory as the one you're monitoring; or nohup writing to it's out file will trigger a reloading loop. I'm redirecting it to / in this example.

With this setup-- I mount the code directory into /opt in the docker-compose.yml; run docker-compose up; and just write code. The service reloads with changes.

I like this approach because I don't have to modify code, or install framework specific packages/CLI tools, to be able to live reload.
The docker-compose.yml parts-

version: "3.9"
services:
  sandbox:
    build: .
    volumes:
      - ./code:/opt
    ports:
      - "5000:5000"

[fiq]

A watch and rebuild would be a really handy feature. As per the previous comment, a common work around is mounting one's app over the immutable version you have baked into your image. I see this often and it defeats the intent of containerisation + introduces false confidence as the running image is not representative of a particular build.

[Mahoney]

Started playing with a wrapper script to add --watch here:
https://github.com/Mahoney/docker-compose-watch/blob/main/docker-compose

Immediate weaknesses:

• It just uses pwd rather than parsing the yaml to decide which contexts to use
• It’s pretty inefficient because it doesn’t know about .dockerignore - so a change to an ignored file results in the entire context being calculated and sent to the docker daemon, before docker then decides there’s nothing to do.

[patientplatypus6]

I'm surprised this is still not a feature. ndeloof is working on this for Docker v2, but this should be in v3 itself as well.
Here's the problem that I'm currently facing.
I'm developing an application using docker-compose v3.9 on mac - which means that I can't use inotif-tools, but have to use fswatch. Which is a bit frustrating, because if I eventually want to port this over to a linux container that means that I'll have to then switch to inotify-tools.
OK fine, minor issue.
However
Watching for file changes means that I spawn a new process - so if I docker-compose up and <watch for file changes> in the same bash file then a new process is spawned, meaning I can't restart the container in a process that doesn't have the container running! So, that leaves me with the option of writing entrypoint scripts for each of the containers - but look at this example

This is the entrypoint.sh file for a backend Django/Python application -

#!/bin/bash

echo "inside entrypoint"
echo "deleting prior static images directory"
rm -rf ./static_images
echo "makemigrations"
python3 manage.py makemigrations
python3 manage.py makemigrations lightone
echo "migrate"
python3 manage.py migrate
python3 manage.py migrate lightone

echo "manage.py createsuperuser"
python3 manage.py createsuperuser \
  --noinput \
  --username pixel \
  --email [email protected]
# --database lightchan

echo "now running the server"
daphne -b 0.0.0.0 -p 8000 lightchan.asgi:application
echo "now running the server"

The container itself is alpine linux so I could use a file watch like inotify-tools. The issue now is that inotify-tools is blocking! So how do I simultaneously run Daphne while watching for file changes? Maybe I could run Daphne as a background process using a trailing &, but again, does that mean that Daphne will pick up file changes if inotify-tools kills the container and then the script restarts?

This is a major major issue. Will somebody fix this?

[ndeloof]

ndeloof is working on this for Docker v2, but this should be in v3 itself as well.
There's a (common) confusion here between the file format (which is now obsolete with Compose Specification) and the tool version. "Docker Compose v2" is docker compose command developed on this repository as v2 branch.

I expected inotify was working as expected under Docker Desktop, will need to change. A potential implementation of watch command for application stacks which support reloading could be for docker compose watch command to send a SIGHUP signal.

[aradalvand]

No updates on this after 8 years?!

[Lukortech]

Sadly, no.
I've given up on docker desktop for now.

[ydennisy]

@ndeloof any news on your PR?

[nbx0]

+1!

[ndeloof]

The experimental watch command has been released, closing this issue now

[ramnes]

Here is more context for those wondering:

• [ENV-44] introduce experimental watch command (skeletton) #10163
• mark alpha command as experimental #10170
• Tilt watcher #10218
• use CGO to enable fsevent on OSX [ENV-37] #10249
• [BUG] watch: can't watch a service without a build section #10270
• watch: ignore ephemeral files & minor output tweaks #10311

TL;DR: docker compose alpha watch on HEAD