Enable Production Mode via DEBUG or docker-compose.ci.yml

[KevinMind]

Relates to: mozilla/addons#14972

Description

This PR achieves 2 goals.

1. Makes running addons-server in DEBUG=false i.e. "production" mode possible
2. Fixes existing issues with serving static files in development mode

The specific changes include:

• respecting host defined DEBUG variable to run the container in. when debug true enable serving static files from ./static directory mounted from host as well as from django static server to load dependency assets.
• modifying the way we mount static files to ensure that development mode does not require subsequent calls to update_assets in order to run addons-server and that production mode contains necessary and collected static files from site-static.
• using a package django-node-assets to load node_modules as static assets directly from our node_modules directory. This simplifies our setup code, and like the previous step ensures we do not need additional commands to run the containers in dev mode.
• move assets that we "build" ourselves to a separate directory, that is added to STATICFILES_DIRS. The allows us to freely mount ./static in development without overriding built assets.

Context

Why not just mount ./static and be done with it? A few problems

• during our docker build we copy ./static but then we modify and add files. If you mount ./static again after building, the host files (which do not have the "build" assets) get removed.. so you end up back at the beginning.
• some files are available in ./static and some are in django dependencies or npm dependencies. We were using some complex scripting and mounting to make them available, but this can be simplified for development where we don't need collected static assets.

Why not just use runserver or serve for all assest

We use uwsgi to run django even in development because this more closely mirrors how our app runs in prod. We also cannot always rely on static_serve because in DEBUG=False (production mode) this will either not work, or even if we force it, it will be missing files.

The simpler solution is to

1. load files from whatever is in nginx locally
2. fallback to a request to addons-server static serve.

In production mode, 1 will have all the files and so it's fine that 2 will always fail.
In development mode, 1 will have most of the files from ./static which is fast and any files that are missing will be served from serve_static.

Testing

You should run several different iterations of addons-server and verify each is able to serve the expected static assets. This can be verified by running:

• login flow
• devhub (just load pages, it should all work)
• admin (/admin page should load)
• swagger (/api/v5/swagger. page should load)

Each scenario is toggling whether we target development or production dependencies, and whether to run in DEBUG mode or not. These are independent features and can be toggled accordingly.

Additionally you can set DEBUG directly in make commands by adding it as an argument, but the value is by default set to true and when docker-compose.ci.yml is included in the project the value is hard coded to "DEBUG=" regardless of the value set in the host environment or arguments.

IMPORTANT: before running each test you should "make down" to clear any image/volume artifacts, make sure to use exact versions since you are changing a lot between each run. After running make up, you should run "make reindex" to repopulate elasticsearch indexes as they are cleared when you make down and currently not repopulated automatically

1. Development/DEBUG=True

make up DOCKER_VERSION=local DOCKER_TARGET=development COMPOSE_FILE=docker-compose.yml

2. Development/DEBUG=

make up DOCKER_VERSION=local DOCKER_TARGET=development COMPOSE_FILE=docker-compose.yml:docker-compose.ci.yml

3. Production/DEBUG=True

make up DOCKER_VERSION=local DOCKER_TARGET=production COMPOSE_FILE=docker-compose.yml

4. Production/DEBUG=

make up DOCKER_VERSION=local DOCKER_TARGET=production COMPOSE_FILE=docker-compose.yml:docker-compose.ci.yml

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[diox]

As mentioned on slack, we need a way for assets to load using DEBUG=False. SERVE_STATIC variable is a good start but not enough: At the moment, with this patch, if I've got DEBUG=False set, then the URLs we are going to be referencing are going to be the ones with the hash that are in site-static/, but serve() only knows how to serve the source stuff from static/, not the built ones.

With DEBUG=False and SERVE_STATIC=True http://olympia.test/static/img/favicon.3cec07462a3d.ico fails for me for instance (and all js/css/images as well obviously).

We need some way of handling that scenario (where the developer wants a configuration closer to production - with "real" compiled assets, while still being able to develop locally, either to build some feature/fix some bug that requires real assets, or to reproduce an issue with production).

[KevinMind]

@diox it is ready for review and QA. Note, you either need a stage account for fxa or you need to force use_fake_fxa to return true for the 2nd and 4th case where we use docker-compose.ci.yml as this sets DEBUG= and will trigger real auth.

[diox]

This is confusing me - Why not keep /srv/static (and keeping the old alias directive intact in addons.conf) ?

[KevinMind]

we mount either "./static" or "/data/olympia/site-static" depending on which set of compose files you use.

As far as nginx is concerned it serves from one location but the source of the files changes.