Stop using Django development server in production

[abeedvisram]

Motivation

Move away from using the Django development server in production as it's not suitable, see issue #39

API changes

None

Implementation Notes

• Reordered Dockerfile to make slower changing steps appear higher up, e.g. system-level packages and python (virtual) environment packages
• Use of Supervisord to monitor web server process
• Use Gunicorn as the production WSGI
• Added in convenience scripts for running in dev/prod setup

Functional Tests

Not been able to verify that this runs locally, see issue #47

Screenshots/Logs

Will be updated once #47 has been resolved.

[cgreene]

I love the goal of this PR! 👍

[awm33]

@abeedvisram It looks like you included some changes from the other PR in cognoma_site/settings.py. Can you remove those changes in order to keep within the scope of this PR?

Currently, we're using a sharing a single core for this web process. Do we want to change gunicorn.conf workers = 3 to workers = 2? That may be a nitpick.

Other than that, looking good!

[abeedvisram]

Updated and tested locally using the "prod" settings of Supervisord + Gunicorn. Works for me when testing on localhost:8080, log outputs below.

Dev using Django development server

Attaching to coreservice_core_db_1, coreservice_core_1, coreservice_nginx_1
core_db_1  | LOG:  database system was interrupted; last known up at 2017-01-24 01:27:43 UTC
core_db_1  | LOG:  database system was not properly shut down; automatic recovery in progress
core_db_1  | LOG:  invalid record length at 0/1794960
core_db_1  | LOG:  redo is not required
core_db_1  | LOG:  MultiXact member wraparound protections are now enabled
core_db_1  | LOG:  database system is ready to accept connections
core_db_1  | LOG:  autovacuum launcher started
core_1     | + python manage.py migrate -v3 --no-input
core_1     | Operations to perform:
core_1     |   Apply all migrations: api, organisms, genes, contenttypes
core_1     | Running pre-migrate handlers for application contenttypes
core_1     | Running pre-migrate handlers for application rest_framework
core_1     | Running pre-migrate handlers for application api
core_1     | Running pre-migrate handlers for application organisms
core_1     | Running pre-migrate handlers for application genes
core_1     | Running migrations:
core_1     |   No migrations to apply.
core_1     |   Your models have changes that are not yet reflected in a migration, and so won't be applied.
core_1     |   Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
core_1     | Running post-migrate handlers for application contenttypes
core_1     | Running post-migrate handlers for application rest_framework
core_1     | Running post-migrate handlers for application api
core_1     | Running post-migrate handlers for application organisms
core_1     | Running post-migrate handlers for application genes
core_1     | + python manage.py runserver 0.0.0.0:8000
core_1     | Performing system checks...
core_1     | 
core_1     | System check identified no issues (0 silenced).
core_1     | January 24, 2017 - 01:29:06
core_1     | Django version 1.9.8, using settings 'cognoma_site.settings'
core_1     | Starting development server at http://0.0.0.0:8000/
core_1     | Quit the server with CONTROL-C.
core_1     | [24/Jan/2017 01:29:10] "GET /genes?format=json HTTP/1.0" 200 52
nginx_1    | 172.24.0.1 - - [24/Jan/2017:01:29:10 +0000] "GET /genes?format=json HTTP/1.1" 200 74 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"

Prod using Supervisord + Gunicorn

Starting coreservice_core_db_1
Recreating coreservice_core_1
Recreating coreservice_nginx_1
Attaching to coreservice_core_db_1, coreservice_core_1, coreservice_nginx_1
core_db_1  | LOG:  database system was interrupted; last known up at 2017-01-24 01:22:20 UTC
core_db_1  | LOG:  database system was not properly shut down; automatic recovery in progress
core_db_1  | LOG:  invalid record length at 0/17948F0
core_db_1  | LOG:  redo is not required
core_db_1  | LOG:  MultiXact member wraparound protections are now enabled
core_db_1  | LOG:  database system is ready to accept connections
core_db_1  | LOG:  autovacuum launcher started
core_1     | + supervisord -c /code/config/prod/supervisord.conf
core_1     | 2017-01-24 01:27:44,123 CRIT Supervisor running as root (no user in config file)
core_1     | 2017-01-24 01:27:44,126 INFO supervisord started with pid 8
core_1     | 2017-01-24 01:27:45,128 INFO spawned: 'core_service' with pid 11
core_1     | [2017-01-24 01:27:45 +0000] [11] [INFO] Starting gunicorn 19.6.0
core_1     | [2017-01-24 01:27:45 +0000] [11] [INFO] Listening at: http://0.0.0.0:8000 (11)
core_1     | [2017-01-24 01:27:45 +0000] [11] [INFO] Using worker: eventlet
core_1     | [2017-01-24 01:27:45 +0000] [14] [INFO] Booting worker with pid: 14
core_1     | [2017-01-24 01:27:45 +0000] [15] [INFO] Booting worker with pid: 15
core_1     | 2017-01-24 01:27:46,596 INFO success: core_service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
nginx_1    | 172.24.0.1 - - [24/Jan/2017:01:28:09 +0000] "GET /genes?format=json HTTP/1.1" 200 74 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"

Process running as "nobody" under Supervisord

root@86df599524e8:/code# ps -ef --forest
UID        PID  PPID  C STIME TTY          TIME CMD
root        17     0  0 01:32 ?        00:00:00 bash
root        23    17  0 01:32 ?        00:00:00  \_ ps -ef --forest
root         1     0  0 01:32 ?        00:00:00 /bin/sh -c "/code/scripts/run_prod.sh"
root         6     1  0 01:32 ?        00:00:00 /bin/bash -x /code/scripts/run_prod.sh
root         7     6  0 01:32 ?        00:00:00  \_ /usr/bin/python /usr/bin/supervisord -c /code/config/prod/supervisord.conf
nobody      10     7  1 01:32 ?        00:00:00      \_ /usr/local/bin/python3.5 /usr/local/bin/gunicorn cognoma_site.wsgi:application -c /code/config/prod/gunicorn.conf
nobody      13    10  1 01:32 ?        00:00:00          \_ /usr/local/bin/python3.5 /usr/local/bin/gunicorn cognoma_site.wsgi:application -c /code/config/prod/gunicorn.conf
nobody      14    10  0 01:32 ?        00:00:00          \_ /usr/local/bin/python3.5 /usr/local/bin/gunicorn cognoma_site.wsgi:application -c /code/config/prod/gunicorn.conf

[cgreene]

woo! great 👍