Automatically reload REST server when file changes are detected

[yifanmai]

Reasons for making this change

The REST server will automatically reload when file changes are detected when the service is started in development mode i.e. ./codalab_service.py start -bd. This eliminates the need to restart the REST server manually.

Related issues

Fixes #1182
Fixes #4199

Checklist

• I've added a screenshot of the changes, if this is a frontend change
• I've added and/or updated tests, if this is a backend change
• I've run the pre-commit.sh script
• I've updated docs, if needed

[percyliang]

I remember that the reason we didn't activate this was that for the bundle manager, you would still have to manually restart. But I guess we can try to do autorestart for that too?

Are there any downsides of doing this?

[yifanmai]

Yes, the bundle manager still needs to be manually restarted; the updated docs does mention this.

The main disadvantage that when there's an error on worker startup (i.e. a Python exception when importing the modules, caused by a bug in the code), gunicorn shuts down entirely. gunicorn restarts automatically when the bug is fixed. This can be confusing for the developer because the website will appear to stop working, and they will have to look at Docker logs to find the exception. Other frameworks (i.e. Django, Rails, Play) keep the server running and return the exception in the browser instead. This seems to be an inherent limitation of gunicorn.

The other options I tried were:

• Setting reloader=True in bottle.run This reloads the gunicorn worker, but the code changes don't show for some reason.
• Setting --reload on the gunicorn command line - The code changes show up, but an error on worker startup causes gunicorn to shutdown permanently, which means the developer has to manually restart the server; in contrast, --watch restarts the server automatically after the error is fixed.

The other disadvantage is that restarting the REST server causes it to lose state (e.g. in-memory global variables, persistent connections), but it is better for the REST server to be stateless anyway.