Error in case of no space left on controller machine

[vschiavoni]

When the disk space of the controller machine is exhausted and the controller daemon cannot write logs on disk, the following messages are printed on screen:

E, [2015-07-14 11:43:37 #29518] ERROR -- : Errno::ENOSPC: No space left on device - /home/splayd/splay/src/controller/logs/32
/home/splayd/splay/src/controller/lib/logd.rb:196:in `write'
/home/splayd/splay/src/controller/lib/logd.rb:196:in `puts'
/home/splayd/splay/src/controller/lib/logd.rb:196:in `run'
/home/splayd/splay/src/controller/lib/logd.rb:129:in `initialize'
/home/splayd/splay/src/controller/lib/logd.rb:129:in `new'
/home/splayd/splay/src/controller/lib/logd.rb:129:in `run'
/home/splayd/splay/src/controller/lib/logd.rb:74:in `main'
/home/splayd/splay/src/controller/lib/logd.rb:60:in `loop'
/home/splayd/splay/src/controller/lib/logd.rb:60:in `main'
/home/splayd/splay/src/controller/lib/logd.rb:35:in `run'
/home/splayd/splay/src/controller/lib/logd.rb:34:in `initialize'
/home/splayd/splay/src/controller/lib/logd.rb:34:in `new'
/home/splayd/splay/src/controller/lib/logd.rb:34:in `run'
controller_fork.rb:55
controller_fork.rb:53:in `fork'
controller_fork.rb:53
controller_fork.rb:52:in `each'
controller_fork.rb:52

The controller itself does not crash, but we should maybe provide a less scary message to the administrator.

[vschiavoni]

In terms of usability, it might be also good to prevent further job submissions while the controller is in this state of 'reduced operability' mode. Jobs should be either refused or enqueued with some explanatory message sent back to the client.