Keyboard Interrupt of the dask-scheduler CLI results in tornado.util.TimeoutError

[bryanwweber]

What happened:

Running then cancelling with ^C the dask-scheduler CLI script gives me a tornado TimeoutError:

$ dask-scheduler
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -   Scheduler at:  tcp://192.168.4.130:8786
distributed.scheduler - INFO -   dashboard at:                     :8787
^Cdistributed.scheduler - INFO - End scheduler at 'tcp://192.168.4.130:8786'
Traceback (most recent call last):
  File "/Users/bryan/mambaforge/envs/parquet/bin/dask-scheduler", line 10, in <module>
    sys.exit(go())
  File "/Users/bryan/mambaforge/envs/parquet/lib/python3.9/site-packages/distributed/cli/dask_scheduler.py", line 217, in go
    main()
  File "/Users/bryan/mambaforge/envs/parquet/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/bryan/mambaforge/envs/parquet/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/bryan/mambaforge/envs/parquet/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/bryan/mambaforge/envs/parquet/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/bryan/mambaforge/envs/parquet/lib/python3.9/site-packages/distributed/cli/dask_scheduler.py", line 208, in main
    loop.run_sync(run)
  File "/Users/bryan/mambaforge/envs/parquet/lib/python3.9/site-packages/tornado/ioloop.py", line 529, in run_sync
    raise TimeoutError("Operation timed out after %s seconds" % timeout)
tornado.util.TimeoutError: Operation timed out after None seconds

What you expected to happen:

No error is raised when you cancel the process.

Environment:

• Dask version: dask-scheduler, version 2022.2.1
• Python version: 3.9
• Operating System: macOS
• Install method (conda, pip, source): conda

[graingert]

I think doing something like this should allow a clean shutdown:

def handle_sigint(*args, **kwargs):
    signal.signal(signal.SIGINT, old_handler)
    loop.call_soon_threadsafe(stop_event.wait)

old_handler = signal.signal(signal.SIGINT, handle_sigint)
stop_event = asyncio.Event()

async with scheduler(...) as s: 
    await stop_event.wait()