Merge pull request #36 from kiwix/improve-scheduler-docs

improve documentation of scheduler settings

dev/README.md

@@ -29,10 +29,12 @@ This container is responsible for starting the task containers, setting up a wir
 ```sh
 docker compose up --build
 ```
+
 The above command only starts the `backend`, `postgresqldb` and `scheduler`
 services.
 
 To start the `worker-manager`, [you need to register a worker](#registering-a-worker). Afterwards, run:
+
 ```sh
 docker compose --profile worker up --build
 ```
@@ -42,40 +44,43 @@ docker compose --profile worker up --build
 - [Get wireguard configuration files](#get-wireguard-configuration-files)
 - [Create a test worker](#create-a-test-worker)
 
-
 **NOTE:** Unless otherwise stated, all files and commands are with respective to the `dev` directory.
 
-
 ### Get wireguard configuration files
 
--  Move the [Mullvad](https://mullvad.net/) configuration files into the `data` directory.
-    Configuration files should be named in the format `<country_code>.conf` where
-    `<country_code>` is the ISO 3166-1 alpha-2 code of the country.
-    On start, the `worker-manager` will select a random file to start up the `wireguard` container.
-    On receiving a task from the backend scheduler, it will search for a configuration file belonging to that task and reconfigure the wireguard container.
-    If the configuration file does not exist, it will skip the test.
+- Move the [Mullvad](https://mullvad.net/) configuration files into the `data` directory.
+  Configuration files should be named in the format `<country_code>.conf` where
+  `<country_code>` is the ISO 3166-1 alpha-2 code of the country.
+  On start, the `worker-manager` will select a random file to start up the `wireguard` container.
+  On receiving a task from the backend scheduler, it will search for a configuration file belonging to that task and reconfigure the wireguard container.
+  If the configuration file does not exist, it will skip the test.
 
 ## create a test worker
 
 - Generate a private key:
-    ```sh
-    openssl genrsa -out id_rsa 2048
-    ```
-    The key name `id_rsa` is used as a bind mount in the compose file for the worker container.
+
+  ```sh
+  openssl genrsa -out id_rsa 2048
+  ```
+
+  The key name `id_rsa` is used as a bind mount in the compose file for the worker container.
 
 - Generate a public key for creating the worker on the database.
-    ```sh
-    openssl rsa -in id_rsa -pubout -out pubkey.pem
-    ```
+
+  ```sh
+  openssl rsa -in id_rsa -pubout -out pubkey.pem
+  ```
 
 - Assuming the backend service is up (`docker compose up backend`), create a worker and assign them a list of countries to test for.
-    If no countries are provided, all available countries in the DB wiil be assigned to the worker. You can update the countries using `mirrors-qa-backend update-worker`.
+  If no countries are provided, all available countries in the DB wiil be assigned to the worker. You can update the countries using `mirrors-qa-backend update-worker`.
+
+  In this example, we create a worker named `test` to test for mirrors in France, United States and Canada using the public key file
+  named `pubkey.pem`.
+
+  ```sh
+  docker exec -i mirrors-qa-backend mirrors-qa-backend create-worker --countries=us,fr,ca test < ./pubkey.pem
+  ```
 
-    In this example, we create a worker named `test` to test for mirrors in France, United States and Canada using the public key file
-    named `pubkey.pem`.
-    ```sh
-    docker exec -i mirrors-qa-backend mirrors-qa-backend create-worker --countries=us,fr,ca test < ./pubkey.pem
-    ```
 - Set the name of the worker to the `WORKER_ID` variable in the `.env` file.
 
 - Start the services with the worker enabled using `docker compose --profile worker up --build`
@@ -103,12 +108,14 @@ The `backend` code houses the `scheduler` and the `RESTful API`. The following e
 ### scheduler
 
 - `SCHEDULER_SLEEP_DURATION`: how long the scheduler should sleep after creating tests for idle workers
-- `IDLE_WORKER_DURATION`: mark a worker as idle if it hasn't been seen within duration
+- `IDLE_WORKER_DURATION`: duration since a worker was last seen to be considered "idle". Generally,
+  we want to set this to the same value as `SCHEDULER_SLEEP_DURATION` since a worker that hasn't
+  submitted tests throughout `SCHEDULER_SLEEP_DURATION` is idle.
 - `EXPIRE_TEST_DURATION`: expire tests whose results are still pending after duration
 
 ### worker-manager
 
-- `SLEEP_DURATION`: how long the manager should sleep before polling the REST API
+- `SLEEP_DURATION`: how long the manager should sleep before polling the REST API for pending tests.
 - `BACKEND_API_URI`
 - `DOCKER_SOCKET`
 - `PRIVATE_KEY_FILE`: name of private key file

dev/docker-compose.yaml

@@ -44,8 +44,8 @@ services:
     container_name: mirrors-qa-scheduler
     environment:
       POSTGRES_URI: postgresql+psycopg://mirrors_qa:mirrors_qa@postgresdb:5432/mirrors_qa
-      IDLE_WORKER_DURATION: 10m
-      SCHEDULER_SLEEP_DURATION: 30m
+      IDLE_WORKER_DURATION: 15m
+      SCHEDULER_SLEEP_DURATION: 15m
       EXPIRE_TEST_DURATION: 1h
       DEBUG: true
     command: mirrors-qa-backend scheduler