c8y-firmware-plugin timer actor does not react to a shutdown message

[reubenmiller]

Describe the bug

The c8y-firmware-plugin does not restart cleanly (e.g. less than 60 seconds) when a child firmware operation is in progress.

The logs seem to indicate that the Timer actor does not react to the shutdown message. All other actors within the c8y-firmware-plugin actors respond (e.g. HealthMonitorActor, C8YJwtRetriever, MQTT, Downloader, Signal-Handler and FirmwareManager)

To Reproduce

Given the following setup

• Main device running tedge
• Child device with support for the c8y_Firmware operation

Then

1. Create a Cumulocity firmware operation for the child device

2. Whilst the operation is processing, restart the firmware plugin

   systemctl restart c8y-firmware-plugin

   Note: The restart command will take 60 seconds, and the following log message is shown in the journald logs (e.g. journalctl -fu c8y-firmware-plugin -n 100)

   ERROR Runtime: Timeout waiting for all actors to shutdown
   

Expected behavior

The Timer actor should listen to the shutdown message and stop the timer action so that it the actor Runtime does not need to forcefully shutdown the process.

Screenshots

Below shows the log output of the c8y-firmware-plugin after the systemctl restart c8y-firmware-plugin command was executed.

Apr 27 11:17:04 1090b1469950 systemd[1]: Stopping Thin-edge device firmware management for Cumulocity...
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485696536Z  INFO Signal-Handler: send Shutdown
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485792703Z  INFO Runtime: Shutting down
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485810953Z  INFO Runtime: Successfully sent shutdown request to HealthMonitorActor-6
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485822536Z  INFO Runtime: Successfully sent shutdown request to C8YJwtRetriever-2
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485830536Z  INFO Runtime: Successfully sent shutdown request to MQTT-1
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485836078Z  INFO Runtime: Successfully sent shutdown request to Downloader-3
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485845536Z  INFO Runtime: Successfully sent shutdown request to Signal-Handler-0
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485850953Z  INFO Runtime: Successfully sent shutdown request to FirmwareManager-4
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485855994Z  INFO Runtime: Successfully sent shutdown request to Timer-5
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485875411Z  INFO Timer: recv None
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485891078Z  INFO c8y-firmware-plugin: recv None
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485932203Z  INFO Runtime: Actor has finished: HealthMonitorActor-6
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485943411Z  INFO C8YJwtRetriever: recv None
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485964536Z  INFO Runtime: Actor has finished: C8YJwtRetriever-2
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.485973078Z  INFO MQTT: recv Err(Shutdown)
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.486000286Z  INFO Runtime: Actor has finished: MQTT-1
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.486008453Z  INFO Downloader: recv None
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.486017828Z  INFO Runtime: Actor has finished: Downloader-3
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.486023244Z  INFO Signal-Handler: recv None
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.486142703Z  INFO Runtime: Actor has finished: Signal-Handler-0
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.486156119Z  INFO C8Y-Firmware-Manager: recv None
Apr 27 11:17:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:17:04.486184328Z  INFO Runtime: Actor has finished: FirmwareManager-4
Apr 27 11:18:04 1090b1469950 c8y-firmware-plugin[60]: 2023-04-27T11:18:04.491625023Z ERROR Runtime: Timeout waiting for all actors to shutdown
Apr 27 11:18:04 1090b1469950 systemd[1]: c8y-firmware-plugin.service: Succeeded.

Environment (please complete the following information):

• OS [incl. version]: Debian 11
• Hardware [incl. revision]: Docker container
• System-Architecture [e.g. result of "uname -a"]: aarch64
• thin-edge.io version [e.g. 0.1.0]: 0.10.0-175-g79566a55

Additional context