Clear old PollData

[hankquan]

Current behavior

PollData(worker info) is not deleted even the worker instance dead for a long time, and maybe the worker instance will never come back, but the PollData item is still there. Eg: In kubernetes, if the worker pod deleted and created again, the workId(hostname as default) changes everytime.

Proposals

Alternative 1

Provide a scheduled job to clean old PollData by lastUpdateTime, make the workers for a queue freshly.

pros: Support for all kind of persistences
cons: Require extra thread for cleaning

Alternative 2

For redis persistence, we can expire the PollData key in a TTL.

pros: It is very efficiency and no extra cost
cons: It can be implemented easily in redis, but for mysql etc, no idea

Alternative 3

Provide API to delete PollData.

[apanicker-nflx]

For Conductor, alternative 1 makes the most sense. However, in the interest of time/effort investment, you could choose to implement it as alternative 2.

The PollDataDAO is a separate interface and can be implemented in any persistence configuration. You can add a method to this DAO to clear out old poll data. Each persistence layer can then implement the cleanup per their choice. For redis-peristence, you could implement this as just setting the TTL for poll data records.

Alternative 3 does not sound appealing to me since we would be leaking out implementation details from the application and would require an external process to handle the cleanup.

Please let me know your thoughts

[hankquan]

Hi @apanicker-nflx , thanks for your reply. I'd like to take your advice and try redis-persistence firstly.