TimedRotatingFileHandler: reliable algorithm to determine the files to delete

[serhiy-storchaka]

Bug report

There were many issues related to removing old files in TimedRotatingFileHandler:

• bpo-44753/backupCount is not respected in TimedRotatingFileHandler when namer is specified #88916
• bpo-45628/TimedRotatingFileHandler backupCount not working #89791
• bpo-46063/TimedRotatingFileHandler deletes wrong files #90221
• Multiple TimedRotatingFileHandler with similar names but different backup counts do not work #93205

It is complicated because the namer attribute allows arbitrary transformation of the file name.

1. Multiple TimedRotatingFileHandler with similar names but different backup counts do not work #93205 almost nailed the case without namer. But when there are two handlers with the same basename, and one has no namer, while other has a simple namer that adds an extension, the former handler will delete logs of the latter one. This is because the regular expression extMatch includes an optional extension. It is a remnant from former attempts to handle rotation, and is not needed now. The issue can be fixed by removing it.

2. It is even more complicated when we have more complex namer. It can add more complex suffix, it can modify the prefix. We can only expect that it leaves the datetime part unchanged. I think that the most reliable way to find candidates for deletion is to find the datetime part in the file, then generate a new filename for this datetime and compare it with the original filename. It should work for arbitrary deterministic namer that does not modify the datetime part, and should not produce false positives.

Linked PRs

• gh-115809: Improve TimedRotatingFileHandler.getFilesToDelete() #115812
• [3.12] gh-115809: Improve TimedRotatingFileHandler.getFilesToDelete() (GH-115812) #116261
• [3.11] gh-115809: Improve TimedRotatingFileHandler.getFilesToDelete() (GH-115812) #116262