Check collection job equivalence by query, not batch ID. #1878
Conversation
This allows current-batch requests to be reissued in fixed-size tasks, simplifying implementation of a Collector. While I'm at it, I also update `get_collection_job` to take a task ID in addition to a collection job ID. This provides isolation between different tasks, which is especially imporant now that the Collector generates the collection job IDs -- otherwise, it would be trivial for Collectors to generate the same collection job ID for two tasks, which would cause some very strange behavior in Janus.
branlwyd
added
the
allow-changed-migrations
There was a problem hiding this comment.
While I'm at it, I also update get_collection_job to take a task ID in addition to a collection job ID. This provides isolation between different tasks, which is especially imporant now that the Collector generates the collection job IDs -- otherwise, it would be trivial for Collectors to generate the same collection job ID for two tasks, which would cause some very strange behavior in Janus.
I imagine this should be backported to our other active branches?
Yes, I think this change should be backported. The batch-equivalence check should probably also be backported; we must backport it if we want to implement #1860 (comment) (since implementing that requires knowing the query that was used to generate a collection job). |
This allows current-batch requests to be reissued in fixed-size tasks, simplifying implementation of a Collector & avoiding the potential for loss of a collection job due to an inopportunely-timed task failure.
While I'm at it, I also update
get_collection_jobto take a task ID in addition to a collection job ID. This provides isolation between different tasks, which is especially imporant now that the Collector generates the collection job IDs -- otherwise, it would be trivial for Collectors to generate the same collection job ID for two tasks, which would cause some very strange behavior in Janus.Resolves #1877. Part of #1860.