Add an index to better support smaller_number_is_higher_priority

[mkrfowler]

The preexisting index (introduced in #726) gave a direct answer to the materialized subquery used to find candidate jobs, subject to the ordering rules that were in place at that time. In #883, GoodJob deprecated that ordering in favour of a lower-is-more-important scheme, aligning with Active Job, but the index was not updated at that time.

I observed a substantial improvement in job throughput on applying an equivalent index, in the context of a jobs table with some 10M+ completed jobs. This catches the index change mentioned in #1005.

Plan changes for the subquery (admittedly, not the most recent GoodJob version) before and after applying:

Before:

Limit  (cost=14528.90..14529.15 rows=100 width=44) (actual time=517.784..517.794 rows=70 loops=1)
  ->  Sort  (cost=14528.90..14532.43 rows=1415 width=44) (actual time=517.783..517.787 rows=70 loops=1)
        Sort Key: priority, created_at
        Sort Method: quicksort  Memory: 30kB
        ->  Bitmap Heap Scan on good_jobs  (cost=8951.83..14474.81 rows=1415 width=44) (actual time=111.465..517.719 rows=70 loops=1)
              Recheck Cond: (finished_at IS NULL)
              Rows Removed by Index Recheck: 1168042
              Filter: ((scheduled_at <= '2024-01-12 02:51:38.927646'::timestamp without time zone) OR (scheduled_at IS NULL))
              Rows Removed by Filter: 847
              Heap Blocks: exact=557 lossy=99411
              ->  Bitmap Index Scan on index_good_jobs_on_concurrency_key_when_unfinished  (cost=0.00..8951.48 rows=1415 width=0) (actual time=56.387..56.388 rows=552240 loops=1)
Planning Time: 1.276 ms
Execution Time: 518.545 ms

After (a different index name):

Limit  (cost=5.90..5.91 rows=1 width=44) (actual time=0.022..0.023 rows=0 loops=1)
  ->  Sort  (cost=5.90..5.91 rows=1 width=44) (actual time=0.021..0.022 rows=0 loops=1)
        Sort Key: priority, created_at
        Sort Method: quicksort  Memory: 25kB
        ->  Index Scan using index_good_jobs_jobs_on_priority_created_at_when_unfinished on good_jobs  (cost=0.12..5.89 rows=1 width=44) (actual time=0.004..0.004 rows=0 loops=1)
              Filter: ((scheduled_at <= '2024-01-12 02:51:38.927646'::timestamp without time zone) OR (scheduled_at IS NULL))
Planning Time: 0.668 ms
Execution Time: 0.041 ms

[bensheldon]

Thank you! This is excellent work and I'm impressed you navigated all of the various migration configuration! 🚀