Implement In-Memory Priority Queue for Auto-Analyze

[Rustin170506]

Overview

Implement a new priority queue that maintains a heap in memory to improve performance and reduce the overhead of rebuilding the queue for auto-analyze operations.

Goals

• Avoid scanning the table information every time
• Reflect DDL changes in the priority queue
• Ensure DDL events are received with correct reliability and order

Tasks

1. Implement basic heap structure with thread-safe operations
2. Implement DML change handling
   • Incremental updates from stats cache
   • Score recalculation based on modify_count and count changes
3. Implement DDL change handling
   • Create mysql.ddl_notifier table
   • Implement DDLNotifier for fetching and delivering DDL events
   • Implement subscription mechanism for DDL events
4. Implement time-based updates for analysis intervals
5. Implement failover handling
   • Priority queue rebuild after node restart
   • Priority queue rebuild after owner switch
6. Implement test suite
   • Functional tests
   • Scenario tests
   • Benchmark tests

Performance Considerations

• Optimize memory usage of in-memory priority queue
• Benchmark performance with large number of tables (1 million+)
• Measure performance of handling large number of DDL events
• Optimize priority queue rebuild process

Documentation

• Update design document with final implementation details
• Create user documentation for new auto-analyze behavior
• Update developer documentation with new priority queue implementation

Related PRs

• Design Document PR: docs: Add maintaining the priority queue in memory design #55903

Unresolved Questions

• Determine most efficient method for looking up table IDs with updates (query mysql.stats_meta vs. querying stats cache)