[ModuleCatalog] Extend Catalog sync to handle both ModuleTemplate and ModuleReleaseMeta CRs

[jeremyharisch]

Note

The original issue was about creating a dedicated sync controller.
For now though, we decided to just add the implementation to the current codebase with necessary refactorings.
I am leaving the original issue unchanged (below) for reference; this will be extracted to a separate issue after the first version of the extended CatalogSync code is implemented and released.

Current Acceptance Criteria:

• Extend the module catalog sync with support for ModuleReleaseMeta CRs
• Update the documentation
• Extend the test "CRDs sync to SKR and annotations updated in KCP kyma" to cover ModuleReleaseMeta CRs

Original Description:

The synchronization process of ModuleTemplate CRs to all SKR clusters is currently embedded within the Kyma reconcile loop. We have decided to expand this functionality to include ModuleReleaseMeta CRs and to relocate these synchronization tasks to a new, dedicated controller. This approach is intended to enhance performance by offloading the sync operations from the main Kyma reconcile loop.

Detailed decision ticket: #1815

Current Implementation:

The existing synchronization of ModuleTemplate CRs is integrated into the Kyma reconcile loop, detailed here.

Objective:

• Develop a Dedicated Sync Controller:

  • This new controller will handle the synchronization of both ModuleTemplate and ModuleReleaseMeta CRs from the KCP cluster to all SKR clusters.
  • It will manage the lifecycle of these CRs—handling creation, updates, and deletions—and will ensure that all corresponding Kyma CRs are requeued accordingly.

• Performance Optimization:

  • By isolating the synchronization process, we aim to reduce the overhead in the Kyma reconcile loop, which currently dedicates approximately 20% of its time to sync operations.

• Testing and Validation:

  • Comprehensive E2E and integration testing will be conducted to ensure the new controller operates effectively and integrates smoothly with existing systems.
  • Adjustments will be made to existing tests that are affected by this change.

Benefits of Separation of Concerns:

• Enhanced Performance: Removing the sync tasks from the Kyma controller allows for more focused and efficient reconciliation processes.
• Reduced Complexity in Core Operations: Segregating the synchronization functionality simplifies the main Kyma reconcile loop, making it easier to maintain and scale.

Potential Challenges:

• System Complexity: Introducing a new controller adds a layer of complexity to the overall architecture.
• Race Conditions: Separate controllers working on related resources could lead to conflicts, although these are generally manageable within Kubernetes through established patterns.

Acceptance Criteria:

• Implementation of the New Controller: The controller successfully synchronizes ModuleTemplate and ModuleReleaseMeta CRs across SKR clusters.
  • Make sure code is easy testable, stick to our current implementation patterns.
  • If a ModuleTemplate or ModuleReleaseMeta CR gets created, updated or deleted, all Kyma CRs should be requeued, since all are affected automatically.
  • Name of controller: SyncResources
  • Cover controller with basic metrics
  • Update only affects the .status.condition
• Performance Improvements Documented: Metrics demonstrating the reduced load on the Kyma reconcile loop.
• Comprehensive Test Coverage: The new functionality is fully covered by E2E and integration tests.
• Documentation: Updated documentation reflecting the new controller’s role and functionality within the ecosystem.

[Tomasz-Smelcerz-SAP]

I think we need to refine on the AC: "Name of controller: SyncResources". The SyncResources is too generic and vague. I suggest to use just: catalogsync