[CPDEV-100486] Support kernel parameters/modules per group/node

[ilia1243]

Description

• Support kernel modules per group/node.
• Support kernel parameters per group/node.

Solution

• Format of services.modprobe is extended to support dictionary with properties modulename, nodes, groups, install.
• Format of services.sysctl is extended to support dictionary with properties value, nodes, groups, install.
• Existing defaults are moved to the new format, and distributed among roles.
• Added patch that sets previous default kernel parameters and modules in the user inventory,
  to keep them on all nodes, and preserve potential references to them in jinja templates.
• Implemented new patches section of the inventory that allows to specify different values of kernel parameters for different nodes.
• Other fixes and improvements:
  • All sysctl.d configs are now reloaded through sysctl --system.
    This allows to perform early validation of parameters.
  • More robust check modprobe.is_modprobe_valid() that the required modules are present.

How to apply

Run kubemarine migrate_kubemarine --force-apply preserve_compatibility_kernel

Test Cases

TestCase 1

Installation with default kernel parameters/modules per group/node.

Steps:

1. Install cluster with default parameters/modules.

ER: The parameters / modules are installed only on necessary nodes/groups.

TestCase 2

Migrate inventory.

Steps:

1. Install cluster using old Kubemarine version.
2. Run kubemarine migrate_kubemarine --force-apply preserve_compatibility_kernel using new Kubemarine version.

ER: previous default kernel parameters/modules are set in the inventory.

3. Add some new parameter in services.sysctl.
4. Run install --tasks prepare.system.sysctl using new Kubemarine version.

ER: The parameters / modules are installed on all nodes.

TestCase 3

Installation with kernel parameters/modules per group/node.

Steps:

1. Configure inventory with extended format of services.sysctl / services.modprobe and specify subset of nodes/groups to install the parameters/modules on.
2. Install the cluster.

ER: The parameters / modules are installed only on specified nodes/groups.

3. Run check_paas --tasks services.system.modprobe,services.system.sysctl

ER: Check is successful.

TestCase 4

Install kernel parameters with different values per group/node using patches.

Steps:

1. Configure inventory with patches and specify different kernel parameter values to install on different nodes/groups.
2. Install the cluster.

ER: Different parameters values are installed as per the configuration.

3. Run check_paas --tasks services.system.sysctl

ER: Check is successful.

TestCase 5

PaaS check with changed parameters/modules per group/node.

Steps:

1. Configure inventory with extended format of services.sysctl / services.modprobe and specify subset of nodes/groups to install the parameters/modules on.
2. Install the cluster.
3. Add group/node to one of existing parameter/module in the inventory.
4. Run check_paas --tasks services.system.modprobe,services.system.sysctl

ER: The checks fail with some nodes having not installed parameters/modules.

TestCase 6

PaaS check with changed kernel parameter values.

1. Install the cluster.
2. Add patches with different kernel parameters values.
3. Run check_paas --tasks services.system.sysctl

ER: The checks fail with some nodes not having the expected parameter values.

TestCase 7

Early sysctl validation without reboot.

1. Configure kernel parameter with the higher priority, for example in /etc/sysctl.d/99-*.conf.
2. Configure different value for the parameter in the inventory.
3. Run install --exclude-cumulative-points-methods kubemarine.system.reboot_nodes

Results:

Before	After
Installation is successful	Installation fails because the parameter value is not expected

4. Reboot the cluster
5. Run check_paas --tasks services.system.sysctl

AR=ER: the check fails.

TestCase 8

Robust PaaS check that kernel modules are valid.

Steps:

1. Configure inventory

services:
  modprobe:
    <OS family>:
      - '0'

2. Run check_paas --tasks services.system.modprobe

Results:

Before	After
Check is successful	Check fails with "Kernel module 0 is not found"

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• Integration CI passed
• Unit tests. If Yes list of new/changed tests with brief description
• There is no merge conflicts

Unit tests

test_defaults.py - cover new cases with enrichment of integer/boolean values.
test_sysctl.py, test_modprobe.py - new tests to cover enrichment of sysctl and modprobe.

[ilia1243]

@shmo1218 please review JSON schemas in kubemarine/resources/schemas modified in this PR apart from usual docs.