[system-health] "AssertionError: External checker does not work" after subprocess with "shell=True" has been removed.

[ihorchekh]

Description

This PR #12572 has broken the test: test_system_health::test_external_checker
The test is adding an external checker into the next file:
system_health_monitoring_config.json

{
    "services_to_ignore": [],
    "devices_to_ignore": [],
    "user_defined_checkers": [**"cat /tmp/mock_valid_external_checker.tx**t"],
    "polling_interval": 10,
    "led_color": {
      "fault": "orange",
      "normal": "green",
      "booting": "orange_blink"
    },
    "boot_timeout": 300
}

And it's not loading into STATE_DB because of this mentioned PR, which doesn't allow us to run a subprocess with "shell=True" anymore.
In the result we've got the next in the DB:

redis-cli -n 6 HGETALL "SYSTEM_HEALTH_INFO"

1) "UserDefinedChecker - /usr/bin/cat /tmp/mock_valid_external_checker.txt"
2) "**Failed to get output of command \"/usr/bin/cat /tmp/mock_valid_external_checker.txt\**""
3) "summary"
4) "Not OK"

Steps to reproduce the issue:

Note: For this example was used msn4600c. Please adjust the paths below for your platform.

1. Create a backup file for system_health_monitoring_config.json
   mv -f /usr/share/sonic/device/x86_64-mlnx_msn4600c-r0/system_health_monitoring_config.json /usr/share/sonic/device/x86_64-mlnx_msn4600c-r0/system_health_monitoring_config.json.bak

2. Create the next files with the next contents:

• vim /tmp/mock_valid_external_checker.txt

ExternalCategory
ExternalService:Service is not working
ExternalDevice:Device is broken

• vim /usr/share/sonic/device/x86_64-mlnx_msn4600c-r0/system_health_monitoring_config.json

{
    "services_to_ignore": [],
    "devices_to_ignore": [],
    "user_defined_checkers": ["cat /tmp/mock_valid_external_checker.txt"],
    "polling_interval": 10,
    "led_color": {
      "fault": "orange",
      "normal": "green",
      "booting": "orange_blink"
    },
    "boot_timeout": 300
}

3. Observe the issue in STATE_DB after a few seconds:

redis-cli -n 6 HGETALL SYSTEM_HEALTH_INFO

1) "summary"
2) "Not OK"
3) "UserDefinedChecker - cat /tmp/mock_valid_external_checker.txt"
4) "Failed to get output of command \"cat /tmp/mock_valid_external_checker.txt\""

4. For the cleanup after that, return the original file.
   mv -f /usr/share/sonic/device/x86_64-mlnx_msn4600c-r0/system_health_monitoring_config.json.bak /usr/share/sonic/device/x86_64-mlnx_msn4600c-r0/system_health_monitoring_config.json

Describe the results you received:

"Failed to get output of command \"cat /tmp/mock_valid_external_checker.txt\""

Describe the results you expected:

External checker should be read from the provided mock file

Output of show version:

SONiC Software Version: SONiC.202211_RC1.3-f75df985c_Internal
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: f75df985c
Build date: Mon Jan 16 06:58:56 UTC 2023
Built by: sw-r2d2-bot@r-build-sonic-ci03-244

Platform: x86_64-mlnx_msn4600c-r0
HwSKU: Mellanox-SN4600C-C64
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2138X00017
Model Number: MSN4600-CS2FO
Hardware Revision: A1
Uptime: 19:25:54 up 5 min,  1 user,  load average: 1.36, 0.86, 0.42
Date: Wed 18 Jan 2023 19:25:54

Docker images:
REPOSITORY                                         TAG                               IMAGE ID       SIZE
docker-platform-monitor                            202211_RC1.3-f75df985c_Internal   044a79ffc9f9   919MB
docker-platform-monitor                            latest                            044a79ffc9f9   919MB
docker-syncd-mlnx                                  202211_RC1.3-f75df985c_Internal   41a58ae1af31   914MB
docker-syncd-mlnx                                  latest                            41a58ae1af31   914MB
docker-orchagent                                   202211_RC1.3-f75df985c_Internal   82223e43ab79   527MB
docker-orchagent                                   latest                            82223e43ab79   527MB
docker-fpm-frr                                     202211_RC1.3-f75df985c_Internal   94715a457670   538MB
docker-fpm-frr                                     latest                            94715a457670   538MB
docker-teamd                                       202211_RC1.3-f75df985c_Internal   ca00cb922564   508MB
docker-teamd                                       latest                            ca00cb922564   508MB
docker-snmp                                        202211_RC1.3-f75df985c_Internal   9a4576a3804e   538MB
docker-snmp                                        latest                            9a4576a3804e   538MB
docker-macsec                                      latest                            b383b7b2b4ff   511MB
docker-dhcp-relay                                  latest                            714676b68a57   502MB
docker-sonic-p4rt                                  202211_RC1.3-f75df985c_Internal   e41876b1d8a6   574MB
docker-sonic-p4rt                                  latest                            e41876b1d8a6   574MB
docker-eventd                                      202211_RC1.3-f75df985c_Internal   4160f1bd9968   492MB
docker-eventd                                      latest                            4160f1bd9968   492MB
docker-sonic-telemetry                             202211_RC1.3-f75df985c_Internal   a3ebba5fcbcc   790MB
docker-sonic-telemetry                             latest                            a3ebba5fcbcc   790MB
docker-lldp                                        202211_RC1.3-f75df985c_Internal   c3dcdddfb4a0   534MB
docker-lldp                                        latest                            c3dcdddfb4a0   534MB
docker-router-advertiser                           202211_RC1.3-f75df985c_Internal   b2ddb1c195e9   492MB
docker-router-advertiser                           latest                            b2ddb1c195e9   492MB
docker-database                                    202211_RC1.3-f75df985c_Internal   48ca7292407a   492MB
docker-database                                    latest                            48ca7292407a   492MB
docker-mux                                         202211_RC1.3-f75df985c_Internal   c89c1002cfa8   541MB
docker-mux                                         latest                            c89c1002cfa8   541MB
docker-sonic-mgmt-framework                        202211_RC1.3-f75df985c_Internal   df629c6e2924   611MB
docker-sonic-mgmt-framework                        latest                            df629c6e2924   611MB
docker-nat                                         202211_RC1.3-f75df985c_Internal   7b748096bbf7   479MB
docker-nat                                         latest                            7b748096bbf7   479MB
docker-sflow                                       202211_RC1.3-f75df985c_Internal   600fd3a28d79   477MB
docker-sflow                                       latest                            600fd3a28d79   477MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   1.0.0-master-internal-105         23ddfcb0c5df   327MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/doroce      1.0.0-master-internal-11          9f27047097ac   200MB

Output of show techsupport:

sonic_dump_r-tigon-21_20230118_151922.tar.gz

[dprital]

@maipbui - Are you going to revert your PR ?

[maipbui]

@ihorchekh @dprital I open another PR to revert this PR.