[supervisor][mutilasic] parseDatabaseConfig error and sonic-db-cli fails to set CONFIG_DB_INITIALIZED

[mlok-nokia]

Description

On supervisor card with multiasic, sonic-db-cli fails to set the "CONFIG_DB_INITIALIZED" after configuration is loaded on an asic instance database. This causes all show commands hung. The issue could be related to the issue #12047. But this fails on the parseDatabaseConfig while the previous issue fails on the database_config.json not found.

Steps to reproduce the issue:

This is an intermittent issue. The test case reboots the system three times. The issue occurs on the third time. The following log is shown

Feb  4 19:11:40.428270 ixre-cpm-chassis8 ERR sonic-db-cli: :- parseDatabaseConfig: Sonic database config file syntax error >> parse error - unexpected end of input
Feb  4 19:11:40.428527 ixre-cpm-chassis8 ERR sonic-db-cli: :- initializeGlobalConfig: Sonic database config file syntax error >> Sonic database config file syntax error >> parse error - unexpected end of input
Feb  4 19:11:40.428768 ixre-cpm-chassis8 INFO database.sh[7393]: terminate called after throwing an instance of 'std::runtime_error'
Feb  4 19:11:40.428961 ixre-cpm-chassis8 INFO database.sh[7393]:   what():  Sonic database config file syntax error >> Sonic database config file syntax error >> parse error - unexpected end of input
Feb  4 19:11:40.491831 ixre-cpm-chassis8 INFO database.sh[2236]: /usr/bin/database.sh: line 128:  7393 Aborted                 (core dumped) $SONIC_DB_CLI CONFIG_DB SET "CONFIG_DB_INITIALIZED" "1"

Describe the results you received:

All show commands hung and the following log are shown in syslog.

Feb  4 19:11:40.428270 ixre-cpm-chassis8 ERR sonic-db-cli: :- parseDatabaseConfig: Sonic database config file syntax error >> parse error - unexpected end of input
Feb  4 19:11:40.428527 ixre-cpm-chassis8 ERR sonic-db-cli: :- initializeGlobalConfig: Sonic database config file syntax error >> Sonic database config file syntax error >> parse error - unexpected end of input
Feb  4 19:11:40.428768 ixre-cpm-chassis8 INFO database.sh[7393]: terminate called after throwing an instance of 'std::runtime_error'
Feb  4 19:11:40.428961 ixre-cpm-chassis8 INFO database.sh[7393]:   what():  Sonic database config file syntax error >> Sonic database config file syntax error >> parse error - unexpected end of input
Feb  4 19:11:40.491831 ixre-cpm-chassis8 INFO database.sh[2236]: /usr/bin/database.sh: line 128:  7393 Aborted                 (core dumped) $SONIC_DB_CLI CONFIG_DB SET "CONFIG_DB_INITIALIZED" "1"

Describe the results you expected:

Show command should work and load config should be ok.

Output of show version:

Issues are seen on the latest code.

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[arlakshm]

@qiluo-msft, can you take a look at this issue?

[liuh-80]

Here is the code from https://github.com/sonic-net/sonic-buildimage/blob/master/files/build_templates/docker_image_ctl.j2#L236

        # If there is a config_db.json dump file, load it.
        if [ -r /etc/sonic/config_db$DEV.json ]; then

	# For multi-asic, all /var/run/redis$DEV/sonic-db/database_config.json need to ready
            # for loading config with --write-to-db
	waitForAllInstanceDatabaseConfigJsonFilesReady

            if [ -r /etc/sonic/init_cfg.json ]; then
                $SONIC_CFGGEN -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db$DEV.json --write-to-db
            else
                $SONIC_CFGGEN -j /etc/sonic/config_db$DEV.json --write-to-db
            fi
        fi

        if [[ "$BOOT_TYPE" == "fast" ]]; then
            # set the key to expire in 3 minutes
            $SONIC_DB_CLI STATE_DB SET "FAST_REBOOT|system" "1" "EX" "180"
        fi

        **$SONIC_DB_CLI CONFIG_DB SET "CONFIG_DB_INITIALIZED" "1"**

In above code, '$SONIC_DB_CLI CONFIG_DB SET "CONFIG_DB_INITIALIZED" "1"' will failed with following error:
parseDatabaseConfig: Sonic database config file syntax error

Which means the database config file generated incorrectly.
I think this PR #13207 will also fix this issue.

@mlok-nokia, do you mean this issue still exist after PR #13207?

[mlok-nokia]

Here is the code from https://github.com/sonic-net/sonic-buildimage/blob/master/files/build_templates/docker_image_ctl.j2#L236

        # If there is a config_db.json dump file, load it.
        if [ -r /etc/sonic/config_db$DEV.json ]; then

	# For multi-asic, all /var/run/redis$DEV/sonic-db/database_config.json need to ready
            # for loading config with --write-to-db
	waitForAllInstanceDatabaseConfigJsonFilesReady

            if [ -r /etc/sonic/init_cfg.json ]; then
                $SONIC_CFGGEN -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db$DEV.json --write-to-db
            else
                $SONIC_CFGGEN -j /etc/sonic/config_db$DEV.json --write-to-db
            fi
        fi

        if [[ "$BOOT_TYPE" == "fast" ]]; then
            # set the key to expire in 3 minutes
            $SONIC_DB_CLI STATE_DB SET "FAST_REBOOT|system" "1" "EX" "180"
        fi

        **$SONIC_DB_CLI CONFIG_DB SET "CONFIG_DB_INITIALIZED" "1"**

In above code, '$SONIC_DB_CLI CONFIG_DB SET "CONFIG_DB_INITIALIZED" "1"' will failed with following error: parseDatabaseConfig: Sonic database config file syntax error

Which means the database config file generated incorrectly. I think this PR #13207 will also fix this issue.

@mlok-nokia, do you mean this issue still exist after PR #13207?

No. I have not seen it with the PR#13207. I thought PR#13207 may also fix this issue also

[liuh-80]

When config file incorrect, both c++ version and python version will crash with exception:

admin@vlab-08:~$ sonic-db-cli -n asic0 PING
terminate called after throwing an instance of 'std::runtime_error'
what(): Sonic database config file syntax error >> Sonic database config file syntax error >> parse error - unexpected end of input; expected '}'
Aborted (core dumped)

admin@vlab-08:$ ./sonic-db-cli -n asic0 PING
Traceback (most recent call last):
File "/home/admin/./sonic-db-cli", line 124, in
main()
File "/home/admin/./sonic-db-cli", line 110, in main
swsssdk.SonicDBConfig.load_sonic_global_db_config(namespace=args.namespace)
File "/usr/local/lib/python3.9/dist-packages/swsssdk/dbconnector.py", line 77, in load_sonic_global_db_config
SonicDBConfig._sonic_db_config[ns] = json.load(inc_file)
File "/usr/lib/python3.9/json/init.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.9/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 84 column 1 (char 1950)
admin@vlab-08:$

I think here is an issue: the c++ version crash will generate core dump, and python version crash will not, will improve c++ version to not generate core dump.

However, this issue caused by incorrect database config file, will be fixed by PR #13207