[202205][VOQ] pfcwd failed in SonicDBConfig::validateNamespace #12062

ysmanman · 2022-09-13T21:07:55Z

Description

With 202205 image, we noticed the following error when running sonic-mgmt test:

    "  File \\"/usr/local/bin/pfcwd\\", line 8, in <module>",
    "    sys.exit(cli())",
    "  File \\"/usr/local/lib/python3.9/dist-packages/click/core.py\\", line 764, in __call__",
    "    return self.main(*args, **kwargs)",
    "  File \\"/usr/local/lib/python3.9/dist-packages/click/core.py\\", line 717, in main",
    "    rv = self.invoke(ctx)",
    "  File \\"/usr/local/lib/python3.9/dist-packages/click/core.py\\", line 1137, in invoke",
    "    return _process_result(sub_ctx.command.invoke(sub_ctx))",
    "  File \\"/usr/local/lib/python3.9/dist-packages/click/core.py\\", line 956, in invoke",
    "    return ctx.invoke(self.callback, **ctx.params)",
    "  File \\"/usr/local/lib/python3.9/dist-packages/click/core.py\\", line 555, in invoke",
    "    return callback(*args, **kwargs)",
    "  File \\"/usr/local/lib/python3.9/dist-packages/click/decorators.py\\", line 57, in new_func",
    "    obj = ctx.ensure_object(object_type)",
    "  File \\"/usr/local/lib/python3.9/dist-packages/click/core.py\\", line 477, in ensure_object",
    "    self.obj = rv = object_type()",
    "  File \\"/usr/local/lib/python3.9/dist-packages/utilities_common/db.py\\", line 36, in __init__",
    "    multi_asic.connect_config_db_for_ns(ns)",
    "  File \\"/usr/local/lib/python3.9/dist-packages/sonic_py_common/multi_asic.py\\", line 47, in connect_config_db_for_ns",
    "    config_db = swsscommon.ConfigDBConnector(namespace=namespace)",
    "  File \\"/usr/lib/python3/dist-packages/swsscommon/swsscommon.py\\", line 1906, in __init__",
    "    super(ConfigDBConnector, self).__init__(use_unix_socket_path = use_unix_socket_path, namespace = namespace)",
    "  File \\"/usr/lib/python3/dist-packages/swsscommon/swsscommon.py\\", line 1584, in __init__",
    "    for db_name in self.get_db_list():",
    "  File \\"/usr/lib/python3/dist-packages/swsscommon/swsscommon.py\\", line 1520, in get_db_list",
    "    return _swsscommon.SonicV2Connector_Native_get_db_list(self)",
    "RuntimeError: :- validateNamespace: Initialize global DB config using API SonicDBConfig::initializeGlobalConfig"

The cmd caused the above error is: pfcwd start_default

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of `show version`:

(paste your output here)

Output of `show techsupport`:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

The text was updated successfully, but these errors were encountered:

ysmanman · 2022-09-13T21:09:26Z

Add @arlakshm for viz.

ysmanman · 2022-09-13T21:25:59Z

We may need include sonic-net/sonic-utilities#2238 in 202205 branch.

liuh-80 · 2022-09-14T01:51:23Z

Can't reproduce this issue with latest 202205 branch build:

admin@vlab-01:$ pfcwd start_default
Root privileges are required for this operation
admin@vlab-01:$ sudo pfcwd start_default
admin@vlab-01:~$

After check the source code of pfcwd, the issue happened when try to validate namespace but not initialize global config.
Issue may caused by some mock in sonic-mgmt, need more information to debug.

@ysmanman , could you please give more detail about:

which UT in sonic-mgmt failed?
The test environment？hardware SKU and sonic image version?
please also share the failed UT log.

ysmanman · 2022-09-14T05:38:29Z

Can't reproduce this issue with latest 202205 branch build:

admin@vlab-01:~~$ pfcwd start_default Root privileges are required for this operation admin@vlab-01:~~$ sudo pfcwd start_default admin@vlab-01:~$

After check the source code of pfcwd, the issue happened when try to validate namespace but not initialize global config. Issue may caused by some mock in sonic-mgmt, need more information to debug.

@ysmanman , could you please give more detail about:

which UT in sonic-mgmt failed?

The test environment？hardware SKU and sonic image version?

please also share the failed UT log.

Hi Hua,

We first noticed the error because deploy-mg failed. According the traceback, we were able to manually reproduce the error with 'pfcwd start_default'. One thing to notice is the issue was only seen with multi-ASIC. We observed this in both sup and also LC that have 2 ASICs.

admin@cmp210:/$ show platform summary
Platform: x86_64-arista_7800_sup
HwSKU: Arista-7804R3-FM
ASIC: broadcom
ASIC Count: 12
Serial Number: SSN20200015
Model Number: DCS-7800A-SUP1A
Hardware Revision: 01.0a
Switch Type: fabric
admin@cmp210:/$ pfcwd start_default
Traceback (most recent call last):
File "/usr/local/bin/pfcwd", line 8, in
sys.exit(cli())
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/click/decorators.py", line 57, in new_func
obj = ctx.ensure_object(object_type)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 477, in ensure_object
self.obj = rv = object_type()
File "/usr/local/lib/python3.9/dist-packages/utilities_common/db.py", line 36, in init
multi_asic.connect_config_db_for_ns(ns)
File "/usr/local/lib/python3.9/dist-packages/sonic_py_common/multi_asic.py", line 47, in connect_config_db_for_ns
config_db = swsscommon.ConfigDBConnector(namespace=namespace)
File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1906, in init
super(ConfigDBConnector, self).init(use_unix_socket_path = use_unix_socket_path, namespace = namespace)
File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1584, in init
for db_name in self.get_db_list():
File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1520, in get_db_list
return _swsscommon.SonicV2Connector_Native_get_db_list(self)
RuntimeError: :- validateNamespace: Initialize global DB config using API SonicDBConfig::initializeGlobalConfig
admin@cmp210:/$

The error was not seen after we patched your change in sonic-net/sonic-utilities#2238.

liuh-80 · 2022-09-14T07:33:26Z

@ysmanman Thanks, I add the request for 202205 label to this PR for cherry pick:
sonic-net/sonic-utilities#2238

zhangyanzhao · 2022-09-14T15:15:55Z

Is this issue fixed? Can we close it? @ysmanman Thanks.

rlhui · 2022-09-14T15:30:41Z

@vmittal-msft - fyi

ysmanman · 2022-09-14T16:36:16Z

Is this issue fixed? Can we close it? @ysmanman Thanks.

@zhangyanzhao The issue is still in 202205. sonic-net/sonic-utilities#2238 should fix this issue. @liuh-80 will request to cherry pick the PR in 202205.

rlhui · 2022-10-01T19:01:56Z

@ysmanman - this PR is supposed to fix it. sonic-net/sonic-utilities#2372 could you please confirm if issue is fixed? Thanks.

ysmanman · 2022-10-04T03:23:25Z

@ysmanman - this PR is supposed to fix it. sonic-net/sonic-utilities#2372 could you please confirm if issue is fixed? Thanks.

Hi @rlhui yes, sonic-net/sonic-utilities#2372 fixed the issue.

ysmanman mentioned this issue Sep 13, 2022

Fix DBConfig not initialize issue in pfcwd sonic-net/sonic-utilities#2238

Merged

qiluo-msft assigned liuh-80 Sep 13, 2022

zhangyanzhao added Triaged this issue has been triaged MSFT labels Sep 14, 2022

zhangyanzhao assigned yxieca Sep 14, 2022

rlhui added the Chassis 🤖 Modular chassis support label Sep 14, 2022

ysmanman closed this as completed Oct 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[202205][VOQ] pfcwd failed in SonicDBConfig::validateNamespace #12062

[202205][VOQ] pfcwd failed in SonicDBConfig::validateNamespace #12062

ysmanman commented Sep 13, 2022

ysmanman commented Sep 13, 2022

ysmanman commented Sep 13, 2022

liuh-80 commented Sep 14, 2022 •

edited

Loading

ysmanman commented Sep 14, 2022

liuh-80 commented Sep 14, 2022

zhangyanzhao commented Sep 14, 2022

rlhui commented Sep 14, 2022

ysmanman commented Sep 14, 2022

rlhui commented Oct 1, 2022

ysmanman commented Oct 4, 2022

[202205][VOQ] pfcwd failed in SonicDBConfig::validateNamespace #12062

[202205][VOQ] pfcwd failed in SonicDBConfig::validateNamespace #12062

Comments

ysmanman commented Sep 13, 2022

Description

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

Output of show techsupport:

Additional information you deem important (e.g. issue happens only occasionally):

ysmanman commented Sep 13, 2022

ysmanman commented Sep 13, 2022

liuh-80 commented Sep 14, 2022 • edited Loading

ysmanman commented Sep 14, 2022

liuh-80 commented Sep 14, 2022

zhangyanzhao commented Sep 14, 2022

rlhui commented Sep 14, 2022

ysmanman commented Sep 14, 2022

rlhui commented Oct 1, 2022

ysmanman commented Oct 4, 2022

Output of `show version`:

Output of `show techsupport`:

liuh-80 commented Sep 14, 2022 •

edited

Loading