[Bug]: telemetry/test_telemetry.py::test_telemetry_ouput and /test_mgmt_ipv6_only.py::test_telemetry_output_ipv6_only failed at internal PR test #14448

StormLiangMS · 2024-09-06T02:24:59Z

Issue Description

https://elastictest.org/scheduler/testplan/66da09e1f90c6cd947f4e495?testcase=telemetry%2Ftest_telemetry.py%7C%7C%7C2&type=console&leftSideViewMode=detail&prop=status&order=ascending&searchTestCase=

always run into failure, which blocked the internal PR test for weeks.

Results you see

elemetry/test_telemetry.py::test_telemetry_ouput[vlab-03-False]
-------------------------------- live log call ---------------------------------
21:19:53 init.pytest_runtest_call L0040 ERROR | Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 1788, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 513, in call
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 139, in _multicall
raise exception.with_traceback(exception.traceback)
File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 103, in _multicall
res = hook_impl.function(*args)
File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/var/src/sonic-mgmt/tests/telemetry/test_telemetry.py", line 122, in test_telemetry_ouput
show_gnmi_out = ptfhost.shell(cmd)['stdout']
File "/var/src/sonic-mgmt/tests/common/devices/base.py", line 131, in _run
raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res)
tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
{"changed": true, "cmd": "python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.0.105 -p 50051 -m get -x COUNTERS/Ethernet0 -xt COUNTERS_DB -o "ndastreamingservertest"", "delta": "0:00:00.948821", "end": "2024-09-05 21:19:53.614096", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2024-09-05 21:19:52.665275", "stderr": "", "stderr_lines": [], "stdout": "Performing GetRequest, encoding=JSON_IETF to 10.250.0.105 with the following gNMI Path\n ------------------------- \n [elem {\n name: "COUNTERS"\n}\nelem {\n name: "Ethernet0"\n}\n]\nGRPC error\n key GLOBAL:3 not exists in COUNTERS_QUEUE_NAME_MAP", "stdout_lines": ["Performing GetRequest, encoding=JSON_IETF to 10.250.0.105 with the following gNMI Path", " ------------------------- ", " [elem {", " name: "COUNTERS"", "}", "elem {", " name: "Ethernet0"", "}", "]", "GRPC error", " key GLOBAL:3 not exists in COUNTERS_QUEUE_NAME_MAP"]}

FAILED [ 33%]Thursday 05 September 2024 21:20:00 +0000 (0:00:07.053) 0:02:22.499 ****
Thursday 05 September 2024 21:20:01 +0000 (0:00:00.639) 0:02:23.138 ****
Thursday 05 September 2024 21:20:02 +0000 (0:00:00.790) 0:02:23.928 ****
Thursday 05 September 2024 21:20:02 +0000 (0:00:00.699) 0:02:24.628 ****
Thursday 05 September 2024 21:20:08 +0000 (0:00:05.481) 0:02:30.110 ****
Thursday 05 September 2024 21:20:08 +0000 (0:00:00.538) 0:02:30.648 ****
Thursday 05 September 2024 21:20:09 +0000 (0:00:00.598) 0:02:31.247 ****
Thursday 05 September 2024 21:20:10 +0000 (0:00:00.762) 0:02:32.010 ****
Thursday 05 September 2024 21:20:10 +0000 (0:00:00.491) 0:02:32.502 ****

=================================== FAILURES ===================================
_____________________ test_telemetry_ouput[vlab-03-False] ______________________

duthosts = []
enum_rand_one_per_hwsku_hostname = 'vlab-03'
ptfhost = <tests.common.devices.ptf.PTFHost object at 0x7f21606b7550>
setup_streaming_telemetry = None, gnxi_path = '/root/gnxi/'

@pytest.mark.parametrize('setup_streaming_telemetry', [False], indirect=True)
def test_telemetry_ouput(duthosts, enum_rand_one_per_hwsku_hostname, ptfhost,
                         setup_streaming_telemetry, gnxi_path):
    """Run pyclient from ptfdocker and show gnmi server outputself.
    """
    duthost = duthosts[enum_rand_one_per_hwsku_hostname]
    env = GNMIEnvironment(duthost, GNMIEnvironment.TELEMETRY_MODE)
    if duthost.is_supervisor_node():
        pytest.skip(
            "Skipping test as no Ethernet0 frontpanel port on supervisor")
    logger.info('start telemetry output testing')
    dut_ip = duthost.mgmt_ip
    cmd = 'python ' + gnxi_path + 'gnmi_cli_py/py_gnmicli.py -g -t {0} -p {1} -m get -x COUNTERS/Ethernet0 -xt \
        COUNTERS_DB -o "ndastreamingservertest"'.format(dut_ip, env.gnmi_port)

  show_gnmi_out = ptfhost.shell(cmd)['stdout']

cmd = 'python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.0.105 -p 50051 -m get -x COUNTERS/Ethernet0 -xt COUNTERS_DB -o "ndastreamingservertest"'
dut_ip = '10.250.0.105'
duthost =
duthosts = []
enum_rand_one_per_hwsku_hostname = 'vlab-03'
env = <tests.common.helpers.gnmi_utils.GNMIEnvironment object at 0x7f2158394220>
gnxi_path = '/root/gnxi/'
ptfhost = <tests.common.devices.ptf.PTFHost object at 0x7f21606b7550>
setup_streaming_telemetry = None

telemetry/test_telemetry.py:122:

Results you expected to see

test can pass

Is it platform specific

generic

Relevant log output

No response

Output of `show version`

No response

Attach files (if any)

No response

The text was updated successfully, but these errors were encountered:

StormLiangMS · 2024-09-06T02:25:54Z

/test_mgmt_ipv6_only.py::test_telemetry_output_ipv6_only[vlab-03-True]
-------------------------------- live log call ---------------------------------
21:12:05 init.pytest_runtest_call L0040 ERROR | Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 1788, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 513, in call
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 139, in _multicall
raise exception.with_traceback(exception.traceback)
File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 103, in _multicall
res = hook_impl.function(*args)
File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/var/src/sonic-mgmt/tests/ip/test_mgmt_ipv6_only.py", line 197, in test_telemetry_output_ipv6_only
show_gnmi_out = duthost.shell(cmd)['stdout']
File "/var/src/sonic-mgmt/tests/common/devices/multi_asic.py", line 134, in _run_on_asics
return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)
File "/var/src/sonic-mgmt/tests/common/devices/base.py", line 131, in _run
raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res)
tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
{"changed": true, "cmd": "~/gnmi_get -xpath_target COUNTERS_DB -xpath COUNTERS/Ethernet0 -target_addr [fec0::ffff:afa:5]:50051 -logtostderr -insecure", "delta": "0:00:00.172866", "end": "2024-09-05 21:12:04.389352", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2024-09-05 21:12:04.216486", "stderr": "F0905 21:12:04.384017 132017 gnmi_get.go:145] Get failed: rpc error: code = NotFound desc = key GLOBAL:3 not exists in COUNTERS_QUEUE_NAME_MAP", "stderr_lines": ["F0905 21:12:04.384017 132017 gnmi_get.go:145] Get failed: rpc error: code = NotFound desc = key GLOBAL:3 not exists in COUNTERS_QUEUE_NAME_MAP"], "stdout": "== getRequest:\nprefix: <\n target: "COUNTERS_DB"\n>\npath: <\n elem: <\n name: "COUNTERS"\n >\n elem: <\n name: "Ethernet0"\n >\n>\nencoding: JSON_IETF", "stdout_lines": ["== getRequest:", "prefix: <", " target: "COUNTERS_DB"", ">", "path: <", " elem: <", " name: "COUNTERS"", " >", " elem: <", " name: "Ethernet0"", " >", ">", "encoding: JSON_IETF"]}

FAILED Thursday 05 September 2024 21:12:16 +0000 (0:00:00.551) Thursday 05 September 2024 21:12:48 +0000 (0:00:32.094) Thursday 05 September 2024 21:13:26 +0000 (0:00:38.828) Thursday 05 September 2024 21:13:48 +0000 (0:00:21.509) Thursday 05 September 2024 21:14:09 +0000 (0:00:20.984) Thursday 05 September 2024 21:14:10 +0000 (0:00:00.858) Thursday 05 September 2024 21:14:14 +0000 (0:00:03.873) Thursday 05 September 2024 21:14:25 +0000 (0:00:11.702) Thursday 05 September 2024 21:14:27 +0000 (0:00:01.932) Thursday 05 September 2024 21:14:30 +0000 (0:00:02.252) Thursday 05 September 2024 21:14:31 +0000 (0:00:01.802) Thursday 05 September 2024 21:14:32 +0000 (0:00:00.609) Thursday 05 September 2024 21:14:35 +0000 (0:00:02.553) Thursday 05 September 2024 21:14:35 +0000 (0:00:00.823) Thursday 05 September 2024 21:14:36 +0000 (0:00:00.672) Thursday 05 September 2024 21:14:37 +0000 (0:00:00.628) Thursday 05 September 2024 21:14:37 +0000 (0:00:00.613) Thursday 05 September 2024 21:14:38 +0000 (0:00:00.524) Thursday 05 September 2024 21:14:38 +0000 (0:00:00.644) [ 90%]Thursday 05 September 2024 21:12:15 +0000 (0:00:06.753) 0:11:49.458 ****
0:11:50.010 ****
0:12:22.104 ****
0:13:00.933 ****
0:13:22.443 ****
0:13:43.427 ****
0:13:44.286 ****
0:13:48.160 ****
0:13:59.862 ****
0:14:01.794 ****
0:14:04.046 ****
0:14:05.849 ****
0:14:06.459 ****
0:14:09.012 ****
0:14:09.835 ****
0:14:10.508 ****
0:14:11.136 ****
0:14:11.750 ****
0:14:12.275 ****
0:14:12.920 ****

=================================== FAILURES ===================================
________________ test_telemetry_output_ipv6_only[vlab-03-True] _________________

convert_and_restore_config_db_to_ipv6_only = None
duthosts = []
enum_rand_one_per_hwsku_hostname = 'vlab-03'
setup_streaming_telemetry_func = None

@pytest.mark.parametrize('setup_streaming_telemetry_func', [True], indirect=True)
def test_telemetry_output_ipv6_only(convert_and_restore_config_db_to_ipv6_only, # noqa F811
                                    duthosts, enum_rand_one_per_hwsku_hostname,
                                    setup_streaming_telemetry_func): # noqa F811
    # Add a temporary debug log to see if DUTs are reachable via IPv6 mgmt-ip. Will remove later
    log_eth0_interface_info(duthosts)
    duthost = duthosts[enum_rand_one_per_hwsku_hostname]
    env = GNMIEnvironment(duthost, GNMIEnvironment.TELEMETRY_MODE)
    if duthost.is_supervisor_node():
        pytest.skip(
            "Skipping test as no Ethernet0 frontpanel port on supervisor")
    dut_ip = get_mgmt_ipv6(duthost)
    cmd = "~/gnmi_get -xpath_target COUNTERS_DB -xpath COUNTERS/Ethernet0 -target_addr \
          [%s]:%s -logtostderr -insecure" % (dut_ip, env.gnmi_port)

  show_gnmi_out = duthost.shell(cmd)['stdout']

cmd = '~/gnmi_get -xpath_target COUNTERS_DB -xpath COUNTERS/Ethernet0 -target_addr [fec0::ffff:afa:5]:50051 -logtostderr -insecure'
convert_and_restore_config_db_to_ipv6_only = None
dut_ip = 'fec0::ffff:afa:5'
duthost =
duthosts = []
enum_rand_one_per_hwsku_hostname = 'vlab-03'
env = <tests.common.helpers.gnmi_utils.GNMIEnvironment object at 0x7f5c006230d0>
setup_streaming_telemetry_func = None

ip/test_mgmt_ipv6_only.py:197:

common/devices/multi_asic.py:134: in _run_on_asics
return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)
complex_args = {}
module_args = ('~/gnmi_get -xpath_target COUNTERS_DB -xpath COUNTERS/Ethernet0 -target_addr [fec0::ffff:afa:5]:50051 -logtostderr -insecure',)
self =

self =
module_args = ['~/gnmi_get -xpath_target COUNTERS_DB -xpath COUNTERS/Ethernet0 -target_addr [fec0::ffff:afa:5]:50051 -logtostderr -insecure']
complex_args = {}
previous_frame = <frame at 0x7f5c11b1d040, file '/var/src/sonic-mgmt/tests/common/devices/multi_asic.py', line 134, code _run_on_asics>
filename = '/var/src/sonic-mgmt/tests/common/devices/multi_asic.py'
line_number = 134, function_name = '_run_on_asics'
lines = [' return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)\n']
index = 0, verbose = True, module_ignore_errors = False, module_async = False

def _run(self, *module_args, **complex_args):

    previous_frame = inspect.currentframe().f_back
    filename, line_number, function_name, lines, index = inspect.getframeinfo(previous_frame)

    verbose = complex_args.pop('verbose', True)

    if verbose:
        logger.debug(
            "{}::{}#{}: [{}] AnsibleModule::{}, args={}, kwargs={}".format(
                filename,
                function_name,
                line_number,
                self.hostname,
                self.module_name,
                json.dumps(module_args, cls=AnsibleHostBase.CustomEncoder),
                json.dumps(complex_args, cls=AnsibleHostBase.CustomEncoder)
            )
        )
    else:
        logger.debug(
            "{}::{}#{}: [{}] AnsibleModule::{} executing...".format(
                filename,
                function_name,
                line_number,
                self.hostname,
                self.module_name
            )
        )

    module_ignore_errors = complex_args.pop('module_ignore_errors', False)
    module_async = complex_args.pop('module_async', False)

    if module_async:
        def run_module(module_args, complex_args):
            return self.module(*module_args, **complex_args)[self.hostname]
        pool = ThreadPool()
        result = pool.apply_async(run_module, (module_args, complex_args))
        return pool, result

    module_args = json.loads(json.dumps(module_args, cls=AnsibleHostBase.CustomEncoder))
    complex_args = json.loads(json.dumps(complex_args, cls=AnsibleHostBase.CustomEncoder))
    res = self.module(*module_args, **complex_args)[self.hostname]

    if verbose:
        logger.debug(
            "{}::{}#{}: [{}] AnsibleModule::{} Result => {}".format(
                filename,
                function_name,
                line_number,
                self.hostname,
                self.module_name, json.dumps(res, cls=AnsibleHostBase.CustomEncoder)
            )
        )
    else:
        logger.debug(
            "{}::{}#{}: [{}] AnsibleModule::{} done, is_failed={}, rc={}".format(
                filename,
                function_name,
                line_number,
                self.hostname,
                self.module_name,
                res.is_failed,
                res.get('rc', None)
            )
        )

    if (res.is_failed or 'exception' in res) and not module_ignore_errors:

      raise RunAnsibleModuleFail("run module {} failed".format(self.module_name), res)

E tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
E {"changed": true, "cmd": "~/gnmi_get -xpath_target COUNTERS_DB -xpath COUNTERS/Ethernet0 -target_addr [fec0::ffff:afa:5]:50051 -logtostderr -insecure", "delta": "0:00:00.172866", "end": "2024-09-05 21:12:04.389352", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2024-09-05 21:12:04.216486", "stderr": "F0905 21:12:04.384017 132017 gnmi_get.go:145] Get failed: rpc error: code = NotFound desc = key GLOBAL:3 not exists in COUNTERS_QUEUE_NAME_MAP", "stderr_lines": ["F0905 21:12:04.384017 132017 gnmi_get.go:145] Get failed: rpc error: code = NotFound desc = key GLOBAL:3 not exists in COUNTERS_QUEUE_NAME_MAP"], "stdout": "== getRequest:\nprefix: <\n target: "COUNTERS_DB"\n>\npath: <\n elem: <\n name: "COUNTERS"\n >\n elem: <\n name: "Ethernet0"\n >\n>\nencoding: JSON_IETF", "stdout_lines": ["== getRequest:", "prefix: <", " target: "COUNTERS_DB"", ">", "path: <", " elem: <", " name: "COUNTERS"", " >", " elem: <", " name: "Ethernet0"", " >", ">", "encoding: JSON_IETF"]}

complex_args = {}
filename = '/var/src/sonic-mgmt/tests/common/devices/multi_asic.py'
function_name = '_run_on_asics'
index = 0
line_number = 134
lines = [' return getattr(self.sonichost, self.multi_asic_attr)(*module_args, **complex_args)\n']
module_args = ['~/gnmi_get -xpath_target COUNTERS_DB -xpath COUNTERS/Ethernet0 -target_addr [fec0::ffff:afa:5]:50051 -logtostderr -insecure']
module_async = False
module_ignore_errors = False
previous_frame = <frame at 0x7f5c11b1d040, file '/var/src/sonic-mgmt/tests/common/devices/multi_asic.py', line 134, code _run_on_asics>
res = {'failed': True, 'changed': True, 'stdout': '== getRequest:\nprefix: <\n target: "COUNTERS_DB"\n>\npath: <\n elem: <...ailed: rpc error: code = NotFound desc = key GLOBAL:3 not exists in COUNTERS_QUEUE_NAME_MAP'], '_ansible_no_log': None}
self =
verbose = True

common/devices/base.py:131: RunAnsibleModuleFail

What is the motivation for this PR? #14448 #14449 these 2 issues block the internal sync and PR testing, skip them for now.

zbud-msft · 2024-09-06T18:12:17Z

Fixed with sonic-net/sonic-buildimage#20158, will wait for internal branch to pick up change, then close.

What is the motivation for this PR? sonic-net#14448 sonic-net#14449 these 2 issues block the internal sync and PR testing, skip them for now.

…14702) What is the motivation for this PR? In this PR, we add the telemetry test removed by #14451. The issue has been fixed in #14448. At the same time, we remove two unnecessary scripts in PR test.

What is the motivation for this PR? sonic-net#14448 sonic-net#14449 these 2 issues block the internal sync and PR testing, skip them for now.

…onic-net#14702) What is the motivation for this PR? In this PR, we add the telemetry test removed by sonic-net#14451. The issue has been fixed in sonic-net#14448. At the same time, we remove two unnecessary scripts in PR test.

StormLiangMS assigned zbud-msft Sep 6, 2024

StormLiangMS changed the title ~~[Bug]: telemetry/test_telemetry.py::test_telemetry_ouput failed at internal PR test~~ [Bug]: telemetry/test_telemetry.py::test_telemetry_ouput and /test_mgmt_ipv6_only.py::test_telemetry_output_ipv6_only failed at internal PR test Sep 6, 2024

StormLiangMS mentioned this issue Sep 6, 2024

skip cases which failed consistently #14451

Merged

8 tasks

StormLiangMS added a commit that referenced this issue Sep 6, 2024

skip cases which failed consistently (#14451)

358ae9e

What is the motivation for this PR? #14448 #14449 these 2 issues block the internal sync and PR testing, skip them for now.

zbud-msft closed this as completed Sep 19, 2024

hdwhdw pushed a commit to hdwhdw/sonic-mgmt that referenced this issue Sep 20, 2024

skip cases which failed consistently (sonic-net#14451)

8394b70

What is the motivation for this PR? sonic-net#14448 sonic-net#14449 these 2 issues block the internal sync and PR testing, skip them for now.

yutongzhang-microsoft mentioned this issue Sep 24, 2024

Restore telemetry test and clean up unnecessary scripts in PR Test #14702

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: telemetry/test_telemetry.py::test_telemetry_ouput and /test_mgmt_ipv6_only.py::test_telemetry_output_ipv6_only failed at internal PR test #14448

[Bug]: telemetry/test_telemetry.py::test_telemetry_ouput and /test_mgmt_ipv6_only.py::test_telemetry_output_ipv6_only failed at internal PR test #14448

StormLiangMS commented Sep 6, 2024

StormLiangMS commented Sep 6, 2024

zbud-msft commented Sep 6, 2024

[Bug]: telemetry/test_telemetry.py::test_telemetry_ouput and /test_mgmt_ipv6_only.py::test_telemetry_output_ipv6_only failed at internal PR test #14448

[Bug]: telemetry/test_telemetry.py::test_telemetry_ouput and /test_mgmt_ipv6_only.py::test_telemetry_output_ipv6_only failed at internal PR test #14448

Comments

StormLiangMS commented Sep 6, 2024

Issue Description

Results you see

Results you expected to see

Is it platform specific

Relevant log output

Output of show version

Attach files (if any)

StormLiangMS commented Sep 6, 2024

zbud-msft commented Sep 6, 2024

Output of `show version`