[pytest] improvements for pytest infrastructure #2

stepanblyschak · 2019-07-25T15:33:52Z

Description of PR

Summary:

pytest:
- W/A some issues with pytest-ansible plugin and provided fixtures
- add ptf adapter package and fixture to send/receive PTF traffic controlled from pytest
- translate fdb test to use PTF adapter
- Divide LLDP test into two test cases
ansible
- add pytest_runner.yml to wrap pytest in ansible run
- remove lldp.yml test code ansible and lldp_neighbor.yml. Instead lldp.yml will run pytest test_lldp.py

Fixes # (issue)

Type of change

[] Bug fix
Testbed and Framework(new/improvement)
[] Test case(new/improvement)

Approach

How did you do it?

How did you verify/test it?

stepanb@62c67c04c2bd:/stepanb/sonic-mgmt/tests$ py.test -k 'not platform'  --testbed=$SWITCH-t0 --testbed_file=testbed.csv --inventory=../ansible/inventory --host-pattern $SWITCH-t0 -v --ignore=ptftests
=========================================================================================================== test session starts ===========================================================================================================
platform linux2 -- Python 2.7.12, pytest-4.6.4, py-1.8.0, pluggy-0.12.0 -- /usr/bin/python
cachedir: .pytest_cache
Using --randomly-seed=1564067815
ansible: 2.0.0.2
rootdir: /stepanb/sonic-mgmt/tests, inifile: pytest.ini
plugins: randomly-1.2.3, ansible-2.0.2
collected 16 items / 11 deselected / 5 selected                                                                                                                                                                                           

test_bgp_fact.py::test_bgp_facts PASSED                                                                                                                                                                                             [ 20%]
test_lldp.py::test_lldp PASSED                                                                                                                                                                                                      [ 40%]
test_lldp.py::test_lldp_neighbor PASSED                                                                                                                                                                                             [ 60%]
test_bgp_speaker.py::test_bgp_speaker SKIPPED                                                                                                                                                                                       [ 80%]
fdb/test_fdb.py::test_fdb PASSED                                                                                                                                                                                                    [100%]

========================================================================================== 4 passed, 1 skipped, 11 deselected in 147.06 seconds ===========================================================================================

Verify ansible LLDP test

ansible-playbook test_sonic.yml -i inventory -l r-boxer-sw01 -e testbed_name=r-boxer-sw01-t0 -e topo=t0 -e testcase_name=lldp -vvvvvv

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

…e issues There are two issues with pytest-ansible I found: - ansible_adhoc is limited to function scope which makes it not usable for wider scope test fixtures - localhost fixture has some hidden dependency on ansible_adhoc (even without changing default scope) e.g.: def test_x(ansible_adhoc, testbed): duthost = AnsibleHost(ansible_adhoc, testbed['dut']) pass def test_y(ansible_adhoc, localhost, testbed): pass This snippet of test code fails to evaluate localhost fixture in test_y: test_x.py::test_x PASSED [ 50%] test_x.py::test_y ERROR [100%] Part of error log: host = self.get_host(hostname) if host is None: > raise AnsibleError("no host vars as host is not in inventory: %s" % hostname) E AnsibleError: ERROR! no host vars as host is not in inventory: arc-switch1025 Signed-off-by: Stepan Blyschak <[email protected]>

…trol PTF traffic from sonic-mgmt node Signed-off-by: Stepan Blyschak <[email protected]>

Signed-off-by: Stepan Blyschak <[email protected]>

…f framework Signed-off-by: Stepan Blyschak <[email protected]>

Signed-off-by: Stepan Blyschak <[email protected]>

…ention Signed-off-by: Stepan Blyschak <[email protected]>

Signed-off-by: Stepan Blyschak <[email protected]>

…unity string Signed-off-by: Stepan Blyschak <[email protected]>

… cases for lldp should be done in pytest Signed-off-by: Stepan Blyschak <[email protected]>

Signed-off-by: Stepan Blyschak <[email protected]>

wangxin · 2019-07-26T08:42:01Z

Regarding tests/fdb/init.py, is it necessary to make the fdb testing scripts a package?

wangxin · 2019-07-26T08:43:11Z

tests/fdb/test_fdb.py

+    res = duthost.command('show mac')
+    logger.debug('"show mac" output on DUT')
+    for line in res['stdout_lines']:
+        logger.debug('    {}'.format(line))


Package pprint is for this kind of job.

Thanks, I replaced with pprint

Signed-off-by: Stepan Blyschak <[email protected]>

stepanblyschak · 2019-07-29T12:19:42Z

Regarding tests/fdb/init.py, is it necessary to make the fdb testing scripts a package?

No, thanks

Signed-off-by: Roman Kachur <[email protected]>

Stepan Blyschak added 20 commits July 25, 2019 13:40

[pytest-infra] introduce PtfTestAdapter and ptfadapter fixture to con…

329e67c

…trol PTF traffic from sonic-mgmt node Signed-off-by: Stepan Blyschak <[email protected]>

[ptfadapter] add default ptf config

a003a28

Signed-off-by: Stepan Blyschak <[email protected]>

[conftest] move overridden ansible fixture to seperate file

d5830e1

Signed-off-by: Stepan Blyschak <[email protected]>

[ptf_runner] rename ptf.py to ptf_runner.py to avoid conflict with pt…

7221045

…f framework Signed-off-by: Stepan Blyschak <[email protected]>

[gitignore] update with cpython compiled files

9328407

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/fdb] rewrite fdb test with ptfadapter & fix change_mac.sh script

e890945

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/ansible_host] rename to AnsibleHost to meet python naming conv…

a930acb

…ention Signed-off-by: Stepan Blyschak <[email protected]>

[tests/ptfadapter] move qlen/relax to PtfTestAdapter object

1f335c6

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/ptfadapter] non need for disable_* config keys for newer ptf

6be1b29

Signed-off-by: Stepan Blyschak <[email protected]>

[tests] change ansible_host to AnsibleHost

45aa8b9

Signed-off-by: Stepan Blyschak <[email protected]>

[conftest] remove extra new line

37054a0

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/fdb] reduce info to debug

f80916f

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/fdb] cleanup FDB test

bcf541d

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/ptfadapter] correct on the comment

6a1cba1

Signed-off-by: Stepan Blyschak <[email protected]>

[pytest] introduce pytest_runner playbook

e4a7bfe

Signed-off-by: Stepan Blyschak <[email protected]>

[test_lldp] divide into two test cases and put a TODO about SNMP comm…

bebc1f4

…unity string Signed-off-by: Stepan Blyschak <[email protected]>

[lldp] call py.test to run lldp test; from now on any change/new test…

7911f51

… cases for lldp should be done in pytest Signed-off-by: Stepan Blyschak <[email protected]>

[tests/bgp_speaker] skip until test is fixed

46eece9

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/ptfadapter] add README.md on how to use ptfadapter

d6be436

Signed-off-by: Stepan Blyschak <[email protected]>

stepanblyschak force-pushed the ptfadapter_infra branch from 4bb2f32 to d6be436 Compare July 25, 2019 16:58

wangxin reviewed Jul 26, 2019

View reviewed changes

Stepan Blyschak added 4 commits July 29, 2019 15:18

[.gitignore] ignore pytest_cached

bfb27c3

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/conftest] add shortcut fixtures for DUT and PTF

a6847d3

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/fdb] remove __init__.py from fdb test

20150f1

Signed-off-by: Stepan Blyschak <[email protected]>

[tests/fdb] use pprint

0b5c57d

Signed-off-by: Stepan Blyschak <[email protected]>

stepanblyschak closed this Jul 29, 2019

stepanblyschak pushed a commit that referenced this pull request Nov 4, 2019

anaconda testbed. 40G #2

2fc814c

Signed-off-by: Roman Kachur <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pytest] improvements for pytest infrastructure #2

[pytest] improvements for pytest infrastructure #2

stepanblyschak commented Jul 25, 2019 •

edited

Loading

wangxin commented Jul 26, 2019

wangxin Jul 26, 2019

stepanblyschak Jul 29, 2019

stepanblyschak commented Jul 29, 2019

[pytest] improvements for pytest infrastructure #2

[pytest] improvements for pytest infrastructure #2

Conversation

stepanblyschak commented Jul 25, 2019 • edited Loading

Description of PR

Type of change

Approach

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

wangxin commented Jul 26, 2019

wangxin Jul 26, 2019

Choose a reason for hiding this comment

stepanblyschak Jul 29, 2019

Choose a reason for hiding this comment

stepanblyschak commented Jul 29, 2019

stepanblyschak commented Jul 25, 2019 •

edited

Loading