Flaky pulse deprecation tests

[jakelishman]

Environment

• Qiskit Terra version: main @ eb4c5cb
• Python version: any
• Operating system: seen on Windows CI

What is happening?

CI is failing frequently with the following error:

test.python.pulse.test_parameter_manager.TestParameterSetter.test_complex_valued_parameter
------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "D:\a\1\s\test\python\pulse\test_parameter_manager.py", line 348, in test_complex_valued_parameter
    assigned = visitor.visit(test_obj)

      File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\unittest\case.py", line 270, in __exit__
    self._raiseFailure("{} not triggered".format(exc_name))

      File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\unittest\case.py", line 135, in _raiseFailure
    raise self.test_case.failureException(msg)

    AssertionError: PendingDeprecationWarning not triggered

This appears to be related to #9735.

How can we reproduce the issue?

Flaky CI failures.

What should happen?

No flakiness.

Any suggestions?

@TsafrirA and @nkanazawa1989, please can you look at this urgently - it's effectively blocking all CI on Terra.

[TsafrirA]

@jakelishman
Do you want to revert #9735 while I sort this out? Nothing really depends on it at the moment.

[jakelishman]

Thanks for the quick response! Yeah, if you think it might take a little while to get to the bottom of (it appears to be Windows only, which is usually a recipe for an annoying debugging experience), I can revert it and you'll not need to rush.

[TsafrirA]

Thanks!

[TsafrirA]

So, the issue seems to be the caching of format_parameter_value. There are two tests (in this testing unit alone, there are possibly others in other units) which format the value 0.1j, and so if the two tests are ran sufficiently close to one another, the second one will get the cached value, and not reach the warning in format_parameter_value.

I guess different operating systems run the tests in different order, failing the test or passing it according to the number of caches between the different 0.1j values?

I will replace the PR shortly.

[TsafrirA]

(I don't know how the cache is handled across different threads, maybe some operating systems have both calls to format_parameter_value executed "on the same time" thus avoiding the cache?)

[mtreinish]

In general the execution order of the tests is not fixed and it will change between runs and they also execute in parallel. We're probably seeing it on windows only primarily as a function of the number of vCPUs in the windows VM that ci jobs run in (which effects the split of tests), but that's not the only possible reason. When adding tests it's best to ensure there is no data dependency between tests to avoid issues like this. There are some tools builtin to stestr to try and work with and debug this such as:

• https://stestr.readthedocs.io/en/latest/MANUAL.html#automated-test-isolation-bisection
• https://stestr.readthedocs.io/en/latest/MANUAL.html#forcing-isolation
• https://stestr.readthedocs.io/en/latest/MANUAL.html#test-scheduling (specifically --random can be handy to force random execution order)

you can pass any of these options to stestr via tox with something like: tox -epy -- --random

[nkanazawa1989]

resolved by #9897