subtests with spaces no longer appear as different tests

[alisonatwork]

Type: Bug

Behaviour

Expected vs. Actual

When running Python tests with subtests, each of those subtests should appear as a separate entry in the Testing sidebar. This works for subtests without a space in them, but when there is a space the subtest name is truncated at the first space and then collapsed into all other subtests with the same prefix.

This is a regression. A few months ago this didn't happen, although I am not able to nail down exactly the version where it started breaking.

Steps to reproduce:

1. Create any Python unit test with a subtest
2. Pass a string with a space into it as the subtest argument
   -> string is truncated at the first space

Note that this also happens even if the string with a space is inside a subtest kwarg.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.10.11
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2023-08-02 12:36:20.210 [info] Running UNITTEST execution for the following test ids: hello_test.HelloTest.test_normal,hello_test.HelloTest.test_subtest_no_space,hello_test.HelloTest.test_subtest_with_space
2023-08-02 12:36:20.210 [info] Server listening on port 45983
2023-08-02 12:36:20.223 [info] Running unittests with arguments: /home/user/.vscode-server/extensions/ms-python.python-2023.13.12131007/pythonFiles/unittestadapter/execution.py,--port,45883,--uuid,6055708b-fc27-436a-80e9-cd18426580df,--udiscovery,-v,-s,.,-p,*_test.py

2023-08-02 12:36:20.223 [info] > ./.venv/bin/python ~/.vscode-server/extensions/ms-python.python-2023.13.12131007/pythonFiles/unittestadapter/execution.py --port 45883 --uuid 6055708b-fc27-436a-80e9-cd18426580df --udiscovery -v -s . -p *_test.py
2023-08-02 12:36:20.223 [info] cwd: .
2023-08-02 12:36:20.274 [info] Test server connected to a client.
2023-08-02 12:36:20.274 [info] Test server connected to a client.
2023-08-02 12:36:20.274 [info] Test server connected to a client.
2023-08-02 12:36:20.274 [info] Test server connected to a client.
2023-08-02 12:36:20.274 [info] Test server connected to a client.
2023-08-02 12:36:20.275 [info] Test server connected to a client.
2023-08-02 12:36:20.275 [info] Test server connected to a client.
2023-08-02 12:36:20.286 [info] Client disconnected
2023-08-02 12:36:20.294 [info] Exec server closed. 6055708b-fc27-436a-80e9-cd18426580df

User Settings

languageServer: "Pylance"

testing
• unittestArgs: "<placeholder>"
• unittestEnabled: true

Extension version: 2023.12.0
VS Code version: Code 1.80.2 (2ccd690cbff1569e4a83d7c43d45101f817401dc, 2023-07-27T20:40:28.909Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Remote OS version: Linux x64 5.10.16.3-microsoft-standard-WSL2
Remote OS version: Linux x64 5.10.16.3-microsoft-standard-WSL2

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz (8 x 2304)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	15.73GB (4.43GB free)
Process Argv	--crash-reporter-id 94afb580-e7da-46c8-95df-7ae255046cc7
Screen Reader	no
VM	0%

Item	Value
Remote	WSL: podman42-dev
OS	Linux x64 5.10.16.3-microsoft-standard-WSL2
CPUs	Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz (8 x 2304)
Memory (System)	7.63GB (5.00GB free)
VM	0%

Item	Value
Remote	WSL: podman42-dev
OS	Linux x64 5.10.16.3-microsoft-standard-WSL2
CPUs	Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz (8 x 2304)
Memory (System)	7.63GB (5.00GB free)
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263:30335439
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
py29gd2263:30792226
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
vscccc:30803845
282f8724:30602487
f6dab269:30613381
vscrpc:30673769
2i9eh265:30646982
showlangstatbar:30737416
vsctsb:30748421
03d35959:30757346
24365598:30736109
pythonfmttext:30731395
pythoncmvfstrcf:30756944
fixshowwlkth:30771522
hideindicator:30785051
pythongtdpath:30769146
i26e3531:30792625
gsofb:30797622
pythonnosmt12:30797651
pythonnoceb:30797650
e537b577:30795824
dsvsc013:30795093
dsvsc014:30804076

[eleanorjboyd]

Thank you for your issue report. I am seeing this bug myself You are likely on the new testing rewrite and this is why you saw a change in behavior since we just moved this experiment to a larger group of people receiving it. You can opt out of the rewrite as I get this fix in by setting this in your user settings: "python.experiments.optOutFrom": ["pythonTestAdapter"],. If this doesn't work let me know as this will mean it is a different issue. Thank you and I will send updates in this thread as I get a fix in.

[alisonatwork]

Thanks for the response. I tried setting this config in my user settings, but it seemed to not have any effect, even after I restarted VS Code. I have had similar problems before with trying to get the Microsoft Python extensions to pick up configuration, and I think it might be something to do with running VS Code on my local (Windows) machine and having the project inside a WSL 2 Linux VM.

That said, for the purpose of reporting this ticket I deliberately moved forward all the Python-related extensions to "pre-release version" to see if the problem was fixed there (it was not) and have since rolled back to standard/release version (where the problem still exists for me), so maybe on release version the experiment opt out doesn't work?

Either way, as long as this is on the agenda to get fixed, I will just grit my teeth and bear it for now 😬

[eleanorjboyd]

hm very interesting. Both release and pre-release are able to handle the experiment opt-out. How are you connecting to your remote machine? SSH? To check if you are getting opted out during running set "python.analysis.logLevel": "Trace", in your user settings and developer log level also to trace then check for Experiment 'pythonTestAdapter' in your python logs. It should be at the top and should say something along the lines of "off". Let me know if you don't see it.

We haven't made any changes to the non-rewrite code in the past few months which is why I am unsure of how a regression could have happened in this time period. I will fix this on the rewrite so hopefully that will come soon but would love to get you a fix in the meantime

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[alisonatwork]

What? This is not just still a bug, it's a confirmed bug!

[alisonatwork]

How am I supposed to reopen this, there is no button to do it?

[eleanorjboyd]

@alisonatwork are you on the experiment for the rewrite or not?

[alisonatwork]

Hi @eleanorjboyd as I mentioned in my earlier comment, none of the settings you mentioned appear to have any effect on my system.

I have put "python.analysis.logLevel": "Trace" into %APPDATA%\Code\User\settings.json and still only info level logs appear in the Python output tab.

I have also tried putting "python.experiments.optOutFrom": ["pythonTestAdapter"] in the same settings file and also nothing different happens to the Testing section.

As I said before, I am connecting to a WSL VM for Python development, so perhaps for some reason the changes to the settings from local %APPDATA% are not being picked up by the Python language server in the VM?

If it helps to try figure out what version I am on, recently (just in the past week or two) suddenly I started getting a new Test Results tab that never existed before, and that also has the broken subtest names.

In fact, that new Test Results tab is also annoying me because previously I could at least look at Python Test Log in the Output tab and it would show me all of the stdout logs that were emitted during the tests, but now in the Test Results tab I don't see anything (except the broken names of the subtests). It's especially annoying because every time I run the test again, something automatically switches to Test Results tab, even when I explicitly selected the Output tab, so I keep having to click back and forth all the time to see the test output. But I assume this is an unrelated "feature" so I haven't bothered reporting it.

In any case, although this bug is annoying me every day (I am a professional developer using VS Code and Python at work), I do not have enough free time to keep digging around in different config files to try to opt out of an A/B test that appears to exist outside of the "Switch to Pre-Release Version" functionality. It would be great if users just got the known-working version by default and explicitly had to opt in to these experiments.

[eleanorjboyd]

Hi @alisonatwork, sorry for the back and forth! I see a bug on the rewrite when trying myself and I was able to find a change to subtest ID code which I see now is the cause. The submitted PR should fix it so when that is merged you will see it on vscode insiders tomorrow. I have also tested this bug not on the rewrite and the issue does not appear at all so it seems you are in the rewrite since you are seeing this bug.

Secondly @karthiknadig any thoughts on why "python.experiments.optOutFrom": ["pythonTestAdapter"] isn't working?

Thank you for your patience @alisonatwork, we value professional developers like you and aim to make VS Code the best place for python developers so your input is always very helpful!

[karthiknadig]

@eleanorjboyd We need the initial part of the logs that captures the experiment states to determine what is going on.

[alisonatwork]

@karthiknadig here is the inital part of my logs when opening the project for the first time:

2023-09-08 11:15:08.454 [info] Experiment 'pythonPromptNewFormatterExt' is active
2023-09-08 11:15:08.454 [info] Experiment 'pythonPromptNewToolsExt' is active
2023-09-08 11:15:08.454 [info] Experiment 'pythonSurveyNotification' is active
2023-09-08 11:15:08.454 [info] Experiment 'pythonTestAdapter' is active
2023-09-08 11:15:08.454 [info] Test server listening.
2023-09-08 11:15:09.218 [info] > ./.venv/bin/python -I ~/.vscode-server/extensions/ms-python.python-2023.16.0/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2023.16.0/pythonFiles/interpreterInfo.py
2023-09-08 11:15:09.261 [info] Python interpreter path: ./.venv/bin/python
2023-09-08 11:15:11.150 [info] Starting Pylance language server

Thanks for looking into this issue and opening the PR, @eleanorjboyd!

[eleanorjboyd]

Hi @alisonatwork, I was taking a second look at this and don't think the fix I put in is correct. I realized that looking at your screenshot the result you got was unexpected- I thought the subtest name would have a parenthesis in it similar to this screenshot. Do you have any special settings that render square brackets instead?

[alisonatwork]

Hi @eleanorjboyd I am just back from vacation and able to look at this.

We have subtests that could potentially have a square bracket in the name, yes. But also there are plenty of subtests that could have curly brackets or round brackets or spaces or anything. The code I work on is proprietary, so I can't post it openly on GitHub, but one typical use case for subtests is testing that a parser function handles both good input and bad input. So you can imagine calling (for example) an HTML parser function and deliberately testing "weird" inputs like stray&ampersand, <unclosed tag, [square] and so on, and then it's nice if the name of the input also matches the name of the subtest. I would expect it to be able to write with self.subTest(f"with: {input}") and have it show up like this:

tests.py
├ test_the_function
├─ with: stray&ampersand
├─ with: <unclosed tag
├─ with: [square]
└ test_the_next_function

Personally I don't see any need for VS Code to add brackets or anything, because imo the best use of subTest is to simply use the msg string argument to describe what the test is testing. However, I know that some developers prefer to pass kwargs into subTest as the **params (see https://docs.python.org/3/library/unittest.html#unittest.TestCase.subTest), and then I imagine they would expect those to be rendered as stringified dict. So with self.subTest(input=input) should perhaps render something like:

tests.py
├ test_the_function
├─ {input='stray&ampersand'}
├─ {input='<unclosed tag'}
├─ {input='[square]'}
└ test_the_next_function

But from my perspective the top priority is just getting msg right, since at least then developers have a workaround to stringify their kwargs and use the msg parameter. Right now there is no workaround - no matter what you put it will truncate on the first space.

[eleanorjboyd]

Hi @alisonatwork, thank you for all your help with this- it is great to have someone so knowledgeable to test and provide feedback. I am seeing right now this is the result of the examples you gave. Does this look right? Or are the parenthesis still incorrect? I would normally just opt for not splitting the name of the subtest but they tend to run out of room on the sidebar if I keep the whole thing.

[alisonatwork]

Hi @eleanorjboyd, that new screenshot looks great!

I also commented on your merge request, but I thought about one more case that you should probably deal with, which is the combination of both msg (string) and params (kwargs) in the same subtest. I don't see it very often, but it is technically possible to do. Python unittest renders this by default with both square and round brackets.

Example that would work in your screenshot case: with self.subTest("item < 5", input = item). I think the way Python unittest will render it then would be: [item < 5] (input='stray&ampersand'). The use case for this might be to iterate through the same list of 20 inputs and then wrap each assertion in its own subtest, so you can test each assertion individually and have the test continue to the next one afterwards, even if one of them failed.