Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle paths case-insensitively when running tests #4395

Merged
merged 2 commits into from
Feb 27, 2024
Merged

Handle paths case-insensitively when running tests #4395

merged 2 commits into from
Feb 27, 2024

Conversation

StephanTLavavej
Copy link
Member

Fixes #4393.

@AlexGuteniev discovered that attempting to run tests with a drive letter of a different case would fail (see lowercase d:):

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\std\tests\VSO_0157762_feature_test_macros
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'd:\\GitHub\\STL\\tests\\std\\tests\\VSO_0157762_feature_test_macros' contained no tests
error: did not discover any tests for provided path(s)

Later, I discovered that case variation in the middle of the path would also fail (see uppercase STD):

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\STD\tests\VSO_0157762_feature_test_macros
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'D:\\GitHub\\STL\\tests\\STD\\tests\\VSO_0157762_feature_test_macros' contained no tests
error: did not discover any tests for provided path(s)

According to my understanding, this is happening because LLVM's lit machinery isn't performing case normalization before ultimately calling our code, which is performing exact equality checks with ==. We can fix this on our side by using os.path.samefile:

Return True if both pathname arguments refer to the same file or directory. This is determined by the device number and i-node number and raises an exception if an os.stat() call on either pathname fails.

This fixes the problematic scenarios, while not affecting what happens if the user specifies a totally nonexistent test.

Click to expand main's behavior: 
D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\std\tests\VSO_0157762_feature_test_macros
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\std\tests\VSO_0157762_feature_test_macros
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'd:\\GitHub\\STL\\tests\\std\\tests\\VSO_0157762_feature_test_macros' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py ..\..\tests\std\tests\VSO_0157762_feature_test_macros
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>:: Change the middle to STD:
D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\STD\tests\VSO_0157762_feature_test_macros
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'D:\\GitHub\\STL\\tests\\STD\\tests\\VSO_0157762_feature_test_macros' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\STD\tests\VSO_0157762_feature_test_macros
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'd:\\GitHub\\STL\\tests\\STD\\tests\\VSO_0157762_feature_test_macros' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py ..\..\tests\STD\tests\VSO_0157762_feature_test_macros
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input '..\\..\\tests\\STD\\tests\\VSO_0157762_feature_test_macros' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>:: Change the end to vso_0157762_FEATURE_TEST_MACROS:
D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\std\tests\vso_0157762_FEATURE_TEST_MACROS
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\std\tests\vso_0157762_FEATURE_TEST_MACROS
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'd:\\GitHub\\STL\\tests\\std\\tests\\vso_0157762_FEATURE_TEST_MACROS' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py ..\..\tests\std\tests\vso_0157762_FEATURE_TEST_MACROS
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>:: Nonexistent test:
D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\std\tests\UNATCO_0451_helios
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'D:\\GitHub\\STL\\tests\\std\\tests\\UNATCO_0451_helios' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\std\tests\UNATCO_0451_helios
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'd:\\GitHub\\STL\\tests\\std\\tests\\UNATCO_0451_helios' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py ..\..\tests\std\tests\UNATCO_0451_helios
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input '..\\..\\tests\\std\\tests\\UNATCO_0451_helios' contained no tests
error: did not discover any tests for provided path(s)
Click to expand this PR's behavior: 
D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\std\tests\VSO_0157762_feature_test_macros
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\std\tests\VSO_0157762_feature_test_macros
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py ..\..\tests\std\tests\VSO_0157762_feature_test_macros
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>:: Change the middle to STD:
D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\STD\tests\VSO_0157762_feature_test_macros
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\STD\tests\VSO_0157762_feature_test_macros
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py ..\..\tests\STD\tests\VSO_0157762_feature_test_macros
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>:: Change the end to vso_0157762_FEATURE_TEST_MACROS:
D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\std\tests\vso_0157762_FEATURE_TEST_MACROS
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\std\tests\vso_0157762_FEATURE_TEST_MACROS
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py ..\..\tests\std\tests\vso_0157762_FEATURE_TEST_MACROS
-- Testing: 35 tests, 32 workers --
[...]

D:\GitHub\STL\out\x64>:: Nonexistent test:
D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py D:\GitHub\STL\tests\std\tests\UNATCO_0451_helios
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'D:\\GitHub\\STL\\tests\\std\\tests\\UNATCO_0451_helios' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py d:\GitHub\STL\tests\std\tests\UNATCO_0451_helios
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input 'd:\\GitHub\\STL\\tests\\std\\tests\\UNATCO_0451_helios' contained no tests
error: did not discover any tests for provided path(s)

D:\GitHub\STL\out\x64>python tests\utils\stl-lit\stl-lit.py ..\..\tests\std\tests\UNATCO_0451_helios
stl-lit.py: D:\GitHub\STL\llvm-project\llvm\utils\lit\lit\discovery.py:276: warning: input '..\\..\\tests\\std\\tests\\UNATCO_0451_helios' contained no tests
error: did not discover any tests for provided path(s)

@StephanTLavavej StephanTLavavej added the test Related to test code label Feb 16, 2024
@StephanTLavavej StephanTLavavej requested a review from a team as a code owner February 16, 2024 20:27
@StephanTLavavej StephanTLavavej self-assigned this Feb 23, 2024
@StephanTLavavej
Copy link
Member Author

I'm mirroring this to the MSVC-internal repo - please notify me if any further changes are pushed.

@StephanTLavavej StephanTLavavej merged commit 8d3be34 into microsoft:main Feb 27, 2024
35 checks passed
@StephanTLavavej StephanTLavavej deleted the snakes-on-a-repo branch February 27, 2024 05:35
@StephanTLavavej StephanTLavavej mentioned this pull request Apr 22, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CaseyCarter CaseyCarter CaseyCarter approved these changes

@AlexGuteniev AlexGuteniev AlexGuteniev approved these changes

Assignees

@StephanTLavavej StephanTLavavej

Labels
test Related to test code
Projects
STL Code Reviews
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

stl-lit refuses to run tests using small drive letter in test path
3 participants
@StephanTLavavej @CaseyCarter @AlexGuteniev