Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Formatting jobs fails in CI #57196

Closed
ViktorHofer opened this issue Aug 11, 2021 · 14 comments · Fixed by #59806
Closed

Formatting jobs fails in CI #57196

ViktorHofer opened this issue Aug 11, 2021 · 14 comments · Fixed by #59806
Assignees
@kunalspathak
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Milestone
7.0.0

Comments

@ViktorHofer
Copy link
Member

I'm not touching anything in the runtime and still the formatting job fails: https://dev.azure.com/dnceng/public/_build/results?buildId=1287580&view=logs&jobId=db5b9618-eefc-58e8-4554-fbce452d4dac&j=db5b9618-eefc-58e8-4554-fbce452d4dac&t=29bb2718-3138-535b-c6ac-1a4597023815

Downloading https://raw.githubusercontent.com/dotnet/jitutils/main/bootstrap.cmd to C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\bootstrap.cmd
Convert C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\bootstrap.cmd to Windows line endings
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\bootstrap.cmd
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Checked -o windows -c D:\a\1\s\src\coreclr --verbose --projects dll
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Checked -o windows -c D:\a\1\s\src\coreclr --verbose --projects standalone
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Checked -o windows -c D:\a\1\s\src\coreclr --verbose --projects crossgen
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Debug -o windows -c D:\a\1\s\src\coreclr --verbose --projects dll
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Debug -o windows -c D:\a\1\s\src\coreclr --verbose --projects standalone
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Debug -o windows -c D:\a\1\s\src\coreclr --verbose --projects crossgen
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Release -o windows -c D:\a\1\s\src\coreclr --verbose --projects dll
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Release -o windows -c D:\a\1\s\src\coreclr --verbose --projects standalone
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Release -o windows -c D:\a\1\s\src\coreclr --verbose --projects crossgen
Creating patch file D:\a\1\s\src\coreclr\format.patch
Deleting C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\bootstrap.cmd
There were errors in formatting. Please run jit-format locally with: 

	jit-format -a x64 -b Checked -o windows -c <absolute-path-to-coreclr> --verbose --fix --projects dll


Or download and apply generated patch:
1. From the GitHub 'Checks' page on the Pull Request, with the failing Formatting
   job selected (e.g., 'Formatting Linux x64'), click the 'View more details on
   Azure Pipelines' link.
2. Select the '1 artifact produced' at the end of the log.
3. Artifacts are located in alphabetical order, target artifact name is
   'format.<OS>.<architecture>.patch.'. Find appropriate format patch artifact.
4. On the right side of the artifact there is a 'More actions' menu shown by a
   vertical three-dot symbol. Click on it and select 'Download artifacts' option.
5. Unzip the patch file.
6. git apply format.patch
##[error](NETCORE_ENGINEERING_TELEMETRY=Build) Format job found errors, please apply the format patch.
##[error]The process 'C:\hostedtoolcache\windows\Python\3.9.6\x64\python.exe' failed with exit code 4294967295
@ViktorHofer ViktorHofer added this to the Future milestone Aug 11, 2021
@ghost
Copy link

ghost commented Aug 11, 2021

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

I'm not touching anything in the runtime and still the formatting job fails: https://dev.azure.com/dnceng/public/_build/results?buildId=1287580&view=logs&jobId=db5b9618-eefc-58e8-4554-fbce452d4dac&j=db5b9618-eefc-58e8-4554-fbce452d4dac&t=29bb2718-3138-535b-c6ac-1a4597023815

Downloading https://raw.githubusercontent.com/dotnet/jitutils/main/bootstrap.cmd to C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\bootstrap.cmd
Convert C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\bootstrap.cmd to Windows line endings
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\bootstrap.cmd
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Checked -o windows -c D:\a\1\s\src\coreclr --verbose --projects dll
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Checked -o windows -c D:\a\1\s\src\coreclr --verbose --projects standalone
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Checked -o windows -c D:\a\1\s\src\coreclr --verbose --projects crossgen
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Debug -o windows -c D:\a\1\s\src\coreclr --verbose --projects dll
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Debug -o windows -c D:\a\1\s\src\coreclr --verbose --projects standalone
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Debug -o windows -c D:\a\1\s\src\coreclr --verbose --projects crossgen
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Release -o windows -c D:\a\1\s\src\coreclr --verbose --projects dll
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Release -o windows -c D:\a\1\s\src\coreclr --verbose --projects standalone
Running: C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\jitutils\bin\jit-format.bat -a x64 -b Release -o windows -c D:\a\1\s\src\coreclr --verbose --projects crossgen
Creating patch file D:\a\1\s\src\coreclr\format.patch
Deleting C:\Users\VSSADM~1\AppData\Local\Temp\tmp9r8m9v72\bootstrap.cmd
There were errors in formatting. Please run jit-format locally with: 

	jit-format -a x64 -b Checked -o windows -c <absolute-path-to-coreclr> --verbose --fix --projects dll


Or download and apply generated patch:
1. From the GitHub 'Checks' page on the Pull Request, with the failing Formatting
   job selected (e.g., 'Formatting Linux x64'), click the 'View more details on
   Azure Pipelines' link.
2. Select the '1 artifact produced' at the end of the log.
3. Artifacts are located in alphabetical order, target artifact name is
   'format.<OS>.<architecture>.patch.'. Find appropriate format patch artifact.
4. On the right side of the artifact there is a 'More actions' menu shown by a
   vertical three-dot symbol. Click on it and select 'Download artifacts' option.
5. Unzip the patch file.
6. git apply format.patch
##[error](NETCORE_ENGINEERING_TELEMETRY=Build) Format job found errors, please apply the format patch.
##[error]The process 'C:\hostedtoolcache\windows\Python\3.9.6\x64\python.exe' failed with exit code 4294967295
Author: ViktorHofer
Assignees: -
Labels:

area-Infrastructure-coreclr
Milestone: Future

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Aug 11, 2021
@ViktorHofer ViktorHofer removed the untriaged New issue has not been triaged by the area owner label Aug 11, 2021
@ViktorHofer ViktorHofer modified the milestones: Future, 7.0.0 Aug 11, 2021
@ViktorHofer ViktorHofer mentioned this issue Aug 11, 2021
Merged
@hoyosjs hoyosjs added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed area-Infrastructure-coreclr labels Aug 11, 2021
@hoyosjs
Copy link
Member

hoyosjs commented Aug 11, 2021

Moving to codegen since formatting is only on JIT source.

@ViktorHofer
Copy link
Member Author

Failed again in https://github.com/dotnet/runtime/pull/57841/checks?check_run_id=3398783693

Presumably retries are still missing in relevant code paths?

Downloading clang-format to bin directory
--2021-08-23 09:31:53--  https://clrjit.blob.core.windows.net/clang-tools/centos.7-x64/clang-format
Resolving clrjit.blob.core.windows.net (clrjit.blob.core.windows.net)... 13.93.168.80
Connecting to clrjit.blob.core.windows.net (clrjit.blob.core.windows.net)|13.93.168.80|:443... connected.
HTTP request sent, awaiting response... 503 The server is busy.
2021-08-23 09:31:53 ERROR 503: The server is busy..

clang-tidy not found here: https://clrjit.blob.core.windows.net/clang-tools/centos.7-x64/clang-tidy
Either clang-tidy or clang-format was not installed. Please install and put them on the PATH to use jit-format.
Tools can be found at http://llvm.org/releases/download.html#3.8.0
ERROR: Failed to download clang-format and clang-tidy.
('Downloading', 'https://raw.githubusercontent.com/dotnet/jitutils/main/bootstrap.sh', 'to', '/tmp/tmps0siib/bootstrap.sh')
Making bootstrap executable
('Running:', 'bash', '/tmp/tmps0siib/bootstrap.sh')
Deleting /tmp/tmps0siib/bootstrap.sh
Bootstrap failed

@ViktorHofer
Copy link
Member Author

cc @JulieLeeMSFT @BruceForstall @dotnet/jit-contrib

@ViktorHofer ViktorHofer added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Aug 23, 2021
@ViktorHofer ViktorHofer mentioned this issue Aug 23, 2021
Merged
@runfoapp runfoapp bot mentioned this issue Aug 23, 2021
Closed
@kunalspathak
Copy link
Member

Seems to be an intermittent failure. I can see the https://clrjit.blob.core.windows.net/clang-tools/centos.7-x64/clang-tidy blob exist and recent builds passing.

@ViktorHofer
Copy link
Member Author

Does the code path that fetches the tool retry?

@kunalspathak
Copy link
Member

Does the code path that fetches the tool retry?

Yes. It was added in dotnet/jitutils#327.

@kunalspathak
Copy link
Member

Actually, now i read it is linux failure and yes, retry is missing in bootstrap.sh.

@kunalspathak kunalspathak reopened this Aug 23, 2021
@kunalspathak
Copy link
Member

dotnet/jitutils#336

@kunalspathak
Copy link
Member

Added retry for linux.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 22, 2021
@ViktorHofer
Copy link
Member Author

Reopening as it is still failing.

@ViktorHofer ViktorHofer reopened this Sep 30, 2021
@ViktorHofer
Copy link
Member Author

According to runfo multiple hits in the last 7 days: https://runfo.azurewebsites.net/search/build-logs/?bq=started%3A%7E7+definition%3Aruntime+result%3Afailed+kind%3Arolling&lq=text%3A%22Format+job+found+errors%22+logName%3A%22Formatting+windows+x64%22+limit%3A200

@ViktorHofer
Copy link
Member Author

cc @jeffschwMSFT as 25% of the failures in the last 7 days are due to this issue:

image

jakobbotsch added a commit to jakobbotsch/runtime that referenced this issue Sep 30, 2021
@jakobbotsch
Fix retry when downloading clang tools
4f2331d 
Invoke-WebRequest throws an exception when the download fails, so we
should use try-catch instead of status code to check failure.

In addition pass -PassThru to avoid leaving a corrupted download file in
case the download fails. This will buffer the download and write it once
at the end.

Fix dotnet#57196
@BruceForstall
Copy link
Member

cc @jkoritzinsky

PR to fix: #59806

jakobbotsch added a commit that referenced this issue Sep 30, 2021
@jakobbotsch
Fix retry when downloading clang tools (#59806)
5e4bd40 
Invoke-WebRequest throws an exception when the download fails, so we
should use try-catch instead of status code to check failure. We can use
the generic Retry function from tools.ps1 to do this.

In addition pass -PassThru to avoid leaving a corrupted download file in
case the download fails. This will buffer the download and write it once
at the end.

Fix #57196
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kunalspathak kunalspathak

Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Projects
None yet
Milestone
7.0.0
Development

Successfully merging a pull request may close this issue.

Fix retry when downloading clang tools jakobbotsch/runtime
4 participants
@ViktorHofer @BruceForstall @kunalspathak @hoyosjs