Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dump Heap on OOM and Retain Heap Dump #18392

Merged
merged 36 commits into from
Sep 14, 2021
Merged

Dump Heap on OOM and Retain Heap Dump #18392

Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
b9b159d
Attempt to add heap dump on OOM and retain it
alzimmermsft Dec 29, 2020
d496ca3
Add validation that a heap dump was created before attempting to publ…
alzimmermsft Dec 30, 2020
3d1e0e1
Fix task condition
alzimmermsft Dec 30, 2020
2798a27
Merge branch 'master' into AzEng_RetainHeapDumpOnOoM
alzimmermsft Jan 8, 2021
cc12817
Prototype passing additional JVM arguments to the Surefire JVM
alzimmermsft Jan 9, 2021
9eef02f
Fix property name, add support for command-line settable Xmx value
alzimmermsft Jan 9, 2021
2dbe305
Merge branch 'master' into AzEng_RetainHeapDumpOnOoM
alzimmermsft Jan 11, 2021
65ee818
Update Surefire version
alzimmermsft Jan 11, 2021
489708e
Merge branch 'master' into AzEng_RetainHeapDumpOnOoM
alzimmermsft Jan 12, 2021
9fe18c9
Update heap dump in pom configuration and yaml checks for it
alzimmermsft Jan 12, 2021
fbdd133
Support retaining multiple heap dumps, changed heap dump path to a di…
alzimmermsft Jan 12, 2021
bbc950e
Fix expression syntax
alzimmermsft Jan 12, 2021
f827805
Test other ways to track heap dump files
alzimmermsft Jan 12, 2021
1994463
Add logging to troubleshoot
alzimmermsft Jan 12, 2021
c944800
Merge branch 'master' into AzEng_RetainHeapDumpOnOoM
alzimmermsft Jan 12, 2021
5ee5c90
Use Maven to generate directory for heap dumps
alzimmermsft Jan 12, 2021
06a2e90
Update heap dump path, remove CI attempt to retain heap dump for furt…
alzimmermsft Mar 29, 2021
d4ac216
Merge branch 'master' into AzEng_RetainHeapDumpOnOoM
alzimmermsft Apr 5, 2021
792747f
Merge in main
alzimmermsft Aug 9, 2021
f7abba4
Attempt to publish OOM hprofs
alzimmermsft Aug 9, 2021
382eb6a
Merge branch 'main' into AzEng_RetainHeapDumpOnOoM
alzimmermsft Sep 13, 2021
5789ade
Change configuration structure to be centralized in parent POM
alzimmermsft Sep 13, 2021
7677aaf
Update all Core libraries to use new pattern, changed how OOM hprof i…
alzimmermsft Sep 13, 2021
7358731
Fix missing '-' in command name
alzimmermsft Sep 13, 2021
5734a26
Add logging of file copying
alzimmermsft Sep 13, 2021
00aedb6
Change file copying logic
alzimmermsft Sep 13, 2021
55f0879
Publish retain hrpofs
alzimmermsft Sep 13, 2021
0260a40
Zip hprofs before publishing
alzimmermsft Sep 13, 2021
275075e
Merge branch 'main' into AzEng_RetainHeapDumpOnOoM
alzimmermsft Sep 13, 2021
6f5cc82
Turn retention into a standalone step for re-use
alzimmermsft Sep 13, 2021
5d4eaaf
Remove OOM tests as changes have been validated
alzimmermsft Sep 13, 2021
749b4aa
Change Copy-Item to Move-Item
alzimmermsft Sep 14, 2021
6850e35
Fix merge conflict with upstream
alzimmermsft Sep 14, 2021
415f3f7
Merge branch 'main' into AzEng_RetainHeapDumpOnOoM
alzimmermsft Sep 14, 2021
44689fb
Tweak to when the task runs
alzimmermsft Sep 14, 2021
036002b
Match heap OOM retention with test publishing
alzimmermsft Sep 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions eng/pipelines/templates/jobs/ci.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,8 @@ jobs:

- template: ../steps/upload-repository-on-failure.yml

- template: ../steps/retain-heap-dump-hprofs.yml

- task: PublishTestResults@2
condition: and(always(), or(ne(variables['TestFromSource'],'true'), eq(variables['ShouldRunSourceTests'],'true')))
inputs:
Expand Down
2 changes: 2 additions & 0 deletions eng/pipelines/templates/jobs/live.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ jobs:
${{ each var in parameters.EnvVars }}:
${{ var.key }}: ${{ var.value }}

- template: ../steps/retain-heap-dump-hprofs.yml

- ${{ parameters.PostSteps }}

- ${{ if ne(parameters.DisableAzureResourceCreation, 'true') }}:
Expand Down
17 changes: 17 additions & 0 deletions eng/pipelines/templates/steps/retain-heap-dump-hprofs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
steps:
- pwsh: |
New-Item $(Build.ArtifactStagingDirectory)/oom-hprofs -ItemType directory
foreach($hprofFile in (Get-ChildItem -Path . -Recurse -Filter *.hprof -File))
{
$fileFullName = $hprofFile.FullName
weshaggard marked this conversation as resolved.
Show resolved Hide resolved
$fileName = $hprofFile.Name
Copy-Item $fileFullName $(Build.ArtifactStagingDirectory)/oom-hprofs/$fileName -ErrorAction SilentlyContinue
alzimmermsft marked this conversation as resolved.
Show resolved Hide resolved
}
[System.IO.Compression.ZipFile]::CreateFromDirectory("$(Build.ArtifactStagingDirectory)/oom-hprofs","$(Build.ArtifactStagingDirectory)/oom-hprofs.zip")
alzimmermsft marked this conversation as resolved.
Show resolved Hide resolved
displayName: 'Copy OOM hprofs'
condition: always()
alzimmermsft marked this conversation as resolved.
Show resolved Hide resolved

- publish: $(Build.ArtifactStagingDirectory)/oom-hprofs.zip
displayName: 'Publish OOM hprofs'
artifact: oom-hprofs-$(System.StageName)-$(System.JobName)-$(System.JobAttempt)
condition: always()
alzimmermsft marked this conversation as resolved.
Show resolved Hide resolved
23 changes: 7 additions & 16 deletions sdk/core/azure-core-amqp/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,23 +152,14 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-opens com.azure.core.amqp/com.azure.core.amqp.implementation=ALL-UNNAMED
--add-opens com.azure.core.amqp/com.azure.core.amqp.implementation.handler=ALL-UNNAMED
<properties>
<javaModulesSurefireArgLine>
--add-opens com.azure.core.amqp/com.azure.core.amqp.implementation=ALL-UNNAMED
--add-opens com.azure.core.amqp/com.azure.core.amqp.implementation.handler=ALL-UNNAMED

--add-reads com.azure.core.amqp=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
--add-reads com.azure.core.amqp=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>
</project>
21 changes: 6 additions & 15 deletions sdk/core/azure-core-experimental/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,21 +116,12 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-opens com.azure.core.experimental/com.azure.core.experimental.http=ALL-UNNAMED
--add-opens com.azure.core.experimental/com.azure.core.experimental.models=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-opens com.azure.core.experimental/com.azure.core.experimental.http=ALL-UNNAMED
--add-opens com.azure.core.experimental/com.azure.core.experimental.models=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>
</project>
21 changes: 6 additions & 15 deletions sdk/core/azure-core-http-netty/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,21 +218,12 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-opens com.azure.http.netty/com.azure.core.http.netty=ALL-UNNAMED
--add-opens com.azure.http.netty/com.azure.core.http.netty.implementation=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-opens com.azure.http.netty/com.azure.core.http.netty=ALL-UNNAMED
--add-opens com.azure.http.netty/com.azure.core.http.netty.implementation=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>
</project>
21 changes: 6 additions & 15 deletions sdk/core/azure-core-http-okhttp/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,21 +166,12 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-opens com.azure.core.http.okhttp/com.azure.core.http.okhttp=ALL-UNNAMED
--add-opens com.azure.core.http.okhttp/com.azure.core.http.okhttp.implementation=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-opens com.azure.core.http.okhttp/com.azure.core.http.okhttp=ALL-UNNAMED
--add-opens com.azure.core.http.okhttp/com.azure.core.http.okhttp.implementation=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>
</project>
19 changes: 5 additions & 14 deletions sdk/core/azure-core-management/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,20 +123,11 @@
<activation>
<jdk>[9,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-opens com.azure.core.management/com.azure.core.management.implementation.polling=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-opens com.azure.core.management/com.azure.core.management.implementation.polling=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>

Expand Down
21 changes: 6 additions & 15 deletions sdk/core/azure-core-serializer-avro-apache/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,21 +156,12 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-exports com.azure.core.serializer.avro.apache/com.azure.core.serializer.avro.apache.generatedtestsources=org.apache.avro
--add-opens com.azure.core.serializer.avro.apache/com.azure.core.serializer.avro.apache=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-exports com.azure.core.serializer.avro.apache/com.azure.core.serializer.avro.apache.generatedtestsources=org.apache.avro
--add-opens com.azure.core.serializer.avro.apache/com.azure.core.serializer.avro.apache=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>
</project>
21 changes: 6 additions & 15 deletions sdk/core/azure-core-serializer-avro-jackson/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,21 +144,12 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-exports com.azure.core.serializer.avro.jackson/com.azure.core.serializer.avro.jackson.generatedtestsources=com.fasterxml.jackson.databind
--add-opens com.azure.core.serializer.avro.jackson/com.azure.core.serializer.avro.jackson=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-exports com.azure.core.serializer.avro.jackson/com.azure.core.serializer.avro.jackson.generatedtestsources=com.fasterxml.jackson.databind
--add-opens com.azure.core.serializer.avro.jackson/com.azure.core.serializer.avro.jackson=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>
</project>
19 changes: 5 additions & 14 deletions sdk/core/azure-core-serializer-json-gson/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,20 +135,11 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-opens com.azure.core.serializer.json.gson/com.azure.core.serializer.json.gson=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-opens com.azure.core.serializer.json.gson/com.azure.core.serializer.json.gson=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>
</project>
23 changes: 7 additions & 16 deletions sdk/core/azure-core-serializer-json-jackson/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,22 +114,13 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-exports com.azure.core.serializer.json.jackson/com.azure.core.serializer.json.jackson=com.fasterxml.jackson.databind
--add-opens com.azure.core.serializer.json.jackson/com.azure.core.serializer.json.jackson=com.fasterxml.jackson.databind
--add-opens com.azure.core.serializer.json.jackson/com.azure.core.serializer.json.jackson=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-exports com.azure.core.serializer.json.jackson/com.azure.core.serializer.json.jackson=com.fasterxml.jackson.databind
--add-opens com.azure.core.serializer.json.jackson/com.azure.core.serializer.json.jackson=com.fasterxml.jackson.databind
--add-opens com.azure.core.serializer.json.jackson/com.azure.core.serializer.json.jackson=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>

Expand Down
29 changes: 10 additions & 19 deletions sdk/core/azure-core-test/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,25 +143,16 @@
<activation>
<jdk>[11,)</jdk>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-exports org.junit.platform.commons/org.junit.platform.commons.util=ALL-UNNAMED
--add-exports org.junit.platform.commons/org.junit.platform.commons.logging=ALL-UNNAMED

--add-reads com.azure.core.test=ALL-UNNAMED
--add-opens com.azure.core.test/com.azure.core.test.models=ALL-UNNAMED
--add-opens com.azure.core.test/com.azure.core.test.implementation=org.junit.platform.commons
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-exports org.junit.platform.commons/org.junit.platform.commons.util=ALL-UNNAMED
--add-exports org.junit.platform.commons/org.junit.platform.commons.logging=ALL-UNNAMED

--add-reads com.azure.core.test=ALL-UNNAMED
--add-opens com.azure.core.test/com.azure.core.test.models=ALL-UNNAMED
--add-opens com.azure.core.test/com.azure.core.test.implementation=org.junit.platform.commons
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>
</project>
19 changes: 5 additions & 14 deletions sdk/core/azure-core-tracing-opentelemetry/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,20 +126,11 @@
<activation>
<jdk>[9,)</jdk>
alzimmermsft marked this conversation as resolved.
Show resolved Hide resolved
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version> <!-- {x-version-update;org.apache.maven.plugins:maven-surefire-plugin;external_dependency} -->
<configuration>
<argLine>
--add-opens com.azure.core.tracing.opentelemetry/com.azure.core.tracing.opentelemetry.implementation=ALL-UNNAMED
</argLine>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<javaModulesSurefireArgLine>
--add-opens com.azure.core.tracing.opentelemetry/com.azure.core.tracing.opentelemetry.implementation=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>
</profile>
</profiles>

Expand Down
Loading