-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tool to run build libraries using LTS BOM #1982
Changes from all commits
0f01568
4fd8971
c388b00
f926c88
adf1085
f637b90
bb917bf
5b50624
66c6b3c
ef040b5
97046e4
4152bc1
b9a8f21
023392c
408f2ec
ccac387
55f158b
8e05714
891930d
a446102
dbb21b3
25cfcdb
01c9d12
55d00eb
d3fdb3b
bf2fe97
4173c41
f300c8a
8fe135c
916f17e
e5d892b
c30c907
575e41d
567ffa8
a9d5114
01b7ad7
44320c0
bfe382f
f10b629
bc72b7e
037acc8
4d39033
41fa787
fd4ce88
a905528
5d3284d
0b171ca
38924c6
cd7ad9f
ea32bfd
629030b
9bf8fcd
9b4e013
352e86b
0f71e00
e2dba2c
06417ab
412eaf8
aec92bf
c3b676f
3d78d6c
d660806
f3b7525
97bb4fb
4f0d76b
3b818ee
93f6108
ad4fac7
ec175bc
a5b4d58
ea1a55b
3477efc
ac7289d
82dcbf7
30b6651
d501d68
69f7212
ce9a5bf
f1c87ea
65fd97b
bf4de4c
3e974e8
3ab0661
e799d49
3d95172
f59e90a
5de080e
5e8e245
c4239ab
0a2aefd
1b2101c
bebee2e
55a40f3
9bd1968
0278f1c
9c802e2
d3c9dd6
30d6b22
0057f53
3e15562
53878e1
1d1cc19
c622483
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
on: | ||
pull_request: | ||
paths: | ||
- '.github/workflows/lts-test.yaml' | ||
- 'boms/cloud-lts-bom/pom.xml' | ||
- 'lts-test/**' | ||
- 'src/test/resources/repositories.yaml' | ||
name: LTS Library Test | ||
jobs: | ||
lts-library-test: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
repository: | ||
- beam | ||
- google-api-java-client | ||
- google-auth-library-java | ||
- google-http-java-client | ||
- google-maps-services-java | ||
- google-oauth-java-client | ||
- grpc-java | ||
- java-bigquery | ||
- java-bigtable-hbase | ||
- java-datastore | ||
- java-monitoring | ||
- java-pubsub | ||
- java-spanner | ||
- java-storage | ||
- java-trace | ||
steps: | ||
- uses: actions/checkout@v2 | ||
- uses: actions/setup-java@v1 | ||
with: | ||
java-version: 8 | ||
- uses: ./lts-test | ||
with: | ||
repository: ${{ matrix.repository }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -49,7 +49,6 @@ | |
<dependency> | ||
<groupId>xom</groupId> | ||
<artifactId>xom</artifactId> | ||
<version>1.3.7</version> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The version is now set at root pom.xml |
||
<scope>test</scope> | ||
</dependency> | ||
<dependency> | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
This tool is an automated test to check the compatibility of the libraries in | ||
`boms/cloud-lts-bom/pom.xml`. | ||
|
||
# How to execute tests in a repository | ||
|
||
When you want to execute the unit tests, use `lts.test.repository` property. | ||
The value corresponds to the "name" field in `lts-test/src/test/resources/repositories.yaml`. | ||
For example, to run the tests for beam repository, run the following command: | ||
|
||
``` | ||
$ mvn -pl dependencies,lts-test test -Dlts.test.repository=beam \ | ||
-Dtest=com.google.cloud.tools.opensource.lts.LtsBomCompatibilityTest \ | ||
-DfailIfNoTests=false | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
name: 'LTS Library Test' | ||
description: 'Checks the compatibility of the libraries in the LTS BOM' | ||
inputs: | ||
repository: | ||
description: 'The repository to test in this run. Use matrix to specify the values' | ||
required: true | ||
version_override_key: | ||
description: > | ||
The property name in the LTS BOM to test a certain version of a library. Use this input | ||
when you check the compatibility of a library against other libraries in the LTS BOM. | ||
required: false | ||
default: 'unspecified' | ||
version_override_value: | ||
description: > | ||
The version to test its compatibility with other libraries in the LTS BOM. The value must be | ||
unavailable in Maven Central because Gradle might not read local Maven repository first. | ||
required: false | ||
default: '0.1.0-SNAPSHOT' | ||
Comment on lines
+7
to
+18
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These are the input parameters for a library repository to override the version in the BOM. I added this configuration to emphasize that the LTS BOM test in this repository is a special case of the library compatibility test that will run in libraries' repository. If you prefer, I can remove the line 7-18, and 22-37. |
||
runs: | ||
using: "composite" | ||
steps: | ||
- name: Modify a version of a library in the LTS BOM if version override is specified | ||
run: | | ||
if [[ "${{ inputs.version_override_key }}" == "unspecified" ]]; then | ||
echo "inputs.version_override_key is unspecified. Skipping this step." | ||
exit | ||
fi | ||
cd .. | ||
sudo apt-get install libxml2-utils | ||
xmllint --shell boms/cloud-lts-bom/pom.xml << EOF | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm surprised this isn't already in the image. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I hoped it's there. But it requires the installation. |
||
setns x=http://maven.apache.org/POM/4.0.0 | ||
cd x:project/x:properties/x:${{ inputs.version_override_key }} | ||
set ${{ inputs.version_override_value }} | ||
save | ||
EOF | ||
shell: bash | ||
working-directory: ${{ github.action_path }} | ||
Comment on lines
+22
to
+37
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This step modifies the local LTS BOM to use a certain version of a library when specified. For example, this PR for POC https://github.com/googleapis/google-http-java-client/pull/1364/files#diff-a87cbbb4d5466c9d5b78b54ef875851d961bbbeed15a052016fb1d389b9c11cdR50 overrides |
||
- name: Install the snapshot LTS BOM (${{ github.action_path }}) to local Maven repository | ||
run: | | ||
cd .. | ||
mvn -B -ntp -pl boms/cloud-lts-bom install | ||
shell: bash | ||
working-directory: ${{ github.action_path }} | ||
- name: Run the unit tests for ${{ inputs.repository }} using repositories.yaml | ||
run: | | ||
cd .. | ||
mvn -B -ntp -pl dependencies,lts-test test \ | ||
-Dtest=com.google.cloud.tools.opensource.lts.LtsBomCompatibilityTest \ | ||
-Dlts.test.repository=${{ inputs.repository }} \ | ||
-DfailIfNoTests=false | ||
shell: bash | ||
working-directory: ${{ github.action_path }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,129 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<!-- | ||
Copyright 2021 Google LLC. | ||
|
||
Licensed under the Apache License, Version 2.0 (the "License"); | ||
you may not use this file except in compliance with the License. | ||
You may obtain a copy of the License at | ||
|
||
http://www.apache.org/licenses/LICENSE-2.0 | ||
|
||
Unless required by applicable law or agreed to in writing, software | ||
distributed under the License is distributed on an "AS IS" BASIS, | ||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
See the License for the specific language governing permissions and | ||
limitations under the License. | ||
--> | ||
|
||
<project xmlns="http://maven.apache.org/POM/4.0.0" | ||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
<modelVersion>4.0.0</modelVersion> | ||
<parent> | ||
<groupId>com.google.cloud.tools</groupId> | ||
<artifactId>dependencies-parent</artifactId> | ||
<version>1.5.10-SNAPSHOT</version> | ||
</parent> | ||
|
||
<artifactId>lts-test</artifactId> | ||
|
||
<name>LTS Test</name> | ||
<description>GitHub Actions to check the compatibility among the libraries in the LTS BOM</description> | ||
<url>https://github.com/GoogleCloudPlatform/cloud-opensource-java</url> | ||
<organization> | ||
<name>Google LLC.</name> | ||
<url>https://www.google.com</url> | ||
</organization> | ||
<inceptionYear>2021</inceptionYear> | ||
<developers> | ||
<developer> | ||
<name>Elliotte Rusty Harold</name> | ||
</developer> | ||
<developer> | ||
<name>Tomo Suzuki</name> | ||
</developer> | ||
</developers> | ||
|
||
<!-- Environment Settings --> | ||
<issueManagement> | ||
<url>https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues</url> | ||
</issueManagement> | ||
<scm> | ||
<connection>scm:git:[email protected]:GoogleCloudPlatform/cloud-opensource-java.git</connection> | ||
<developerConnection>scm:git:[email protected]:GoogleCloudPlatform/cloud-opensource-java.git | ||
</developerConnection> | ||
<url>https://github.com/GoogleCloudPlatform/cloud-opensource-java/tree/master/linkage-monitor</url> | ||
<tag>HEAD</tag> | ||
</scm> | ||
|
||
<licenses> | ||
<license> | ||
<name>The Apache License, Version 2.0</name> | ||
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url> | ||
</license> | ||
</licenses> | ||
|
||
<properties> | ||
<maven.compiler.source>1.8</maven.compiler.source> | ||
<maven.compiler.target>1.8</maven.compiler.target> | ||
</properties> | ||
|
||
<dependencies> | ||
<dependency> | ||
<groupId>${project.groupId}</groupId> | ||
<artifactId>dependencies</artifactId> | ||
<version>${project.version}</version> | ||
</dependency> | ||
<dependency> | ||
<groupId>com.google.guava</groupId> | ||
<artifactId>guava</artifactId> | ||
</dependency> | ||
<dependency> | ||
<groupId>commons-cli</groupId> | ||
<artifactId>commons-cli</artifactId> | ||
</dependency> | ||
<dependency> | ||
<groupId>org.yaml</groupId> | ||
<artifactId>snakeyaml</artifactId> | ||
<version>1.21</version> | ||
</dependency> | ||
<dependency> | ||
<groupId>xom</groupId> | ||
<artifactId>xom</artifactId> | ||
</dependency> | ||
<dependency> | ||
<groupId>com.google.truth</groupId> | ||
<artifactId>truth</artifactId> | ||
<scope>test</scope> | ||
</dependency> | ||
<dependency> | ||
<groupId>com.google.guava</groupId> | ||
<artifactId>guava-testlib</artifactId> | ||
<scope>test</scope> | ||
</dependency> | ||
<dependency> | ||
<groupId>junit</groupId> | ||
<artifactId>junit</artifactId> | ||
<scope>test</scope> | ||
</dependency> | ||
<dependency> | ||
<groupId>org.mockito</groupId> | ||
<artifactId>mockito-core</artifactId> | ||
<scope>test</scope> | ||
</dependency> | ||
</dependencies> | ||
|
||
<build> | ||
<plugins> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-javadoc-plugin</artifactId> | ||
<version>3.2.0</version> | ||
<configuration> | ||
<!-- Skip; otherwise it fails due to no public class in this module --> | ||
<skip>true</skip> | ||
</configuration> | ||
</plugin> | ||
</plugins> | ||
</build> | ||
</project> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
/* | ||
* Copyright 2021 Google LLC. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package com.google.cloud.tools.opensource.lts; | ||
|
||
import com.google.cloud.tools.opensource.dependencies.Bom; | ||
import com.google.common.base.Verify; | ||
import com.google.common.collect.ImmutableList; | ||
import com.google.common.io.Files; | ||
import com.google.common.io.MoreFiles; | ||
import java.io.IOException; | ||
import java.nio.charset.StandardCharsets; | ||
import java.nio.file.Path; | ||
import java.nio.file.Paths; | ||
|
||
/** | ||
* Modifies Apache Beam's build files to use the libraries in the BOM when running the unit tests. | ||
* Due to its complex build configuration Apache Beam requires special handling. | ||
*/ | ||
class BeamProjectModifier implements BuildFileModifier { | ||
@Override | ||
public void modifyFiles(String name, Path projectDirectory, Bom bom) | ||
throws TestSetupFailureException { | ||
Iterable<Path> paths = MoreFiles.fileTraverser().breadthFirst(projectDirectory); | ||
try { | ||
for (Path path : paths) { | ||
if (path.endsWith("build.gradle")) { | ||
modifyBeamGradleFile(path, bom); | ||
} else if (path.endsWith("BeamModulePlugin.groovy")) { | ||
modifyBeamModulePlugin(path); | ||
} | ||
} | ||
} catch (IOException ex) { | ||
throw new TestSetupFailureException("Unable to modify the build file", ex); | ||
} | ||
} | ||
|
||
// build.gradle files that run as part of the test in the beam section of repositories.yaml | ||
private static final ImmutableList<Path> beamTestSubprojects = | ||
ImmutableList.of( | ||
Paths.get("sdks", "java", "core", "build.gradle"), | ||
Paths.get("sdks", "java", "io", "google-cloud-platform", "build.gradle"), | ||
Paths.get("sdks", "java", "extensions", "google-cloud-platform-core", "build.gradle"), | ||
Paths.get("runners", "google-cloud-dataflow-java", "build.gradle")); | ||
|
||
private static void modifyBeamGradleFile(Path gradleFile, Bom bom) throws IOException { | ||
|
||
// Beam already uses enforcedPlatform(google_cloud_platform_libraries_bom), which prevents | ||
// gcp-lts-bom's setting gRPC library version. | ||
if (beamTestSubprojects.stream().anyMatch(gradleFile::endsWith)) { | ||
String buildGradleContent = | ||
Files.asCharSource(gradleFile.toFile(), StandardCharsets.UTF_8).read(); | ||
|
||
String bomCoordinates = bom.getCoordinates(); | ||
buildGradleContent = | ||
buildGradleContent.replaceAll( | ||
"\ndependencies \\{", | ||
"\ndependencies {\n compile enforcedPlatform('" | ||
+ bomCoordinates | ||
+ "')\n" | ||
+ " testRuntime enforcedPlatform('" | ||
+ bomCoordinates | ||
+ "')\n" | ||
+ " testRuntime library.java.junit\n" | ||
+ " testRuntime library.java.hamcrest_core\n" | ||
// This shouldn't be needed. But without this, GcsUtilTest fails | ||
// with NoSuchMethodError on CacheBuilder.expireAfterWrite(Ljava/time/Duration;) | ||
+ " testRuntime \"com.google.guava:guava:30.1-jre\"\n" | ||
+ " testRuntime library.java.hamcrest_library\n"); | ||
|
||
// Tried compileOnly but analyzeTestClassesDependencies's configuratin cannot resolve | ||
// the dependencies. | ||
// https://github.com/GoogleCloudPlatform/cloud-opensource-java/pull/1982#discussion_r610878573 | ||
// buildGradleContent = | ||
// buildGradleContent.replaceAll( | ||
// "compile enforcedPlatform\\(library.java.google_cloud_platform_libraries_bom\\)", | ||
// "compileOnly enforcedPlatform(library.java.google_cloud_platform_libraries_bom)"); | ||
Files.asCharSink(gradleFile.toFile(), StandardCharsets.UTF_8).write(buildGradleContent); | ||
} | ||
} | ||
|
||
private static void modifyBeamModulePlugin(Path beamModulePluginFile) throws IOException { | ||
// We don't want Beam to `force` dependencies when we run tests, because the `force` overrides | ||
// library versions set in the gcp-lts-bom. | ||
// https://github.com/GoogleCloudPlatform/cloud-opensource-java/pull/1982#discussion_r611911325 | ||
String buildGradleContent = | ||
Files.asCharSource(beamModulePluginFile.toFile(), StandardCharsets.UTF_8).read(); | ||
|
||
String buildGradleContentTestRuntimeClassPathModified = | ||
buildGradleContent.replaceAll( | ||
"config.getName\\(\\) != \"errorprone\"", | ||
"![\"errorprone\", \"testRuntimeClasspath\"].contains(config.getName())"); | ||
|
||
Verify.verify( | ||
!buildGradleContentTestRuntimeClassPathModified.equals(buildGradleContent), | ||
"The step should modify BeamModulePlugin.groovy"); | ||
|
||
Files.asCharSink(beamModulePluginFile.toFile(), StandardCharsets.UTF_8) | ||
.write(buildGradleContentTestRuntimeClassPathModified); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When does this run?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It runs when a pull request of cloud-opensource-java repository touches files matching the paths below (line 3)