Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remove airbyte-commons-docker #21756

Merged
merged 18 commits into from
Feb 1, 2023
Merged

remove airbyte-commons-docker #21756

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
5c9d534
remove references to airbyte-commons-docker
colesnodgrass Jan 23, 2023
8d3fcf0
remove commons-docker
colesnodgrass Jan 23, 2023
ed7a852
update missed DockerUtils reference
colesnodgrass Jan 23, 2023
41e03d3
Merge remote-tracking branch 'origin/master' into cole/rem-commons-do…
colesnodgrass Jan 23, 2023
843eb04
Merge remote-tracking branch 'origin/master' into cole/rem-commons-do…
colesnodgrass Jan 24, 2023
4a2ae44
Merge remote-tracking branch 'origin/master' into cole/rem-commons-do…
colesnodgrass Jan 24, 2023
9ec6e3a
remove commons-docker that was re-added
colesnodgrass Jan 24, 2023
d35a452
Merge remote-tracking branch 'origin/master' into cole/rem-commons-do…
colesnodgrass Jan 24, 2023
9a462f1
Merge remote-tracking branch 'origin/master' into cole/rem-commons-do…
colesnodgrass Jan 24, 2023
4d805ba
remove final references to airbyte-commons-docker
colesnodgrass Jan 24, 2023
16adbba
add bouncycastle test
colesnodgrass Jan 24, 2023
b5eeae3
add bouncycastle test
colesnodgrass Jan 24, 2023
9ba8389
Merge remote-tracking branch 'origin/master' into cole/rem-commons-do…
colesnodgrass Jan 24, 2023
de8e78d
Merge remote-tracking branch 'origin/master' into cole/rem-commons-do…
colesnodgrass Jan 30, 2023
1255437
Merge remote-tracking branch 'origin/master' into cole/rem-commons-do…
colesnodgrass Jan 30, 2023
f4b5127
Merge branch 'master' into cole/rem-commons-docker
colesnodgrass Jan 31, 2023
cec8d02
Merge branch 'master' into cole/rem-commons-docker
colesnodgrass Jan 31, 2023
1d17a10
add bouncycastle to acceptance-tests
colesnodgrass Jan 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/workflows/gradle.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -727,7 +727,6 @@ jobs:
${{ github.workspace }}/airbyte-bootloader/build/reports/jacoco/test/jacocoTestReport.xml,
${{ github.workspace }}/airbyte-commons/build/reports/jacoco/test/jacocoTestReport.xml,
${{ github.workspace }}/airbyte-commons-cli/build/reports/jacoco/test/jacocoTestReport.xml,
${{ github.workspace }}/airbyte-commons-docker/build/reports/jacoco/test/jacocoTestReport.xml,
${{ github.workspace }}/airbyte-commons-protocol/build/reports/jacoco/test/jacocoTestReport.xml,
${{ github.workspace }}/airbyte-commons-temporal/build/reports/jacoco/test/jacocoTestReport.xml,
${{ github.workspace }}/airbyte-commons-worker/build/reports/jacoco/test/jacocoTestReport.xml,
Expand Down Expand Up @@ -1157,7 +1156,7 @@ jobs:
helm-acceptance-test:
name: "Platform: Acceptance Tests (Helm)"
# In case of self-hosted EC2 errors, removed the `needs` line and switch back to running on ubuntu-latest.
needs: [start-helm-acceptance-test-runner] # required to start the main job when the runner is ready
needs: [ start-helm-acceptance-test-runner ] # required to start the main job when the runner is ready
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change expected? Shouldn't it be

needs:
     - start-helm-acceptance-test-runner

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the formatter adding some spaces between the square-braces. From the yaml parsing POV, there isn't a difference between

needs: [ value ]

and

needs:
  - value

runs-on: ${{ needs.start-helm-acceptance-test-runner.outputs.label }} # run the job on the newly created runner
# this is the label of the runner
environment: more-secrets
Expand Down
13 changes: 0 additions & 13 deletions airbyte-commons-docker/build.gradle
View file
Open in desktop

This file was deleted.

3 changes: 0 additions & 3 deletions airbyte-commons-docker/readme.md
View file
Open in desktop

This file was deleted.

39 changes: 0 additions & 39 deletions airbyte-commons-docker/src/main/java/io/airbyte/commons/docker/DockerUtils.java
View file
Open in desktop

This file was deleted.

20 changes: 0 additions & 20 deletions airbyte-commons-docker/src/test/java/io/airbyte/commons/docker/DockerUtilsTest.java
View file
Open in desktop

This file was deleted.

3 changes: 1 addition & 2 deletions airbyte-commons-server/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ dependencies {

implementation project(':airbyte-analytics')
implementation project(':airbyte-api')
implementation project(':airbyte-commons-docker')
implementation project(':airbyte-commons-temporal')
implementation project(':airbyte-commons-worker')
implementation project(':airbyte-config:init')
Expand Down Expand Up @@ -66,4 +65,4 @@ dependencies {
testImplementation 'org.mockito:mockito-inline:4.7.0'
}

Task publishArtifactsTask = getPublishArtifactsTask("$rootProject.ext.version", project)
Task publishArtifactsTask = getPublishArtifactsTask("$rootProject.ext.version", project)
9 changes: 6 additions & 3 deletions ...erver/src/main/java/io/airbyte/commons/server/handlers/DestinationDefinitionsHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
import io.airbyte.api.model.generated.PrivateDestinationDefinitionReadList;
import io.airbyte.api.model.generated.ReleaseStage;
import io.airbyte.api.model.generated.WorkspaceIdRequestBody;
import io.airbyte.commons.docker.DockerUtils;
import io.airbyte.commons.resources.MoreResources;
import io.airbyte.commons.server.ServerConstants;
import io.airbyte.commons.server.converters.ApiPojoConverters;
Expand All @@ -32,7 +31,11 @@
import io.airbyte.commons.version.AirbyteProtocolVersion;
import io.airbyte.commons.version.AirbyteProtocolVersionRange;
import io.airbyte.commons.version.Version;
import io.airbyte.config.*;
import io.airbyte.config.ActorDefinitionResourceRequirements;
import io.airbyte.config.ActorType;
import io.airbyte.config.Configs;
import io.airbyte.config.EnvConfigs;
import io.airbyte.config.StandardDestinationDefinition;
import io.airbyte.config.persistence.ConfigNotFoundException;
import io.airbyte.config.persistence.ConfigRepository;
import io.airbyte.protocol.models.ConnectorSpecification;
Expand Down Expand Up @@ -295,7 +298,7 @@ public void deleteDestinationDefinition(final DestinationDefinitionIdRequestBody

private ConnectorSpecification getSpecForImage(final String dockerRepository, final String imageTag, final boolean isCustomConnector)
throws IOException {
final String imageName = DockerUtils.getTaggedImageName(dockerRepository, imageTag);
final String imageName = dockerRepository + ":" + imageTag;
final SynchronousResponse<ConnectorSpecification> getSpecResponse = schedulerSynchronousClient.createGetSpecJob(imageName, isCustomConnector);
return SpecFetcher.getSpecFromJob(getSpecResponse);
}
Expand Down
13 changes: 6 additions & 7 deletions ...yte-commons-server/src/main/java/io/airbyte/commons/server/handlers/SchedulerHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@
import io.airbyte.api.model.generated.StreamTransform;
import io.airbyte.api.model.generated.StreamTransform.TransformTypeEnum;
import io.airbyte.api.model.generated.SynchronousJobRead;
import io.airbyte.commons.docker.DockerUtils;
import io.airbyte.commons.enums.Enums;
import io.airbyte.commons.features.FeatureFlags;
import io.airbyte.commons.json.Jsons;
Expand Down Expand Up @@ -153,7 +152,7 @@ public CheckConnectionRead checkSourceConnectionFromSourceId(final SourceIdReque
throws ConfigNotFoundException, IOException, JsonValidationException {
final SourceConnection source = configRepository.getSourceConnection(sourceIdRequestBody.getSourceId());
final StandardSourceDefinition sourceDef = configRepository.getStandardSourceDefinition(source.getSourceDefinitionId());
final String imageName = DockerUtils.getTaggedImageName(sourceDef.getDockerRepository(), sourceDef.getDockerImageTag());
final String imageName = sourceDef.getDockerRepository() + ":" + sourceDef.getDockerImageTag();
final boolean isCustomConnector = sourceDef.getCustom();
final Version protocolVersion = new Version(sourceDef.getProtocolVersion());

Expand All @@ -177,7 +176,7 @@ public CheckConnectionRead checkSourceConnectionFromSourceCreate(final SourceCor

final Version protocolVersion = new Version(sourceDef.getProtocolVersion());

final String imageName = DockerUtils.getTaggedImageName(sourceDef.getDockerRepository(), sourceDef.getDockerImageTag());
final String imageName = sourceDef.getDockerRepository() + ":" + sourceDef.getDockerImageTag();
final boolean isCustomConnector = sourceDef.getCustom();
return reportConnectionStatus(synchronousSchedulerClient.createSourceCheckConnectionJob(source, imageName, protocolVersion, isCustomConnector));
}
Expand All @@ -202,7 +201,7 @@ public CheckConnectionRead checkDestinationConnectionFromDestinationId(final Des
throws ConfigNotFoundException, IOException, JsonValidationException {
final DestinationConnection destination = configRepository.getDestinationConnection(destinationIdRequestBody.getDestinationId());
final StandardDestinationDefinition destinationDef = configRepository.getStandardDestinationDefinition(destination.getDestinationDefinitionId());
final String imageName = DockerUtils.getTaggedImageName(destinationDef.getDockerRepository(), destinationDef.getDockerImageTag());
final String imageName = destinationDef.getDockerRepository() + ":" + destinationDef.getDockerImageTag();
final boolean isCustomConnector = destinationDef.getCustom();
final Version protocolVersion = new Version(destinationDef.getProtocolVersion());
return reportConnectionStatus(
Expand All @@ -224,7 +223,7 @@ public CheckConnectionRead checkDestinationConnectionFromDestinationCreate(final
.withDestinationDefinitionId(destinationConfig.getDestinationDefinitionId())
.withConfiguration(partialConfig)
.withWorkspaceId(destinationConfig.getWorkspaceId());
final String imageName = DockerUtils.getTaggedImageName(destDef.getDockerRepository(), destDef.getDockerImageTag());
final String imageName = destDef.getDockerRepository() + ":" + destDef.getDockerImageTag();
final Version protocolVersion = new Version(destDef.getProtocolVersion());
return reportConnectionStatus(
synchronousSchedulerClient.createDestinationCheckConnectionJob(destination, imageName, protocolVersion, isCustomConnector));
Expand All @@ -250,7 +249,7 @@ public SourceDiscoverSchemaRead discoverSchemaForSourceFromSourceId(final Source
throws ConfigNotFoundException, IOException, JsonValidationException {
final SourceConnection source = configRepository.getSourceConnection(discoverSchemaRequestBody.getSourceId());
final StandardSourceDefinition sourceDef = configRepository.getStandardSourceDefinition(source.getSourceDefinitionId());
final String imageName = DockerUtils.getTaggedImageName(sourceDef.getDockerRepository(), sourceDef.getDockerImageTag());
final String imageName = sourceDef.getDockerRepository() + ":" + sourceDef.getDockerImageTag();
final boolean isCustomConnector = sourceDef.getCustom();

final String configHash = HASH_FUNCTION.hashBytes(Jsons.serialize(source.getConfiguration()).getBytes(
Expand Down Expand Up @@ -298,7 +297,7 @@ public SourceDiscoverSchemaRead discoverSchemaForSourceFromSourceCreate(final So
sourceCreate.getConnectionConfiguration(),
sourceDef.getSpec());

final String imageName = DockerUtils.getTaggedImageName(sourceDef.getDockerRepository(), sourceDef.getDockerImageTag());
final String imageName = sourceDef.getDockerRepository() + ":" + sourceDef.getDockerImageTag();
final boolean isCustomConnector = sourceDef.getCustom();
// todo (cgardens) - narrow the struct passed to the client. we are not setting fields that are
// technically declared as required.
Expand Down
9 changes: 6 additions & 3 deletions ...ons-server/src/main/java/io/airbyte/commons/server/handlers/SourceDefinitionsHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
import io.airbyte.api.model.generated.SourceDefinitionUpdate;
import io.airbyte.api.model.generated.SourceRead;
import io.airbyte.api.model.generated.WorkspaceIdRequestBody;
import io.airbyte.commons.docker.DockerUtils;
import io.airbyte.commons.resources.MoreResources;
import io.airbyte.commons.server.ServerConstants;
import io.airbyte.commons.server.converters.ApiPojoConverters;
Expand All @@ -33,7 +32,11 @@
import io.airbyte.commons.version.AirbyteProtocolVersion;
import io.airbyte.commons.version.AirbyteProtocolVersionRange;
import io.airbyte.commons.version.Version;
import io.airbyte.config.*;
import io.airbyte.config.ActorDefinitionResourceRequirements;
import io.airbyte.config.ActorType;
import io.airbyte.config.Configs;
import io.airbyte.config.EnvConfigs;
import io.airbyte.config.StandardSourceDefinition;
import io.airbyte.config.persistence.ConfigNotFoundException;
import io.airbyte.config.persistence.ConfigRepository;
import io.airbyte.protocol.models.ConnectorSpecification;
Expand Down Expand Up @@ -296,7 +299,7 @@ public void deleteSourceDefinition(final SourceDefinitionIdRequestBody sourceDef

private ConnectorSpecification getSpecForImage(final String dockerRepository, final String imageTag, final boolean isCustomConnector)
throws IOException {
final String imageName = DockerUtils.getTaggedImageName(dockerRepository, imageTag);
final String imageName = dockerRepository + ":" + imageTag;
final SynchronousResponse<ConnectorSpecification> getSpecResponse = schedulerSynchronousClient.createGetSpecJob(imageName, isCustomConnector);
return SpecFetcher.getSpecFromJob(getSpecResponse);
}
Expand Down
11 changes: 5 additions & 6 deletions ...r/src/test/java/io/airbyte/commons/server/handlers/DestinationDefinitionsHandlerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@
import io.airbyte.api.model.generated.PrivateDestinationDefinitionReadList;
import io.airbyte.api.model.generated.ReleaseStage;
import io.airbyte.api.model.generated.WorkspaceIdRequestBody;
import io.airbyte.commons.docker.DockerUtils;
import io.airbyte.commons.json.Jsons;
import io.airbyte.commons.server.errors.IdNotFoundKnownException;
import io.airbyte.commons.server.errors.UnsupportedProtocolVersionException;
Expand Down Expand Up @@ -382,7 +381,7 @@ void testCreateDestinationDefinitionShouldCheckProtocolVersion() throws URISynta
final String invalidProtocolVersion = "121.5.6";
final StandardDestinationDefinition destination = generateDestinationDefinition();
destination.getSpec().setProtocolVersion(invalidProtocolVersion);
final String imageName = DockerUtils.getTaggedImageName(destination.getDockerRepository(), destination.getDockerImageTag());
final String imageName = destination.getDockerRepository() + ":" + destination.getDockerImageTag();

when(uuidSupplier.get()).thenReturn(destination.getDestinationDefinitionId());
when(schedulerSynchronousClient.createGetSpecJob(imageName, true)).thenReturn(new SynchronousResponse<>(
Expand Down Expand Up @@ -417,7 +416,7 @@ void testCreateDestinationDefinitionShouldCheckProtocolVersion() throws URISynta
@DisplayName("createCustomDestinationDefinition should correctly create a destinationDefinition")
void testCreateCustomDestinationDefinition() throws URISyntaxException, IOException, JsonValidationException {
final StandardDestinationDefinition destination = generateDestinationDefinition();
final String imageName = DockerUtils.getTaggedImageName(destination.getDockerRepository(), destination.getDockerImageTag());
final String imageName = destination.getDockerRepository() + ":" + destination.getDockerImageTag();

when(uuidSupplier.get()).thenReturn(destination.getDestinationDefinitionId());
when(schedulerSynchronousClient.createGetSpecJob(imageName, true)).thenReturn(new SynchronousResponse<>(
Expand Down Expand Up @@ -474,7 +473,7 @@ void testCreateCustomDestinationDefinitionWithInvalidProtocol() throws URISyntax
final String invalidProtocol = "122.1.22";
final StandardDestinationDefinition destination = generateDestinationDefinition();
destination.getSpec().setProtocolVersion(invalidProtocol);
final String imageName = DockerUtils.getTaggedImageName(destination.getDockerRepository(), destination.getDockerImageTag());
final String imageName = destination.getDockerRepository() + ":" + destination.getDockerImageTag();

when(uuidSupplier.get()).thenReturn(destination.getDestinationDefinitionId());
when(schedulerSynchronousClient.createGetSpecJob(imageName, true)).thenReturn(new SynchronousResponse<>(
Expand Down Expand Up @@ -521,7 +520,7 @@ void testUpdateDestination() throws ConfigNotFoundException, IOException, JsonVa
assertNotEquals(newDockerImageTag, currentTag);
assertNotEquals(newProtocolVersion, currentDestination.getProtocolVersion());

final String newImageName = DockerUtils.getTaggedImageName(destinationDefinition.getDockerRepository(), newDockerImageTag);
final String newImageName = destinationDefinition.getDockerRepository() + ":" + newDockerImageTag;
final ConnectorSpecification newSpec = new ConnectorSpecification()
.withConnectionSpecification(Jsons.jsonNode(ImmutableMap.of("foo2", "bar2")))
.withProtocolVersion(newProtocolVersion);
Expand Down Expand Up @@ -557,7 +556,7 @@ void testOutOfProtocolRangeUpdateDestination() throws ConfigNotFoundException, I
assertNotEquals(newDockerImageTag, currentTag);
assertNotEquals(newProtocolVersion, currentDestination.getProtocolVersion());

final String newImageName = DockerUtils.getTaggedImageName(destinationDefinition.getDockerRepository(), newDockerImageTag);
final String newImageName = destinationDefinition.getDockerRepository() + ":" + newDockerImageTag;
final ConnectorSpecification newSpec = new ConnectorSpecification()
.withConnectionSpecification(Jsons.jsonNode(ImmutableMap.of("foo2", "bar2")))
.withProtocolVersion(newProtocolVersion);
Expand Down
5 changes: 2 additions & 3 deletions ...commons-server/src/test/java/io/airbyte/commons/server/handlers/SchedulerHandlerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@
import io.airbyte.api.model.generated.SourceUpdate;
import io.airbyte.api.model.generated.StreamTransform;
import io.airbyte.api.model.generated.StreamTransform.TransformTypeEnum;
import io.airbyte.commons.docker.DockerUtils;
import io.airbyte.commons.enums.Enums;
import io.airbyte.commons.features.EnvVariableFeatureFlags;
import io.airbyte.commons.json.Jsons;
Expand Down Expand Up @@ -103,12 +102,12 @@ class SchedulerHandlerTest {

private static final String SOURCE_DOCKER_REPO = "srcimage";
private static final String SOURCE_DOCKER_TAG = "tag";
private static final String SOURCE_DOCKER_IMAGE = DockerUtils.getTaggedImageName(SOURCE_DOCKER_REPO, SOURCE_DOCKER_TAG);
private static final String SOURCE_DOCKER_IMAGE = SOURCE_DOCKER_REPO + ":" + SOURCE_DOCKER_TAG;
private static final String SOURCE_PROTOCOL_VERSION = "0.4.5";

private static final String DESTINATION_DOCKER_REPO = "dstimage";
private static final String DESTINATION_DOCKER_TAG = "tag";
private static final String DESTINATION_DOCKER_IMAGE = DockerUtils.getTaggedImageName(DESTINATION_DOCKER_REPO, DESTINATION_DOCKER_TAG);
private static final String DESTINATION_DOCKER_IMAGE = DESTINATION_DOCKER_REPO + ":" + DESTINATION_DOCKER_TAG;
private static final String DESTINATION_PROTOCOL_VERSION = "0.7.9";
private static final String NAME = "name";
private static final String DOGS = "dogs";
Expand Down
Loading