-
Notifications
You must be signed in to change notification settings - Fork 92
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
c4da329
commit c65e698
Showing
20 changed files
with
221 additions
and
72 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
19 changes: 19 additions & 0 deletions
19
docker/test/integration/features/google_cloud_storage.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
Feature: Sending data to Google Cloud Storage using PutGcsObject | ||
|
||
Background: | ||
Given the content of "/tmp/output" is monitored | ||
|
||
Scenario: A MiNiFi instance can upload data to Google Cloud storage | ||
Given a GetFile processor with the "Input Directory" property set to "/tmp/input" | ||
And a file with the content "hello_gcs" is present in "/tmp/input" | ||
And a Google Cloud storage server is set up | ||
And a PutGcsObject processor | ||
And PutGcsObject processor is set up with a GcpCredentialsControllerService to communicate with the Google Cloud storage server | ||
And a PutFile processor with the "Directory" property set to "/tmp/output" | ||
And the "success" relationship of the GetFile processor is connected to the PutGcsObject | ||
And the "success" relationship of the PutGcsObject processor is connected to the PutFile | ||
|
||
When all instances start up | ||
|
||
Then a flowfile with the content "hello_gcs" is placed in the monitored directory in less than 45 seconds | ||
And object with the content "hello_gcs" is present in the Google Cloud storage |
17 changes: 17 additions & 0 deletions
17
docker/test/integration/minifi/controllers/GcpCredentialsControllerService.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
from ..core.ControllerService import ControllerService | ||
|
||
|
||
class GcpCredentialsControllerService(ControllerService): | ||
def __init__(self, name=None, credentials_location=None, json_path=None, raw_json=None): | ||
super(GcpCredentialsControllerService, self).__init__(name=name) | ||
|
||
self.service_class = 'GcpCredentialsControllerService' | ||
|
||
if credentials_location is not None: | ||
self.properties['Credentials Location'] = credentials_location | ||
|
||
if json_path is not None: | ||
self.properties['Service Account JSON File'] = json_path | ||
|
||
if raw_json is not None: | ||
self.properties['Service Account JSON'] = raw_json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
docker/test/integration/minifi/core/FakeGcsServerContainer.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
import logging | ||
import os | ||
from .Container import Container | ||
|
||
|
||
class FakeGcsServerContainer(Container): | ||
def __init__(self, name, vols, network, image_store, command=None): | ||
super().__init__(name, 'fake-gcs-server', vols, network, image_store, command) | ||
|
||
def get_startup_finished_log_entry(self): | ||
return "server started at http" | ||
|
||
def deploy(self): | ||
if not self.set_deployed(): | ||
return | ||
|
||
logging.info('Creating and running google cloud storage server docker container...') | ||
self.client.containers.run( | ||
"fsouza/fake-gcs-server:latest", | ||
detach=True, | ||
name=self.name, | ||
network=self.network.name, | ||
entrypoint=self.command, | ||
ports={'4443/tcp': 4443}, | ||
volumes=[os.environ['TEST_DIRECTORY'] + "/resources/fake-gcs-server-data:/data"], | ||
command='-scheme http -host fake-gcs-server') | ||
logging.info('Added container \'%s\'', self.name) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
from ..core.Processor import Processor | ||
|
||
|
||
class PutGcsObject(Processor): | ||
def __init__( | ||
self): | ||
super(PutGcsObject, self).__init__( | ||
'PutGcsObject', | ||
properties={ | ||
'Bucket Name': 'test-bucket', | ||
'Endpoint Override URL': 'fake-gcs-server:4443', | ||
'Number of retries': 2 | ||
}, | ||
auto_terminate=["success", "failure"]) |
Oops, something went wrong.