Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-XXX] Add missing class references #4644

Merged
merged 1 commit into from
Feb 5, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion airflow/contrib/operators/databricks_operator.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ def _deep_string_coerce(content, json_path='json'):
function will throw if content contains non-string or non-numeric types.

The reason why we have this function is because the ``self.json`` field must be a
dict with only string values. This is because ``render_template`` will fail
dict with only string values. This is because ``render_template`` will fail
for numerical values.
"""
c = _deep_string_coerce
Expand Down Expand Up @@ -302,6 +302,7 @@ class DatabricksRunNowOperator(BaseOperator):
to call the ``api/2.0/jobs/run-now`` endpoint and pass it directly
to our ``DatabricksRunNowOperator`` through the ``json`` parameter.
For example ::

json = {
"job_id": 42,
"notebook_params": {
Expand Down Expand Up @@ -349,6 +350,7 @@ class DatabricksRunNowOperator(BaseOperator):

:param job_id: the job_id of the existing Databricks job.
This field will be templated.

.. seealso::
https://docs.databricks.com/api/latest/jobs.html#run-now
:type job_id: str
Expand Down Expand Up @@ -396,6 +398,7 @@ class DatabricksRunNowOperator(BaseOperator):
in job setting.
The json representation of this field cannot exceed 10,000 bytes.
This field will be templated.

.. seealso::
https://docs.databricks.com/api/latest/jobs.html#run-now
:type spark_submit_params: array of strings
Expand Down
33 changes: 17 additions & 16 deletions airflow/contrib/operators/s3_to_sftp_operator.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,22 +27,23 @@

class S3ToSFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a SFTP server
:param sftp_conn_id: The sftp connection id. The name or
identifier for establishing a connection to the SFTP server.
:type sftp_conn_id: string
:param sftp_path: The sftp remote path. This is the specified
file path for uploading file to the SFTP server.
:type sftp_path: string
:param s3_conn_id: The s3 connection id. The name or identifier for establishing
a connection to S3
:type s3_conn_id: string
:param s3_bucket: The targeted s3 bucket. This is the S3 bucket
from where the file is downloaded.
:type s3_bucket: string
:param s3_key: The targeted s3 key. This is the specified file path
for downloading the file from S3.
:type s3_key: string
This operator enables the transferring of files from S3 to a SFTP server.

:param sftp_conn_id: The sftp connection id. The name or identifier for
establishing a connection to the SFTP server.
:type sftp_conn_id: string
:param sftp_path: The sftp remote path. This is the specified file path for
uploading file to the SFTP server.
:type sftp_path: string
:param s3_conn_id: The s3 connection id. The name or identifier for
establishing a connection to S3
:type s3_conn_id: string
:param s3_bucket: The targeted s3 bucket. This is the S3 bucket from
where the file is downloaded.
:type s3_bucket: string
:param s3_key: The targeted s3 key. This is the specified file path for
downloading the file from S3.
:type s3_key: string
"""

template_fields = ('s3_key', 'sftp_path')
Expand Down
34 changes: 18 additions & 16 deletions airflow/contrib/operators/sftp_to_s3_operator.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,22 +27,24 @@

class SFTPToS3Operator(BaseOperator):
"""
This operator enables the transferring of files from a SFTP server to Amazon S3
:param sftp_conn_id: The sftp connection id. The name or identifier for
establishing a connection to the SFTP server.
:type sftp_conn_id: string
:param sftp_path: The sftp remote path. This is the specified file
path for downloading the file from the SFTP server.
:type sftp_path: string
:param s3_conn_id: The s3 connection id. The name or identifier for
establishing a connection to S3
:type s3_conn_id: string
:param s3_bucket: The targeted s3 bucket. This is the S3 bucket
to where the file is uploaded.
:type s3_bucket: string
:param s3_key: The targeted s3 key. This is the specified path
for uploading the file to S3.
:type s3_key: string
This operator enables the transferring of files from a SFTP server to
Amazon S3.

:param sftp_conn_id: The sftp connection id. The name or identifier for
establishing a connection to the SFTP server.
:type sftp_conn_id: string
:param sftp_path: The sftp remote path. This is the specified file path
for downloading the file from the SFTP server.
:type sftp_path: string
:param s3_conn_id: The s3 connection id. The name or identifier for
establishing a connection to S3
:type s3_conn_id: string
:param s3_bucket: The targeted s3 bucket. This is the S3 bucket to where
the file is uploaded.
:type s3_bucket: string
:param s3_key: The targeted s3 key. This is the specified path for
uploading the file to S3.
:type s3_key: string
"""

template_fields = ('s3_key', 'sftp_path')
Expand Down
Loading