Skip to content

Commit

Permalink
Fix XComObjectStoreBackend config var in docs (apache#38142)
Browse files Browse the repository at this point in the history
* Fix XComObjectStoreBackend config var in docs

* Add objectstore to wordlist

* Set compression to None for empty string
  • Loading branch information
pankajastro authored and utkarsharma2 committed Apr 22, 2024
1 parent 02ca0fd commit 3b0b741
Show file tree
Hide file tree
Showing 5 changed files with 17 additions and 15 deletions.
6 changes: 3 additions & 3 deletions airflow/providers/common/io/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,14 @@ config:
common.io:
description: Common IO configuration section
options:
xcom_objectstorage_path:
xcom_objectstore_path:
description: |
Path to a location on object storage where XComs can be stored in url format.
version_added: 1.3.0
type: string
example: "s3://conn_id@bucket/path"
default: ""
xcom_objectstorage_threshold:
xcom_objectstore_threshold:
description: |
Threshold in bytes for storing XComs in object storage. -1 means always store in the
database. 0 means always store in object storage. Any positive number means
Expand All @@ -69,7 +69,7 @@ config:
type: integer
example: "1000000"
default: "-1"
xcom_objectstorage_compression:
xcom_objectstore_compression:
description: |
Compression algorithm to use when storing XComs in object storage. Supported algorithms
are a.o.: snappy, zip, gzip, bz2, and lzma. If not specified, no compression will be used.
Expand Down
1 change: 1 addition & 0 deletions airflow/providers/common/io/xcom/backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ def serialize_value(
suffix = "." + _get_compression_suffix(compression)
else:
suffix = ""
compression = None

threshold = conf.getint(SECTION, "xcom_objectstore_threshold", fallback=-1)

Expand Down
12 changes: 6 additions & 6 deletions docs/apache-airflow-providers-common-io/xcom_backend.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@ Object Storage XCom Backend

The default XCom backend is the :class:`~airflow.models.xcom.BaseXCom` class, which stores XComs in the Airflow database. This is fine for small values, but can be problematic for large values, or for large numbers of XComs.

To enable storing XComs in an object store, you can set the ``xcom_backend`` configuration option to ``airflow.providers.common.io.xcom.backend.XComObjectStoreBackend``. You will also need to set ``xcom_objectstorage_path`` to the desired location. The connection
id is obtained from the user part of the url the you will provide, e.g. ``xcom_objectstorage_path = s3://conn_id@mybucket/key``. Furthermore, ``xcom_objectstorage_threshold`` is required
To enable storing XComs in an object store, you can set the ``xcom_backend`` configuration option to ``airflow.providers.common.io.xcom.backend.XComObjectStoreBackend``. You will also need to set ``xcom_objectstore_path`` to the desired location. The connection
id is obtained from the user part of the url the you will provide, e.g. ``xcom_objectstore_path = s3://conn_id@mybucket/key``. Furthermore, ``xcom_objectstore_threshold`` is required
to be something larger than -1. Any object smaller than the threshold in bytes will be stored in the database and anything larger will be be
put in object storage. This will allow a hybrid setup. If an xcom is stored on object storage a reference will be
saved in the database. Finally, you can set ``xcom_objectstorage_compression`` to fsspec supported compression methods like ``zip`` or ``snappy`` to
saved in the database. Finally, you can set ``xcom_objectstore_compression`` to fsspec supported compression methods like ``zip`` or ``snappy`` to
compress the data before storing it in object storage.

So for example the following configuration will store anything above 1MB in S3 and will compress it using gzip::
Expand All @@ -33,9 +33,9 @@ So for example the following configuration will store anything above 1MB in S3 a
xcom_backend = airflow.providers.common.io.xcom.backend.XComObjectStoreBackend

[common.io]
xcom_objectstorage_path = s3://conn_id@mybucket/key
xcom_objectstorage_threshold = 1048576
xcom_objectstorage_compression = gzip
xcom_objectstore_path = s3://conn_id@mybucket/key
xcom_objectstore_threshold = 1048576
xcom_objectstore_compression = gzip

.. note::

Expand Down
12 changes: 6 additions & 6 deletions docs/apache-airflow/core-concepts/xcoms.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,11 @@ Object Storage XCom Backend

The default XCom backend is the :class:`~airflow.models.xcom.BaseXCom` class, which stores XComs in the Airflow database. This is fine for small values, but can be problematic for large values, or for large numbers of XComs.

To enable storing XComs in an object store, you can set the ``xcom_backend`` configuration option to ``airflow.providers.common.io.xcom.backend.XComObjectStoreBackend``. You will also need to set ``xcom_objectstorage_path`` to the desired location. The connection
id is obtained from the user part of the url the you will provide, e.g. ``xcom_objectstorage_path = s3://conn_id@mybucket/key``. Furthermore, ``xcom_objectstorage_threshold`` is required
To enable storing XComs in an object store, you can set the ``xcom_backend`` configuration option to ``airflow.providers.common.io.xcom.backend.XComObjectStoreBackend``. You will also need to set ``xcom_objectstore_path`` to the desired location. The connection
id is obtained from the user part of the url the you will provide, e.g. ``xcom_objectstorage_path = s3://conn_id@mybucket/key``. Furthermore, ``xcom_objectstore_threshold`` is required
to be something larger than -1. Any object smaller than the threshold in bytes will be stored in the database and anything larger will be be
put in object storage. This will allow a hybrid setup. If an xcom is stored on object storage a reference will be
saved in the database. Finally, you can set ``xcom_objectstorage_compression`` to fsspec supported compression methods like ``zip`` or ``snappy`` to
saved in the database. Finally, you can set ``xcom_objectstore_compression`` to fsspec supported compression methods like ``zip`` or ``snappy`` to
compress the data before storing it in object storage.

So for example the following configuration will store anything above 1MB in S3 and will compress it using gzip::
Expand All @@ -77,9 +77,9 @@ So for example the following configuration will store anything above 1MB in S3 a
xcom_backend = airflow.providers.common.io.xcom.backend.XComObjectStoreBackend

[common.io]
xcom_objectstorage_path = s3://conn_id@mybucket/key
xcom_objectstorage_threshold = 1048576
xcom_objectstorage_compression = gzip
xcom_objectstore_path = s3://conn_id@mybucket/key
xcom_objectstore_threshold = 1048576
xcom_objectstore_compression = gzip


.. note::
Expand Down
1 change: 1 addition & 0 deletions docs/spelling_wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1092,6 +1092,7 @@ oauth
Oauthlib
objectORfile
objectstorage
objectstore
observability
od
odbc
Expand Down

0 comments on commit 3b0b741

Please sign in to comment.