clients: add OPENLINEAGE_DISABLED environment variable which overrides config to NoopTransport #780
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mobuchowski
force-pushed
the
feat/disable-integration-env
branch
2 times, most recently
from
5531fde
to
3fd77b9
Compare
collado-mike
reviewed
Jun 1, 2022
mobuchowski
force-pushed
the
feat/disable-integration-env
branch
2 times, most recently
from
75b1f65
to
8174ccf
Compare
|
Added disabling Spark. |
…s config to NoopTransport Signed-off-by: Maciej Obuchowski <[email protected]>
mobuchowski
force-pushed
the
feat/disable-integration-env
branch
from
8174ccf
to
14f8246
Compare
collado-mike
approved these changes
Jun 3, 2022
howardyoo
added a commit
that referenced
this pull request
Jun 6, 2022
* Added new mypy static analysis tests to the python source codes (in client/python and integration/airflow) Signed-off-by: howardyoo <[email protected]> * added mypy in the requirements Signed-off-by: howardyoo <[email protected]> * some more fixes on configurations to make mypy use setup.cfg Signed-off-by: howardyoo <[email protected]> * testing whether mypy could use setup.cfg Signed-off-by: howardyoo <[email protected]> * modified setup.py for correctly getting mypy version Signed-off-by: howardyoo <[email protected]> * added verbose option in mypy Signed-off-by: howardyoo <[email protected]> * fixed mypy to only perform analytics on openlineage directory Signed-off-by: howardyoo <[email protected]> * added --install-types to the mypy command Signed-off-by: howardyoo <[email protected]> * added --ignore-missing-imports to mypy Signed-off-by: howardyoo <[email protected]> * added types-* packages required by the python client Signed-off-by: howardyoo <[email protected]> * removed trailing whitespaces Signed-off-by: howardyoo <[email protected]> * applied necessary type packages for integration common. also added entry to CHANGELOG.md Signed-off-by: howardyoo <[email protected]> * snowflake: reuse connections in airflow extractors (#819) Signed-off-by: Maciej Obuchowski <[email protected]> * clients: add OPENLINEAGE_DISABLED environment variable which overrides config to NoopTransport (#780) Signed-off-by: Maciej Obuchowski <[email protected]> * Made listener thread use a queue rather than spinning up many new thr… (#820) * Made listener thread use a queue rather than spinning up many new threads Signed-off-by: Michael Collado <[email protected]> * Added test for hanging extractor that sleeps for 30 seconds before returning Signed-off-by: Michael Collado <[email protected]> * Update changelog for 0.9.0 Signed-off-by: Michael Robinson <[email protected]> * Prepare for release 0.9.0 Signed-off-by: Michael Robinson <[email protected]> * Prepare next development version 0.10.0 Signed-off-by: Michael Robinson <[email protected]> * move mypy next to flake8 so that static analysis can happen before expensive unit tests Signed-off-by: howardyoo <[email protected]> * code changes related to fixing problems reported by mypy for dagster integration (#811) Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][AIRFLOW] fix problems reported by mypy for airflow integration (#823) * fixes for issue #805 - airflow integration related fixes for problems reported by mypy Signed-off-by: howardyoo <[email protected]> * added additional fixes Signed-off-by: howardyoo <[email protected]> * [CLIENT][PYTHON] Static code analysis fixes for OpenLineage Python Client (#809) * implemented fixes for the problems reported by mypy Signed-off-by: howardyoo <[email protected]> * fixed multiple imports on one line. Signed-off-by: howardyoo <[email protected]> * applied ignores to certain codes Signed-off-by: howardyoo <[email protected]> * fixed the code so that multiple if statements are not necessary. Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][COMMON] fix problems reported by mypy for common integration (#822) * Fixes for the problems reported by mypy on common package. Signed-off-by: howardyoo <[email protected]> * fixed import statement to work with both Python and Rust SQL parser. Signed-off-by: howardyoo <[email protected]> * added type ignore to the code to prevent unnecessary confusion. Signed-off-by: howardyoo <[email protected]> * fixed formatting error Signed-off-by: howardyoo <[email protected]> * fixed the spacing issue on the inline comment Signed-off-by: howardyoo <[email protected]> * Added new mypy static analysis tests to the python source codes (in client/python and integration/airflow) Signed-off-by: howardyoo <[email protected]> * added mypy in the requirements Signed-off-by: howardyoo <[email protected]> * some more fixes on configurations to make mypy use setup.cfg Signed-off-by: howardyoo <[email protected]> * testing whether mypy could use setup.cfg Signed-off-by: howardyoo <[email protected]> * modified setup.py for correctly getting mypy version Signed-off-by: howardyoo <[email protected]> * added verbose option in mypy Signed-off-by: howardyoo <[email protected]> * fixed mypy to only perform analytics on openlineage directory Signed-off-by: howardyoo <[email protected]> * added --install-types to the mypy command Signed-off-by: howardyoo <[email protected]> * added --ignore-missing-imports to mypy Signed-off-by: howardyoo <[email protected]> * added types-* packages required by the python client Signed-off-by: howardyoo <[email protected]> * removed trailing whitespaces Signed-off-by: howardyoo <[email protected]> * applied necessary type packages for integration common. also added entry to CHANGELOG.md Signed-off-by: howardyoo <[email protected]> * move mypy next to flake8 so that static analysis can happen before expensive unit tests Signed-off-by: howardyoo <[email protected]> * code changes related to fixing problems reported by mypy for dagster integration (#811) Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][AIRFLOW] fix problems reported by mypy for airflow integration (#823) * fixes for issue #805 - airflow integration related fixes for problems reported by mypy Signed-off-by: howardyoo <[email protected]> * added additional fixes Signed-off-by: howardyoo <[email protected]> * [CLIENT][PYTHON] Static code analysis fixes for OpenLineage Python Client (#809) * implemented fixes for the problems reported by mypy Signed-off-by: howardyoo <[email protected]> * fixed multiple imports on one line. Signed-off-by: howardyoo <[email protected]> * applied ignores to certain codes Signed-off-by: howardyoo <[email protected]> * fixed the code so that multiple if statements are not necessary. Signed-off-by: howardyoo <[email protected]> * fixed formatting error Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][COMMON] fix problems reported by mypy for common integration (#822) * Fixes for the problems reported by mypy on common package. Signed-off-by: howardyoo <[email protected]> * fixed import statement to work with both Python and Rust SQL parser. Signed-off-by: howardyoo <[email protected]> * added type ignore to the code to prevent unnecessary confusion. Signed-off-by: howardyoo <[email protected]> * fixed the spacing issue on the inline comment Signed-off-by: howardyoo <[email protected]> * merged Signed-off-by: howardyoo <[email protected]> * fixed formatting errors Signed-off-by: howardyoo <[email protected]> * fixed incorrect indentation of the next line Signed-off-by: howardyoo <[email protected]> * additional mypy issues fixed Signed-off-by: howardyoo <[email protected]> Co-authored-by: Maciej Obuchowski <[email protected]> Co-authored-by: Michael Collado <[email protected]> Co-authored-by: Michael Robinson <[email protected]>
Tylerpfarris
pushed a commit
to Tylerpfarris/OpenLineage
that referenced
this pull request
Jun 23, 2022
…s config to NoopTransport (OpenLineage#780) Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: Tyler Farris <[email protected]>
Tylerpfarris
pushed a commit
to Tylerpfarris/OpenLineage
that referenced
this pull request
Jun 23, 2022
* Added new mypy static analysis tests to the python source codes (in client/python and integration/airflow) Signed-off-by: howardyoo <[email protected]> * added mypy in the requirements Signed-off-by: howardyoo <[email protected]> * some more fixes on configurations to make mypy use setup.cfg Signed-off-by: howardyoo <[email protected]> * testing whether mypy could use setup.cfg Signed-off-by: howardyoo <[email protected]> * modified setup.py for correctly getting mypy version Signed-off-by: howardyoo <[email protected]> * added verbose option in mypy Signed-off-by: howardyoo <[email protected]> * fixed mypy to only perform analytics on openlineage directory Signed-off-by: howardyoo <[email protected]> * added --install-types to the mypy command Signed-off-by: howardyoo <[email protected]> * added --ignore-missing-imports to mypy Signed-off-by: howardyoo <[email protected]> * added types-* packages required by the python client Signed-off-by: howardyoo <[email protected]> * removed trailing whitespaces Signed-off-by: howardyoo <[email protected]> * applied necessary type packages for integration common. also added entry to CHANGELOG.md Signed-off-by: howardyoo <[email protected]> * snowflake: reuse connections in airflow extractors (OpenLineage#819) Signed-off-by: Maciej Obuchowski <[email protected]> * clients: add OPENLINEAGE_DISABLED environment variable which overrides config to NoopTransport (OpenLineage#780) Signed-off-by: Maciej Obuchowski <[email protected]> * Made listener thread use a queue rather than spinning up many new thr… (OpenLineage#820) * Made listener thread use a queue rather than spinning up many new threads Signed-off-by: Michael Collado <[email protected]> * Added test for hanging extractor that sleeps for 30 seconds before returning Signed-off-by: Michael Collado <[email protected]> * Update changelog for 0.9.0 Signed-off-by: Michael Robinson <[email protected]> * Prepare for release 0.9.0 Signed-off-by: Michael Robinson <[email protected]> * Prepare next development version 0.10.0 Signed-off-by: Michael Robinson <[email protected]> * move mypy next to flake8 so that static analysis can happen before expensive unit tests Signed-off-by: howardyoo <[email protected]> * code changes related to fixing problems reported by mypy for dagster integration (OpenLineage#811) Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][AIRFLOW] fix problems reported by mypy for airflow integration (OpenLineage#823) * fixes for issue OpenLineage#805 - airflow integration related fixes for problems reported by mypy Signed-off-by: howardyoo <[email protected]> * added additional fixes Signed-off-by: howardyoo <[email protected]> * [CLIENT][PYTHON] Static code analysis fixes for OpenLineage Python Client (OpenLineage#809) * implemented fixes for the problems reported by mypy Signed-off-by: howardyoo <[email protected]> * fixed multiple imports on one line. Signed-off-by: howardyoo <[email protected]> * applied ignores to certain codes Signed-off-by: howardyoo <[email protected]> * fixed the code so that multiple if statements are not necessary. Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][COMMON] fix problems reported by mypy for common integration (OpenLineage#822) * Fixes for the problems reported by mypy on common package. Signed-off-by: howardyoo <[email protected]> * fixed import statement to work with both Python and Rust SQL parser. Signed-off-by: howardyoo <[email protected]> * added type ignore to the code to prevent unnecessary confusion. Signed-off-by: howardyoo <[email protected]> * fixed formatting error Signed-off-by: howardyoo <[email protected]> * fixed the spacing issue on the inline comment Signed-off-by: howardyoo <[email protected]> * Added new mypy static analysis tests to the python source codes (in client/python and integration/airflow) Signed-off-by: howardyoo <[email protected]> * added mypy in the requirements Signed-off-by: howardyoo <[email protected]> * some more fixes on configurations to make mypy use setup.cfg Signed-off-by: howardyoo <[email protected]> * testing whether mypy could use setup.cfg Signed-off-by: howardyoo <[email protected]> * modified setup.py for correctly getting mypy version Signed-off-by: howardyoo <[email protected]> * added verbose option in mypy Signed-off-by: howardyoo <[email protected]> * fixed mypy to only perform analytics on openlineage directory Signed-off-by: howardyoo <[email protected]> * added --install-types to the mypy command Signed-off-by: howardyoo <[email protected]> * added --ignore-missing-imports to mypy Signed-off-by: howardyoo <[email protected]> * added types-* packages required by the python client Signed-off-by: howardyoo <[email protected]> * removed trailing whitespaces Signed-off-by: howardyoo <[email protected]> * applied necessary type packages for integration common. also added entry to CHANGELOG.md Signed-off-by: howardyoo <[email protected]> * move mypy next to flake8 so that static analysis can happen before expensive unit tests Signed-off-by: howardyoo <[email protected]> * code changes related to fixing problems reported by mypy for dagster integration (OpenLineage#811) Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][AIRFLOW] fix problems reported by mypy for airflow integration (OpenLineage#823) * fixes for issue OpenLineage#805 - airflow integration related fixes for problems reported by mypy Signed-off-by: howardyoo <[email protected]> * added additional fixes Signed-off-by: howardyoo <[email protected]> * [CLIENT][PYTHON] Static code analysis fixes for OpenLineage Python Client (OpenLineage#809) * implemented fixes for the problems reported by mypy Signed-off-by: howardyoo <[email protected]> * fixed multiple imports on one line. Signed-off-by: howardyoo <[email protected]> * applied ignores to certain codes Signed-off-by: howardyoo <[email protected]> * fixed the code so that multiple if statements are not necessary. Signed-off-by: howardyoo <[email protected]> * fixed formatting error Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][COMMON] fix problems reported by mypy for common integration (OpenLineage#822) * Fixes for the problems reported by mypy on common package. Signed-off-by: howardyoo <[email protected]> * fixed import statement to work with both Python and Rust SQL parser. Signed-off-by: howardyoo <[email protected]> * added type ignore to the code to prevent unnecessary confusion. Signed-off-by: howardyoo <[email protected]> * fixed the spacing issue on the inline comment Signed-off-by: howardyoo <[email protected]> * merged Signed-off-by: howardyoo <[email protected]> * fixed formatting errors Signed-off-by: howardyoo <[email protected]> * fixed incorrect indentation of the next line Signed-off-by: howardyoo <[email protected]> * additional mypy issues fixed Signed-off-by: howardyoo <[email protected]> Co-authored-by: Maciej Obuchowski <[email protected]> Co-authored-by: Michael Collado <[email protected]> Co-authored-by: Michael Robinson <[email protected]> Signed-off-by: Tyler Farris <[email protected]>
Tylerpfarris
pushed a commit
to Tylerpfarris/OpenLineage
that referenced
this pull request
Jun 24, 2022
…s config to NoopTransport (OpenLineage#780) Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: Tyler Farris <[email protected]>
Tylerpfarris
pushed a commit
to Tylerpfarris/OpenLineage
that referenced
this pull request
Jun 24, 2022
* Added new mypy static analysis tests to the python source codes (in client/python and integration/airflow) Signed-off-by: howardyoo <[email protected]> * added mypy in the requirements Signed-off-by: howardyoo <[email protected]> * some more fixes on configurations to make mypy use setup.cfg Signed-off-by: howardyoo <[email protected]> * testing whether mypy could use setup.cfg Signed-off-by: howardyoo <[email protected]> * modified setup.py for correctly getting mypy version Signed-off-by: howardyoo <[email protected]> * added verbose option in mypy Signed-off-by: howardyoo <[email protected]> * fixed mypy to only perform analytics on openlineage directory Signed-off-by: howardyoo <[email protected]> * added --install-types to the mypy command Signed-off-by: howardyoo <[email protected]> * added --ignore-missing-imports to mypy Signed-off-by: howardyoo <[email protected]> * added types-* packages required by the python client Signed-off-by: howardyoo <[email protected]> * removed trailing whitespaces Signed-off-by: howardyoo <[email protected]> * applied necessary type packages for integration common. also added entry to CHANGELOG.md Signed-off-by: howardyoo <[email protected]> * snowflake: reuse connections in airflow extractors (OpenLineage#819) Signed-off-by: Maciej Obuchowski <[email protected]> * clients: add OPENLINEAGE_DISABLED environment variable which overrides config to NoopTransport (OpenLineage#780) Signed-off-by: Maciej Obuchowski <[email protected]> * Made listener thread use a queue rather than spinning up many new thr… (OpenLineage#820) * Made listener thread use a queue rather than spinning up many new threads Signed-off-by: Michael Collado <[email protected]> * Added test for hanging extractor that sleeps for 30 seconds before returning Signed-off-by: Michael Collado <[email protected]> * Update changelog for 0.9.0 Signed-off-by: Michael Robinson <[email protected]> * Prepare for release 0.9.0 Signed-off-by: Michael Robinson <[email protected]> * Prepare next development version 0.10.0 Signed-off-by: Michael Robinson <[email protected]> * move mypy next to flake8 so that static analysis can happen before expensive unit tests Signed-off-by: howardyoo <[email protected]> * code changes related to fixing problems reported by mypy for dagster integration (OpenLineage#811) Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][AIRFLOW] fix problems reported by mypy for airflow integration (OpenLineage#823) * fixes for issue OpenLineage#805 - airflow integration related fixes for problems reported by mypy Signed-off-by: howardyoo <[email protected]> * added additional fixes Signed-off-by: howardyoo <[email protected]> * [CLIENT][PYTHON] Static code analysis fixes for OpenLineage Python Client (OpenLineage#809) * implemented fixes for the problems reported by mypy Signed-off-by: howardyoo <[email protected]> * fixed multiple imports on one line. Signed-off-by: howardyoo <[email protected]> * applied ignores to certain codes Signed-off-by: howardyoo <[email protected]> * fixed the code so that multiple if statements are not necessary. Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][COMMON] fix problems reported by mypy for common integration (OpenLineage#822) * Fixes for the problems reported by mypy on common package. Signed-off-by: howardyoo <[email protected]> * fixed import statement to work with both Python and Rust SQL parser. Signed-off-by: howardyoo <[email protected]> * added type ignore to the code to prevent unnecessary confusion. Signed-off-by: howardyoo <[email protected]> * fixed formatting error Signed-off-by: howardyoo <[email protected]> * fixed the spacing issue on the inline comment Signed-off-by: howardyoo <[email protected]> * Added new mypy static analysis tests to the python source codes (in client/python and integration/airflow) Signed-off-by: howardyoo <[email protected]> * added mypy in the requirements Signed-off-by: howardyoo <[email protected]> * some more fixes on configurations to make mypy use setup.cfg Signed-off-by: howardyoo <[email protected]> * testing whether mypy could use setup.cfg Signed-off-by: howardyoo <[email protected]> * modified setup.py for correctly getting mypy version Signed-off-by: howardyoo <[email protected]> * added verbose option in mypy Signed-off-by: howardyoo <[email protected]> * fixed mypy to only perform analytics on openlineage directory Signed-off-by: howardyoo <[email protected]> * added --install-types to the mypy command Signed-off-by: howardyoo <[email protected]> * added --ignore-missing-imports to mypy Signed-off-by: howardyoo <[email protected]> * added types-* packages required by the python client Signed-off-by: howardyoo <[email protected]> * removed trailing whitespaces Signed-off-by: howardyoo <[email protected]> * applied necessary type packages for integration common. also added entry to CHANGELOG.md Signed-off-by: howardyoo <[email protected]> * move mypy next to flake8 so that static analysis can happen before expensive unit tests Signed-off-by: howardyoo <[email protected]> * code changes related to fixing problems reported by mypy for dagster integration (OpenLineage#811) Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][AIRFLOW] fix problems reported by mypy for airflow integration (OpenLineage#823) * fixes for issue OpenLineage#805 - airflow integration related fixes for problems reported by mypy Signed-off-by: howardyoo <[email protected]> * added additional fixes Signed-off-by: howardyoo <[email protected]> * [CLIENT][PYTHON] Static code analysis fixes for OpenLineage Python Client (OpenLineage#809) * implemented fixes for the problems reported by mypy Signed-off-by: howardyoo <[email protected]> * fixed multiple imports on one line. Signed-off-by: howardyoo <[email protected]> * applied ignores to certain codes Signed-off-by: howardyoo <[email protected]> * fixed the code so that multiple if statements are not necessary. Signed-off-by: howardyoo <[email protected]> * fixed formatting error Signed-off-by: howardyoo <[email protected]> * [INTEGRATION][COMMON] fix problems reported by mypy for common integration (OpenLineage#822) * Fixes for the problems reported by mypy on common package. Signed-off-by: howardyoo <[email protected]> * fixed import statement to work with both Python and Rust SQL parser. Signed-off-by: howardyoo <[email protected]> * added type ignore to the code to prevent unnecessary confusion. Signed-off-by: howardyoo <[email protected]> * fixed the spacing issue on the inline comment Signed-off-by: howardyoo <[email protected]> * merged Signed-off-by: howardyoo <[email protected]> * fixed formatting errors Signed-off-by: howardyoo <[email protected]> * fixed incorrect indentation of the next line Signed-off-by: howardyoo <[email protected]> * additional mypy issues fixed Signed-off-by: howardyoo <[email protected]> Co-authored-by: Maciej Obuchowski <[email protected]> Co-authored-by: Michael Collado <[email protected]> Co-authored-by: Michael Robinson <[email protected]> Signed-off-by: Tyler Farris <[email protected]>
Tylerpfarris
pushed a commit
to Tylerpfarris/OpenLineage
that referenced
this pull request
Jun 24, 2022
…s config to NoopTransport (OpenLineage#780) * rebase Signed-off-by: Tyler Farris <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, configured OpenLineage integrations will always try to emit events.
This PR adds environment variable OPENLINEAGE_DISABLE that when set to
trueoverrides this - and will make integration use (added) NoopTransport when enabled.This does not override clients that were manually provided with other transports - just overrides the default transport factory construction.
Closes #726