Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FlowETL DataPresentSensor trips over null fields #5090

Closed
jc-harrison opened this issue Apr 19, 2022 · 1 comment · Fixed by #5897
Closed

FlowETL DataPresentSensor trips over null fields #5090

jc-harrison opened this issue Apr 19, 2022 · 1 comment · Fixed by #5897
Labels
bug Something isn't working FlowETL

Comments

@jc-harrison
Copy link
Member

The FlowETL DataPresentSensor operator checks whether a file contains any data by attempting to select one row from the foreign data wrapper around the file. DataPresentSensor is an Airflow SqlSensor operator, whose success criterion is that the first cell in the returned row is not in (0, '0', '', None). This means that if the first field in the first row of the file is null (or an empty string), the DataPresentSensor operator will fail to detect data, so ingestion will not proceed.

Given that it's not necessarily the case that a null value in the first column (whatever that may be) invalidates the entire contents of the file, we should find a more robust way of handling this check.

@jc-harrison
Copy link
Member Author

See also #5763

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working FlowETL
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant