You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The FlowETL DataPresentSensor operator checks whether a file contains any data by attempting to select one row from the foreign data wrapper around the file. DataPresentSensor is an Airflow SqlSensor operator, whose success criterion is that the first cell in the returned row is not in (0, '0', '', None). This means that if the first field in the first row of the file is null (or an empty string), the DataPresentSensor operator will fail to detect data, so ingestion will not proceed.
Given that it's not necessarily the case that a null value in the first column (whatever that may be) invalidates the entire contents of the file, we should find a more robust way of handling this check.
The text was updated successfully, but these errors were encountered:
The FlowETL
DataPresentSensor
operator checks whether a file contains any data by attempting to select one row from the foreign data wrapper around the file.DataPresentSensor
is an AirflowSqlSensor
operator, whose success criterion is that the first cell in the returned row is not in(0, '0', '', None)
. This means that if the first field in the first row of the file is null (or an empty string), theDataPresentSensor
operator will fail to detect data, so ingestion will not proceed.Given that it's not necessarily the case that a null value in the first column (whatever that may be) invalidates the entire contents of the file, we should find a more robust way of handling this check.
The text was updated successfully, but these errors were encountered: