🐛 Source Shopify: syncing nested streams (DiscountCodes and PriceRules) fails

[cgalliva]

Environment

• Airbyte version: 0.39.41-alpha
• OS Version / Instance: AWS EC2
• Deployment: Docker
• Source Connector and version: source-shopify - 0.1.37
• Destination Connector and version: destination-s3 - 0.3.12
• Step where error happened: Sync job

Current Behavior

When syncing the DiscountCodes and its parent stream PriceRules the sync job will fail if the updated_at field of a returned discount code query is null. I'll upload the full log but this is the relevant stack trace.

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 114, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 215, in _read_incremental
    for record_counter, record_data in enumerate(records, start=1):
  File "/airbyte/integration_code/source_shopify/source.py", line 240, in read_records
    yield from self.filter_records_newer_than_state(stream_state=stream_state, records_slice=records)
  File "/airbyte/integration_code/source_shopify/source.py", line 132, in filter_records_newer_than_state
    if record.get(self.cursor_field, "") >= stream_state.get(self.cursor_field):
TypeError: '>=' not supported between instances of 'NoneType' and 'str'

Expected Behavior

If the record's cursor field isn't defined, default to yielding/including it

Logs

Error starts at line 116 of the attached log
logs-139.txt

[marcosmarxm]

Thanks for reporting this @cgalliva I added this to connector team backlog. Is it possible to use full refresh method for now?

[cgalliva]

I'll try it now but full refresh will attempt to get all of the data from the start of the configured date each time it syncs right?
If it does it might not be feasible since our store in total has ~100,000 price rules and >1,000,000 discount codes.

[marcosmarxm]

Ow, @cgalliva maybe create an individual connection for these 2 streams and sync only one time a day as a workaround.