[EPIC] Source Shopify to Cloud

[misteryeo]

MLP requirements as surfaced through research:

• 🎉 Source Shopify: Implement 'tender_transactions' stream #10448
• 🐛 Source Shopify: Minor specification re-order, fixed links in documentation #10794
• Support incremental deletes
• Build middle layer in Cloud API for spec mutations #10381
• expose oauth as the only available auth option for shopify in cloud #10817 (depends on https://github.com/airbytehq/airbyte-cloud/pull/1250)
• Shopify Connector not Correctly Labeled w/ Release Phase #11069
• deploy shopify to dev-cloud and get approval from Shopify for our oauth application
• revert Hide shopify in cloud #10783

Outstanding Bugs

• Connecting Shopify connector to Postgres: Database Error in model abandoned_checkouts_customer #10379
• SHOPIFY abandoned_checkouts_customer has a variable type error #10451
• 🐛 Source Shopify: KeyError during normalization #9976
• source shopify: v0.1.26 stream rename causing child stream to run before parent #10595

[bazarnov]

@misteryeo

• Implement 'tender_transactions' stream

It's already there, inside of Transactions stream, providing both transaction details and the user bounded to the transaction per each order_id.

• Support incremental deletes

Shopify does not provide the functionality to retrieve the removed records either using UI or using API calls. They suggesting to use the backups and restore from them once needed.

Considering the answer, what are the future steps for this task?

[misteryeo]

Thanks @bazarnov - are you able to work with @ycherniaiev to ensure the research document is updated to reflect the result of your investigation on tender_transactions?

@sherifnada any thoughts on the incremental deletes?

[bazarnov]

Thanks @bazarnov - are you able to work with @ycherniaiev to ensure the research document is updated to reflect the result of your investigation on tender_transactions?

We've already aligned yesterday on the daily meeting. This subject was discussed. The tender_transactions is the subset of 2 different data objects: Orders + Transactions, with the small amount of fields that includes user_id, order_id and payment_details, basically speaking it's a preset used in GraphQL to make an aggregated result, instead of joining 2 different tables afterwards. We already have Orders, Transactions as separate streams, the user can make all the mutations after the sync on their side.

[ycherniaiev]

Thanks @bazarnov - are you able to work with @ycherniaiev to ensure the research document is updated to reflect the result of your investigation on tender_transactions?

@sherifnada any thoughts on the incremental deletes?

@misteryeo @bazarnov Document is updated. Tender_transactions is still a separate stream by Fivetran according to https://fivetran.com/docs/applications/shopify#schemainformation. So made a note.

[sherifnada]

@bazarnov @misteryeo for incremental deletes: I think we can get them via webhooks, but that's not going to happen by EOQ. So I think we should create a ticket with this context but consider it non-blocking for beta

[misteryeo]

Sounds good, let's move forward with this suggestion @bazarnov

[bazarnov]

UPDATE:
Within the latest API version: 2022-01 we are now able to fetch the tender_transactions data directly from the endpoint.
https://shopify.dev/api/admin-rest/2022-01/resources/tendertransaction#[get]/admin/api/2022-01/tender_transactions.json?processed_at_max=2005-08-05%2010:22:51%20-0400

I'm going to implement and release it ASAP, due to Beta requirements.

[bazarnov]

UPDATE:

The #10448 is implemented and merged into master last week.
The other thing: Support incremental deletes is not currently possible to implement, due to the fact that we don't have the possibility to track the removed records within the data.

Is there anything else we should cover for the Beta status at this moment?

[misteryeo]

Updated the description to reflect some outstanding bugs @bazarnov

Have also added comments in the spreadsheet after reviewing UX/docs that need your help updating. Thanks!

[bazarnov]

#10379
#10451

These 2 issues are related and should be fixed by: #10794

Source Shopify: KeyError during normalization #9976

This one is just the matter of updating the connector >= 0.1.31

#10595

Not sure this is the actual issue at all. I've tested the sync, it was successful, I've also commented in this issue to describe the mechanism of child stream work.

[bazarnov]

Updated the description to reflect some outstanding bugs @bazarnov

Have also added comments in the spreadsheet after reviewing UX/docs that need your help updating. Thanks!

Also, in this PR the specification and documentation were updated as well, based on review comments in the spreadsheet, the spreadsheet was updated as well, see the COMMENTS column for status.

[bazarnov]

Update:

All listed Outstanding Bugs are resolved and merged into master.

[erica-airbyte]

@misteryeo Created this issue for showing the correct connector phase label for Shopify

[misteryeo]

Update:
@girarda is resolving some blockers to get Shopify’s OAuth approval. Once that is done, we’ll let @bazarnov know so he can complete this epic.

[sherifnada]

@bazarnov I've unassigned this issue from you pending completion of #11334

[bazarnov]

@bazarnov I've unassigned this issue from you pending completion of #11334

Ok, feel free to assign it back, once it's finished.

[misteryeo]

Issue was linked to Harvestr Discovery: Source Shopify in GA

[bazarnov]

@YowanR all the issues, related to this Epic and all other small bugs that blocked us previously are resolved. The source-shopify is released as alpha to the Cloud and will remain, until moving to the beta is decided. We can now close this Epic as well.

[bazarnov]

The source-shopify has been released to Cloud as alpha. This Epic could be closed now.