You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Subscriptions are not being fetched from subgraph and added to lake. There is similar logic being used to determine consumptions for df_buyer, but the requirements are different.
Instead, simply use predictSubscriptions w/ a top-level timestamp filter to fetch all subscriptions between the expected [st_ut, fin_ut].
When implementing this, we'll have 2 different subgraph queries being handled by GQL Data Factory, using the same date-range. Due to subgraphs tables having records (or not), the Data Factory should elegantly handle empty gql quries, empty parquet files, and it's basic interface of get_dfs() without any issues.
DoD:
Improve gql_data_factory so it can handle empty fetches, empty parquet files, and general mis-matches between fs.
Improve gql_data_factory to return an empty dataframe of the expected schema if nothing is availbale.
Add fn to fetch subscriptions from subgraph
Create subscription object for models
Create subscription schema for lake
Hook subscriptions to lake
Add mock utils for subscriptions
Add tests that verify that subscription objects, subgraph, and components are working as expected
Add tests that verify that GQL Data Factory that handle empty subgraph fetches + empty parquets files
The text was updated successfully, but these errors were encountered:
Motivation
Subscriptions are not being fetched from subgraph and added to lake. There is similar logic being used to determine consumptions for df_buyer, but the requirements are different.
Instead, simply use predictSubscriptions w/ a top-level timestamp filter to fetch all subscriptions between the expected [st_ut, fin_ut].
When implementing this, we'll have 2 different subgraph queries being handled by GQL Data Factory, using the same date-range. Due to subgraphs tables having records (or not), the Data Factory should elegantly handle empty gql quries, empty parquet files, and it's basic interface of get_dfs() without any issues.
DoD:
The text was updated successfully, but these errors were encountered: