Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add to_bq() to BigQueryRetrievalJob #1637

Closed
mavysavydav opened this issue Jun 10, 2021 · 4 comments · Fixed by #1634
Closed

Add to_bq() to BigQueryRetrievalJob #1637

mavysavydav opened this issue Jun 10, 2021 · 4 comments · Fixed by #1634

Comments

@mavysavydav
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Data too big for dataframes.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Have feature data joining directly put the training dataset in a BQ table. User may be able to choose table name, whether to replace, expiration, etc.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

N/A

Additional context
Add any other context or screenshots about the feature request here.

@codyjlin
Copy link
Contributor

codyjlin commented Jun 10, 2021

I am currently working on this 🤚

Edit: I will let @vtao2 continue working on this in #1634 (see below comments)

@woop
Copy link
Member

woop commented Jun 11, 2021

I believe @vtao2 is also working on this over here https://github.com/feast-dev/feast/pull/1634/files

@woop
Copy link
Member

woop commented Jun 11, 2021

User may be able to choose table name, whether to replace, expiration, etc

What we were thinking is that the user would define some kind of offline store location like

offline_store:
  type: BigQuery
  dataset: my_feast_dataset

and that to_bq() would create a table within that dataset, but without providing the option to set the name, expiration, etc.

Do you have a use case for more granular control?

@mavysavydav
Copy link
Collaborator Author

Got it, thanks for the heads up. I've polled some ppl on which controls they need. For now, please proceed and i'll update u guys once I have the info

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants