Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opt(schema): Load schema and types using Stream framework #8562

Merged
merged 1 commit into from
Jan 17, 2023

Conversation

mangalaman93
Copy link
Contributor

For big datasets, we're seeing a big slowdown due to loading schema and types serially using a single iterator. Using the Stream framework, makes this metadata loading step much faster, resulting in a much faster Alpha initialization.

(cherry picked from commit d03d5ad)

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions github-actions bot added the area/schema Issues related to the schema language and capabilities. label Jan 4, 2023
@mangalaman93 mangalaman93 added the slash-to-main PRs which bring slash branch on par with main. label Jan 4, 2023
@coveralls
Copy link

coveralls commented Jan 4, 2023

Coverage Status

Coverage: 66.611% (+0.2%) from 66.44% when pulling 6d9c83e on aman/schema into 2aed52e on main.

schema/schema.go Outdated Show resolved Hide resolved
Copy link
Contributor

@harshil-goel harshil-goel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left a question.
I feel like the stream framework api is a bit weird. Like we should be just passing the KeyToList function in the object creation itself? What do you think, maybe we should take a look at all the apis that badger has exposed.

schema/schema.go Show resolved Hide resolved
)

For big datasets, we're seeing a big slowdown due to loading schema and
types serially using a single iterator. Using the Stream framework, makes
this metadata loading step much faster, resulting in a much faster Alpha
initialization.

(cherry picked from commit d03d5ad)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/schema Issues related to the schema language and capabilities. slash-to-main PRs which bring slash branch on par with main.
Development

Successfully merging this pull request may close these issues.

7 participants