Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Update to badger v3, and use badger as default in memory store #56

Merged
merged 6 commits into from
Nov 29, 2021

Conversation

AndrewSisley
Copy link
Contributor

@AndrewSisley AndrewSisley commented Nov 23, 2021

Closes #52
Closes #40

Probable production bugs found and fixed with commits:

  • Close datastore iterator on scan close, affecting only limit I think
  • Close superseded iterators before orphaning, affecting joins (possibly only joins inside of child groups)

Commit Create transaction after database has been populated modifies the fetcher tests as previously the transaction was created too early, and the tests only passed as the transaction was ignored by the original MapDatastore

Users should be able to upgrade the file store using the steps in the badger FAQ: https://dgraph.io/docs/badger/faq/ but I have not tested this.

@AndrewSisley AndrewSisley added bug Something isn't working feature New feature or request area/datastore Related to the datastore / storage engine system labels Nov 23, 2021
@AndrewSisley AndrewSisley self-assigned this Nov 23, 2021
db/fetcher/fetcher.go Outdated Show resolved Hide resolved
BadgerDB will panic if these are not closed before disposing the enclosing transaction
Record inserted is not within the context of the transaction, and thus not available to the query on the transaction
Is largely a copy-paste of the v2 ipfs code found here (with version change, and some non-compiling custom-defaults dropped)
Copy link
Member

@jsimnz jsimnz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me. Great thing about these datastore interfaces is theres not much to change.

I think we'll prob revisit the db.Options stuff, but that can be deferred to whenever.

@AndrewSisley AndrewSisley merged commit 6f54be2 into develop Nov 29, 2021
@AndrewSisley AndrewSisley deleted the sisley/chore/I52-badger-3 branch November 29, 2021 19:34
@orpheuslummis orpheuslummis mentioned this pull request Jan 29, 2022
shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Feb 23, 2024
…sourcenetwork#56)

* Close datastore iterator on scan close

* Close superseded iterators before orphaning

BadgerDB will panic if these are not closed before disposing the enclosing transaction

* Create transaction after database has been populated

Record inserted is not within the context of the transaction, and thus not available to the query on the transaction

* Add badger v3 wrapper

Is largely a copy-paste of the v2 ipfs code found here (with version change, and some non-compiling custom-defaults dropped)

* Use badger v3 for file-based datastore

* Replace MapDatastore with badger InMemory store
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/datastore Related to the datastore / storage engine system bug Something isn't working feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update badgerDb version Replace ds.MapDatastore with proper in memory datastore
2 participants