Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ChainExportRange API method #8641

Open
5 of 15 tasks
travisperson opened this issue May 11, 2022 · 3 comments
Open
5 of 15 tasks

Add ChainExportRange API method #8641

travisperson opened this issue May 11, 2022 · 3 comments
Assignees
Labels
kind/enhancement Kind: Enhancement

Comments

@travisperson
Copy link
Contributor

travisperson commented May 11, 2022

Checklist

  • This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.
  • This is not a new feature request. If it is, please file a feature request instead.
  • This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
  • I have a specific, actionable, and well motivated improvement to propose.

Lotus component

  • lotus daemon - chain sync
  • lotus miner - mining and block production
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt)
  • lotus miner/market - storage deal
  • lotus miner/market - retrieval deal
  • lotus miner/market - data transfer
  • lotus client
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other

Improvement Suggestion

The ChainExport command doesn't provide an accurate way to export a segment of the lotus chain.

For archival and testing purposes it would be beneficial to be able to export an exact range of tipsets by adding a new method ChainExportRange that accepted two TipSetKey values as arguments and does not enforce a minimum epoch height between them (eg a call with both arguments the same should export a single TipSet).

The ChainExportRange should also accept a set of options

  • Include Messages
  • Include Message Receipts

It will be important to think through the exact inclusiveness of the method to ensure that no data is missed under the following usage:

ChainExportRange(ChainGetTipSetByHeight(a), ChainGetTipSetByHeight(b))
ChainExportRange(ChainGetTipSetByHeight(b), ChainGetTipSetByHeight(c))

I think ChainGetTipSetByHeight is the correct method to use here, I think as long as it's consistent it should be fine

The above should include the same data as

ChainExportRange(ChainGetTipSetByHeight(a), ChainGetTipSetByHeight(c))

Additionally a command should be added to lotus-shed to support the usage of this API method, mostly because chain export enforces a minimum epochs to export.

The export should construct CARv2 files with the index.

@travisperson
Copy link
Contributor Author

If we want to go crazy we could also add options to make including state and blockheaders optional as well, which would allow this method to be used to extract just the messages. However, ChainGetMessagesInTipset already exists and can be used to grab this data and pack a car file manually.

@travisperson
Copy link
Contributor Author

travisperson commented May 17, 2022

With support of message receipts this will close out #3849

@travisperson
Copy link
Contributor Author

This export should allow for exporting of CARv2 files with the index.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement Kind: Enhancement
Projects
None yet
Development

No branches or pull requests

3 participants
@travisperson @rjan90 and others