Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

curio: feat: break trees task into TreeD(prefetch) and TreeRC #11895

Merged
merged 11 commits into from
May 2, 2024

Conversation

LexLuthr
Copy link
Contributor

@LexLuthr LexLuthr commented Apr 18, 2024

Related Issues

Proposed Changes

  • TreesTask is now broken into 2 Tasks namely TreeD and TreeRC
  • The storage reservation with some free space is performed in TreeD
  • The storage in TypeDetails now take an additional variable (int) MinFreeStoragePercentage.
  • The sealing pipeline is functioning as normal for CC and Deal sectors
Screenshot 2024-04-18 at 11 13 31 PM

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@LexLuthr LexLuthr requested a review from magik6k April 18, 2024 12:25
@LexLuthr
Copy link
Contributor Author

@magik6k I was tested the new Tasks and they are working fine. Can you please review the logic once?

@LexLuthr LexLuthr mentioned this pull request Apr 18, 2024
19 tasks
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some issues to address, but after that seems like this approach should work great!

curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved
storage/paths/local.go Outdated Show resolved Hide resolved
storage/paths/local.go Show resolved Hide resolved
curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved
curiosrc/ffi/sdr_funcs.go Outdated Show resolved Hide resolved
curiosrc/ffi/sdr_funcs.go Outdated Show resolved Hide resolved
curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved
curiosrc/ffi/sdr_funcs.go Outdated Show resolved Hide resolved
curiosrc/seal/task_treerc.go Outdated Show resolved Hide resolved
curiosrc/seal/task_treerc.go Outdated Show resolved Hide resolved
@LexLuthr LexLuthr marked this pull request as draft April 21, 2024 15:30
@LexLuthr
Copy link
Contributor Author

Sealing pipeline test after last commit

Screenshot 2024-04-21 at 7 41 04 PM

@LexLuthr LexLuthr marked this pull request as ready for review April 21, 2024 15:43
@LexLuthr LexLuthr requested a review from snadrus April 22, 2024 16:35
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks almost landable now, just a few comments to address (really mostly the one about release calling)

I'm going to give this a try on my cluster.

curiosrc/seal/task_treed.go Show resolved Hide resolved
curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved
curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved
curiosrc/seal/task_treed.go Outdated Show resolved Hide resolved
@LexLuthr
Copy link
Contributor Author

@snadrus This is ready for your review.

curiosrc/ffi/sdr_funcs.go Outdated Show resolved Hide resolved
curiosrc/ffi/sdr_funcs.go Outdated Show resolved Hide resolved
curiosrc/seal/task_treerc.go Outdated Show resolved Hide resolved
@LexLuthr LexLuthr requested a review from snadrus April 30, 2024 06:50
Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Also appears to work fine on my cluster.

@magik6k magik6k enabled auto-merge (squash) May 2, 2024 09:11
@magik6k magik6k disabled auto-merge May 2, 2024 09:14
@magik6k magik6k merged commit 00edad4 into master May 2, 2024
186 checks passed
@magik6k magik6k deleted the feat/break-treestask branch May 2, 2024 09:17
snadrus added a commit that referenced this pull request May 2, 2024
* Fixing dead links (#11907)

* ci: ci: create gh workflow that updates sorted pr checks (#11861)

* ci: create gh workflow that updates sorted pr checks

* ci: use grouped_by_result template for pr checks sticky comment

* chore: apply pr review suggestion

* Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916)

Co-authored-by: zenground0 <[email protected]>

* Fix mismatched method names in comments (#11913)

Signed-off-by: forcedebug <[email protected]>

* release: v1.26.3 (#11908) (#11915) (#11922)

* deps: update dependencies to address migration memory bloat

to address memory concerns during a heavy migration

Ref: filecoin-project/go-state-types#260
Ref: whyrusleeping/cbor-gen#96
Ref: filecoin-project/go-amt-ipld#90

* release: prep v1.26.3 patch

Prep v1.26.3 patch release:
- Update changelog, version and make gen + make docsgen-cli

* deps: update cbor-gen to tagged version

deps: update cbor-gen to tagged version

* deps: update go-state-types to tagged version

deps: update go-state-types to tagged version v0.13.2

* chore: deps: update go-state-types to v0.13.3

Fixes a panic when we have fewer than 1k proposals.

---------

Co-authored-by: Phi-rjan <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Steven Allen <[email protected]>

* Refactor `LookupID*` APIs in `StateManager` and `StateTree`

The naming of `LookupID` can cause confusion when resolving actor IDs vs
 ID addresses. To avoid this:

* Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it
returns ID address.
* Refactor `StateManager` `LookupID` to
`LookupIDAddress` because it also returns ID address via a chain call to
`StateTree`.
* Introduce a new API `StateManager` dedicated to resolving address to
actor ID, called `LookupID` which returns `abi.ActorID`.

For context, see:
 * #11723 (comment)

* Add v13 support to invariants-checker (#11931)

Add v13 support to invariants-checker

* chore: docs: nv-skeleton documentation (#11065)

* nv-skeleton documentation

Add a tutorial for how one can create a nv-skeleton in Lotus

* Add footnote for `Add migration` step

Add footnote for `Add migration` step

* Indent migration-code

Indent migration-code to make it show properly as a footnote.

* Add ref-fvm and filecoin-ffi checklist

Add ref-fvm and filecoin-ffi checklist

* Add Filecoin-FFI steps

Add Filecoin-FFI steps

* Add step to params_butterfly.go

Add step to params_butterfly.go

* Fix typo

Fix typo

* Add links to reference PRs

Add links to reference PRs

* Update ref-fvm list

Update ref-fvm list

* feat: curio: add StorageInit api (#11918)

* feat: add StorageInit api

* remove unused variables

* fix gen check

* feat: curio: simpler reservation release logic (#11900)

* simpler release logic

* oops, plus simpler

* simpler

* fix NewLine (#11893)

* fix(events): check for sync-in-progress (#11932)

* feat(events): adjust indexes in event index db to match query patterns

Introduces a v4 migration that just adjusts indexes.

Copies some improvements from #11723

Closes: #11909

* fix(pipeline): should return if error occurred when get network version (#11902)

* fix(events): correct log msg for v4 events index db migration

* chore: remove duplicate words in strings and comments

* fix(events): register events index db migration v4

* fix: curio seal: Failed commit retry strategy (#11870)

* ffi: improved-error-handling

* curio seal: Failed commit retry strategy

* use master ffi

* mod tidy

* fix: curio: Update pgx imports, fix db_storage alloc

* feat: curioweb: Improve task_history indexes (#11911)

* mod tidy

* Event index should be unique for tipsets (#11952)

* event index should be unique for tipsets

* fix formatting

* migrate to version 5

* chore: bump build version in master (#11946)

* Bump version

Bump version in master branch in preperation for cutting v1.27.0-rc1

* chore: bump build-version

chore: bump build-version

* feat: curioweb: Show piece info on the sector page (#11955)

* curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895)

* break trees task

* fix TreeD reservation

* fix nil pointer err

* apply suggestions

* fix allocate file types

* fix dbIndex inserts

* set resource, move release func

* refactor func(), update memory

* remove extra release

---------

Signed-off-by: forcedebug <[email protected]>
Co-authored-by: parthshah1 <[email protected]>
Co-authored-by: Piotr Galar <[email protected]>
Co-authored-by: ZenGround0 <[email protected]>
Co-authored-by: zenground0 <[email protected]>
Co-authored-by: forcedebug <[email protected]>
Co-authored-by: Jiaying Wang <[email protected]>
Co-authored-by: Phi-rjan <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Steven Allen <[email protected]>
Co-authored-by: Masih H. Derkani <[email protected]>
Co-authored-by: Lee <[email protected]>
Co-authored-by: Andrew Jackson (Ajax) <[email protected]>
Co-authored-by: beck <[email protected]>
Co-authored-by: 0x5459 <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Aarsh Shah <[email protected]>
magik6k added a commit that referenced this pull request May 4, 2024
* fix: curio: Update pgx imports, fix db_storage alloc

* feat: curioweb: Improve task_history indexes (#11911)

* 1

* relatable

* add and delete layer

* chore: curio: merge master (#11956)

* Fixing dead links (#11907)

* ci: ci: create gh workflow that updates sorted pr checks (#11861)

* ci: create gh workflow that updates sorted pr checks

* ci: use grouped_by_result template for pr checks sticky comment

* chore: apply pr review suggestion

* Avoid cfg lookup on chain remove since unenabled splitstore delete is noop anyway (#11916)

Co-authored-by: zenground0 <[email protected]>

* Fix mismatched method names in comments (#11913)

Signed-off-by: forcedebug <[email protected]>

* release: v1.26.3 (#11908) (#11915) (#11922)

* deps: update dependencies to address migration memory bloat

to address memory concerns during a heavy migration

Ref: filecoin-project/go-state-types#260
Ref: whyrusleeping/cbor-gen#96
Ref: filecoin-project/go-amt-ipld#90

* release: prep v1.26.3 patch

Prep v1.26.3 patch release:
- Update changelog, version and make gen + make docsgen-cli

* deps: update cbor-gen to tagged version

deps: update cbor-gen to tagged version

* deps: update go-state-types to tagged version

deps: update go-state-types to tagged version v0.13.2

* chore: deps: update go-state-types to v0.13.3

Fixes a panic when we have fewer than 1k proposals.

---------

Co-authored-by: Phi-rjan <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Steven Allen <[email protected]>

* Refactor `LookupID*` APIs in `StateManager` and `StateTree`

The naming of `LookupID` can cause confusion when resolving actor IDs vs
 ID addresses. To avoid this:

* Refactor `StateTree` `LookupID` to `LookupIDAddress`, because it
returns ID address.
* Refactor `StateManager` `LookupID` to
`LookupIDAddress` because it also returns ID address via a chain call to
`StateTree`.
* Introduce a new API `StateManager` dedicated to resolving address to
actor ID, called `LookupID` which returns `abi.ActorID`.

For context, see:
 * #11723 (comment)

* Add v13 support to invariants-checker (#11931)

Add v13 support to invariants-checker

* chore: docs: nv-skeleton documentation (#11065)

* nv-skeleton documentation

Add a tutorial for how one can create a nv-skeleton in Lotus

* Add footnote for `Add migration` step

Add footnote for `Add migration` step

* Indent migration-code

Indent migration-code to make it show properly as a footnote.

* Add ref-fvm and filecoin-ffi checklist

Add ref-fvm and filecoin-ffi checklist

* Add Filecoin-FFI steps

Add Filecoin-FFI steps

* Add step to params_butterfly.go

Add step to params_butterfly.go

* Fix typo

Fix typo

* Add links to reference PRs

Add links to reference PRs

* Update ref-fvm list

Update ref-fvm list

* feat: curio: add StorageInit api (#11918)

* feat: add StorageInit api

* remove unused variables

* fix gen check

* feat: curio: simpler reservation release logic (#11900)

* simpler release logic

* oops, plus simpler

* simpler

* fix NewLine (#11893)

* fix(events): check for sync-in-progress (#11932)

* feat(events): adjust indexes in event index db to match query patterns

Introduces a v4 migration that just adjusts indexes.

Copies some improvements from #11723

Closes: #11909

* fix(pipeline): should return if error occurred when get network version (#11902)

* fix(events): correct log msg for v4 events index db migration

* chore: remove duplicate words in strings and comments

* fix(events): register events index db migration v4

* fix: curio seal: Failed commit retry strategy (#11870)

* ffi: improved-error-handling

* curio seal: Failed commit retry strategy

* use master ffi

* mod tidy

* fix: curio: Update pgx imports, fix db_storage alloc

* feat: curioweb: Improve task_history indexes (#11911)

* mod tidy

* Event index should be unique for tipsets (#11952)

* event index should be unique for tipsets

* fix formatting

* migrate to version 5

* chore: bump build version in master (#11946)

* Bump version

Bump version in master branch in preperation for cutting v1.27.0-rc1

* chore: bump build-version

chore: bump build-version

* feat: curioweb: Show piece info on the sector page (#11955)

* curio: feat: break trees task into TreeD(prefetch) and TreeRC (#11895)

* break trees task

* fix TreeD reservation

* fix nil pointer err

* apply suggestions

* fix allocate file types

* fix dbIndex inserts

* set resource, move release func

* refactor func(), update memory

* remove extra release

---------

Signed-off-by: forcedebug <[email protected]>
Co-authored-by: parthshah1 <[email protected]>
Co-authored-by: Piotr Galar <[email protected]>
Co-authored-by: ZenGround0 <[email protected]>
Co-authored-by: zenground0 <[email protected]>
Co-authored-by: forcedebug <[email protected]>
Co-authored-by: Jiaying Wang <[email protected]>
Co-authored-by: Phi-rjan <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Steven Allen <[email protected]>
Co-authored-by: Masih H. Derkani <[email protected]>
Co-authored-by: Lee <[email protected]>
Co-authored-by: Andrew Jackson (Ajax) <[email protected]>
Co-authored-by: beck <[email protected]>
Co-authored-by: 0x5459 <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Aarsh Shah <[email protected]>

* linter oops

* gen cleanup

* fix

* named returns are confusing

---------

Signed-off-by: forcedebug <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: Łukasz Magiera <[email protected]>
Co-authored-by: LexLuthr <[email protected]>
Co-authored-by: parthshah1 <[email protected]>
Co-authored-by: Piotr Galar <[email protected]>
Co-authored-by: ZenGround0 <[email protected]>
Co-authored-by: zenground0 <[email protected]>
Co-authored-by: forcedebug <[email protected]>
Co-authored-by: Jiaying Wang <[email protected]>
Co-authored-by: Phi-rjan <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Steven Allen <[email protected]>
Co-authored-by: Masih H. Derkani <[email protected]>
Co-authored-by: Lee <[email protected]>
Co-authored-by: beck <[email protected]>
Co-authored-by: 0x5459 <[email protected]>
Co-authored-by: Aarsh Shah <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants