Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index publishing work #673

Merged
merged 39 commits into from
Mar 1, 2022
Merged

Index publishing work #673

merged 39 commits into from
Mar 1, 2022

Conversation

aarshkshah1992
Copy link
Collaborator

No description provided.

aarshkshah1992 and others added 6 commits January 26, 2022 10:54
* add fullnodeApi to Provider

* add idxProvHost

* connect to full node before making announcement

* fix compilation

Co-authored-by: Anton Evangelatov <[email protected]>
* use NetAddrListener iface

* fix compilation

* fix imports

Co-authored-by: Aarsh Shah <[email protected]>
@codecov-commenter
Copy link

codecov-commenter commented Feb 2, 2022

Codecov Report

Merging #673 (59b99bf) into master (5e87024) will decrease coverage by 1.05%.
The diff coverage is 30.15%.

❗ Current head 59b99bf differs from pull request most recent head a6e18af. Consider uploading reports for the commit a6e18af to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #673      +/-   ##
==========================================
- Coverage   59.68%   58.63%   -1.04%     
==========================================
  Files          64       64              
  Lines        5262     5356      +94     
==========================================
  Hits         3140     3140              
- Misses       1790     1878      +88     
- Partials      332      338       +6     
Impacted Files Coverage Δ
storagemarket/impl/provider_environments.go 9.18% <0.00%> (-1.35%) ⬇️
stores/dagstore.go 0.00% <ø> (ø)
stores/filestore.go 54.55% <ø> (ø)
storagemarket/impl/provider.go 23.12% <12.33%> (-2.54%) ⬇️
...oragemarket/impl/providerstates/provider_states.go 84.99% <62.50%> (-0.58%) ⬇️
retrievalmarket/impl/provider.go 69.32% <65.00%> (-1.41%) ⬇️
retrievalmarket/impl/provider_environments.go 69.05% <100.00%> (+2.97%) ⬆️
retrievalmarket/impl/client.go 67.22% <0.00%> (-5.39%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f746f85...a6e18af. Read the comment docs.

nonsense and others added 5 commits February 3, 2022 19:24
* remove libp2p host connect in favour of mesh creator interface

* update test harness
Upgrade to the latest `index-provider` which includes two main changes:

1. update to go-legs message format
2. cache eviction bug fix
…-dont-return

log failure to connect to full node without returning it
masih added a commit to filecoin-project/lotus that referenced this pull request Feb 15, 2022
Update to the head of the PR that introduces indexing integration in
`go-fil-markets` so that failure to connect to full node is logged only
instead of crashing markets process.

Relates to:
 - filecoin-project/go-fil-markets#673
 - ipni/index-provider#177
masih added a commit to filecoin-project/lotus that referenced this pull request Feb 15, 2022
Update to the head of the PR that introduces indexing integration in
`go-fil-markets` so that failure to connect to full node is logged only
instead of crashing markets process.

Relates to:
 - filecoin-project/go-fil-markets#673
 - ipni/index-provider#177
masih added a commit to filecoin-project/lotus that referenced this pull request Feb 23, 2022
Update indexing dependencies to latest along with go-fil-markets
dependency, to the head of:
- filecoin-project/go-fil-markets#673
Copy link
Collaborator

@hannahhoward hannahhoward left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My biggest meta comment is you should remove:

  • CIDInfo entirely from the PieceStore -- it's unused
  • Remove the whole "MetadataPath" if that machinery still exists -- it's also unused

It's important to make clear to someone reading the code that the single source of truth on which payloadCIDs are in which pieces is now in the DagStore only.

@@ -225,6 +225,7 @@ func TestClient_FindProviders(t *testing.T) {
// retrieval deal for the same payload CID with the same peer as an existing
// active deal
func TestClient_DuplicateRetrieve(t *testing.T) {
t.Skip("flaky test")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did it become more flaky in this PR?

  1. Yes -- then let's figure that out
  2. No -- I would not throw in a skip as part of a feature PR -- i'd just make an issue and if we can't tackle it now, let's make a seperate PR to add a skip

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -448,6 +461,65 @@ func (p *Provider) pieceInUnsealedSector(ctx context.Context, pieceInfo piecesto
return false
}

func (p *Provider) storageDealsForPiece(clientSpecificPiece bool, payloadCID cid.Cid, pieceInfo piecestore.PieceInfo) ([]abi.DealID, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an existing problem (not introduced by this PR) but this method has a weird logic to it -

  • if the first parameter is true, then the third parameter is used, the second parameter unused
  • if the first parameter is false, the second parameter is used, the third parameter unused

To me this says: make it two methods -- and also, consider the interrelated logic from the the HandleQueryStream/getPieceInfoFromCid and see if there's a way to simply.

Definitely non-blocking, just noting.

@@ -177,6 +190,25 @@ func (p *Provider) Start(ctx context.Context) error {
log.Error(err.Error())
}
}()

// connect the index provider node with the full node and protect that connection
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a very odd concern to put in go-fil-markets... maybe I don't know the context? I would expect it to live in the Lotus markets process. (or later Boost)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fil-markets manages the libp2p host though, i guess?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do wanna make sure that we do this everytime before we announce a deal in markets just to be sure we have a connection to the daemon before making the announcement.

But yeah, will move this to Boost when we integrate the index provider in Boost.

// announce the deal to the network indexer
annCid, err := environment.AnnounceIndex(ctx.Context(), deal)
if err != nil {
log.Errorw("failed to announce index via reference provider", "proposalCid", deal.ProposalCid, "err", err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the fact that an error occurred in publish be saved to the deal state? Other than logs, how do I know which of my deals got announced successfully? Do I always want AnnounceAllDealsToIndex() to announce everything?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hannahhoward The next time a deal gets announced, because of the IPLD magic in the index provider, it will also announce all the previous un-announced deal when the network indexer asks for the latest.

Copy link
Member

@masih masih Feb 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aarshkshah1992 not necessarily.
If the error happens because the advertisement failed to persist then that content will never get advertised.
Looking at the code, such errors can happen before call to NotifyPut e.g. in metadata encoding on markets side or inside the provider before persistence.

The case you mentioned above will only work if the new advertisement is added to the internal IPLD chain of ads but not announced to the network due to say gossipsub error.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, AnnounceAllDealsToIndex is idempotent because the Index provider is idempotent. It wont actually have to serve indices for deals for which it has already served indices.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aarshkshah1992
Copy link
Collaborator Author

My biggest meta comment is you should remove:

  • CIDInfo entirely from the PieceStore -- it's unused
  • Remove the whole "MetadataPath" if that machinery still exists -- it's also unused

It's not really a part of this work. I started doing thus but looks like a bigger refactor given that it's very pervasive in all the tests too. I've created #684 to track this and we can do this in a follow up PR.

It's important to make clear to someone reading the code that the single source of truth on which payloadCIDs are in which pieces is now in the DagStore only.

Have added a comment for this.

@aarshkshah1992 aarshkshah1992 merged commit 871e17d into master Mar 1, 2022
masih added a commit to filecoin-project/lotus that referenced this pull request Mar 2, 2022
- Add comment to clarify the reason for loop in testkit
- Trim common prefix in state printed in CLI commands for better
  readability
- Upgrade to a tagged release of `go-fil-markets` that includes indexing
  work; see: filecoin-project/go-fil-markets#673
- Fix typo in CLI usage.
- Add comments to note that it is safe to use fx `OnStart` context when
  starting the provider engine.
- Fix string concatenation in error message formatting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants