-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Index publishing work #673
Conversation
…rovider (#647) * reference prov integration * Apply suggestions from code review Co-authored-by: dirkmc <[email protected]> * move provider callback to start * fix: lint Co-authored-by: dirkmc <[email protected]>
…o-fil-markets into feat/retrieve-any-cid
6d24ebc
to
eb2fe1f
Compare
* add fullnodeApi to Provider * add idxProvHost * connect to full node before making announcement * fix compilation Co-authored-by: Anton Evangelatov <[email protected]>
* use NetAddrListener iface * fix compilation * fix imports Co-authored-by: Aarsh Shah <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #673 +/- ##
==========================================
- Coverage 59.68% 58.63% -1.04%
==========================================
Files 64 64
Lines 5262 5356 +94
==========================================
Hits 3140 3140
- Misses 1790 1878 +88
- Partials 332 338 +6
Continue to review full report at Codecov.
|
* remove libp2p host connect in favour of mesh creator interface * update test harness
Upgrade to the latest `index-provider` which includes two main changes: 1. update to go-legs message format 2. cache eviction bug fix
Upgrade to `index-provider` `v0.3.0`
…-dont-return log failure to connect to full node without returning it
Update to the head of the PR that introduces indexing integration in `go-fil-markets` so that failure to connect to full node is logged only instead of crashing markets process. Relates to: - filecoin-project/go-fil-markets#673 - ipni/index-provider#177
Update to the head of the PR that introduces indexing integration in `go-fil-markets` so that failure to connect to full node is logged only instead of crashing markets process. Relates to: - filecoin-project/go-fil-markets#673 - ipni/index-provider#177
Depend on a concrete tagged release of dagstore that contains fixes to dagstore repo bloat.
Update indexing dependencies to latest along with go-fil-markets dependency, to the head of: - filecoin-project/go-fil-markets#673
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My biggest meta comment is you should remove:
- CIDInfo entirely from the PieceStore -- it's unused
- Remove the whole "MetadataPath" if that machinery still exists -- it's also unused
It's important to make clear to someone reading the code that the single source of truth on which payloadCIDs are in which pieces is now in the DagStore only.
@@ -225,6 +225,7 @@ func TestClient_FindProviders(t *testing.T) { | |||
// retrieval deal for the same payload CID with the same peer as an existing | |||
// active deal | |||
func TestClient_DuplicateRetrieve(t *testing.T) { | |||
t.Skip("flaky test") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did it become more flaky in this PR?
- Yes -- then let's figure that out
- No -- I would not throw in a skip as part of a feature PR -- i'd just make an issue and if we can't tackle it now, let's make a seperate PR to add a skip
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -448,6 +461,65 @@ func (p *Provider) pieceInUnsealedSector(ctx context.Context, pieceInfo piecesto | |||
return false | |||
} | |||
|
|||
func (p *Provider) storageDealsForPiece(clientSpecificPiece bool, payloadCID cid.Cid, pieceInfo piecestore.PieceInfo) ([]abi.DealID, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an existing problem (not introduced by this PR) but this method has a weird logic to it -
- if the first parameter is true, then the third parameter is used, the second parameter unused
- if the first parameter is false, the second parameter is used, the third parameter unused
To me this says: make it two methods -- and also, consider the interrelated logic from the the HandleQueryStream/getPieceInfoFromCid and see if there's a way to simply.
Definitely non-blocking, just noting.
@@ -177,6 +190,25 @@ func (p *Provider) Start(ctx context.Context) error { | |||
log.Error(err.Error()) | |||
} | |||
}() | |||
|
|||
// connect the index provider node with the full node and protect that connection |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This feels like a very odd concern to put in go-fil-markets... maybe I don't know the context? I would expect it to live in the Lotus markets process. (or later Boost)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fil-markets manages the libp2p host though, i guess?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do wanna make sure that we do this everytime before we announce a deal in markets just to be sure we have a connection to the daemon before making the announcement.
But yeah, will move this to Boost when we integrate the index provider in Boost.
// announce the deal to the network indexer | ||
annCid, err := environment.AnnounceIndex(ctx.Context(), deal) | ||
if err != nil { | ||
log.Errorw("failed to announce index via reference provider", "proposalCid", deal.ProposalCid, "err", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should the fact that an error occurred in publish be saved to the deal state? Other than logs, how do I know which of my deals got announced successfully? Do I always want AnnounceAllDealsToIndex() to announce everything?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hannahhoward The next time a deal gets announced, because of the IPLD magic in the index provider, it will also announce all the previous un-announced deal when the network indexer asks for the latest.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aarshkshah1992 not necessarily.
If the error happens because the advertisement failed to persist then that content will never get advertised.
Looking at the code, such errors can happen before call to NotifyPut
e.g. in metadata encoding on markets side or inside the provider before persistence.
The case you mentioned above will only work if the new advertisement is added to the internal IPLD chain of ads but not announced to the network due to say gossipsub error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, AnnounceAllDealsToIndex
is idempotent because the Index provider is idempotent. It wont actually have to serve indices for deals for which it has already served indices.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not really a part of this work. I started doing thus but looks like a bigger refactor given that it's very pervasive in all the tests too. I've created #684 to track this and we can do this in a follow up PR.
Have added a comment for this. |
- Add comment to clarify the reason for loop in testkit - Trim common prefix in state printed in CLI commands for better readability - Upgrade to a tagged release of `go-fil-markets` that includes indexing work; see: filecoin-project/go-fil-markets#673 - Fix typo in CLI usage. - Add comments to note that it is safe to use fx `OnStart` context when starting the provider engine. - Fix string concatenation in error message formatting.
No description provided.