Skip to content

Commit

Permalink
[Feature] UUIDs, protocol versioning, v2 protocol w/ dag-cbor messagi…
Browse files Browse the repository at this point in the history
…ng (#332)

* feat(net): initial dag-cbor protocol support

also added first roundtrip benchmark

* feat(requestid): use uuids for requestids

Ref: #278
Closes: #279
Closes: #281

* fix(requestmanager): make collect test requests with uuids sortable

* fix(requestid): print requestids as string uuids in logs

* fix(requestid): use string as base type for RequestId

* chore(requestid): wrap requestid string in a struct

* feat(libp2p): add v1.0.0 network compatibility

* chore(net): resolve most cbor + uuid merge problems

* feat(net): to/from ipld bindnode types, more cbor protoc improvements

* feat(net): introduce 2.0.0 protocol for dag-cbor

* fix(net): more bindnode dag-cbor protocol fixes

Not quite working yet, still need some upstream fixes and no extensions work
has been attempted yet.

* chore(metadata): convert metadata to bindnode

* chore(net,extensions): wire up IPLD extensions, expose as Node instead of []byte

* Extensions now working with new dag-cbor network protocol
* dag-cbor network protocol still not default, most tests are still exercising
  the existing v1 protocol
* Metadata now using bindnode instead of cbor-gen
* []byte for deferred extensions decoding is now replaced with datamodel.Node
  everywhere. Internal extensions now using some form of go-ipld-prime
	decode to convert them to local types (metadata using bindnode, others using
	direct inspection).
* V1 protocol also using dag-cbor decode of extensions data and exporting the
  bytes - this may be a breaking change for exising extensions - need to check
	whether this should be done differently. Maybe a try-decode and if it fails
	export a wrapped Bytes Node?

* fix(src): fix imports

* fix(mod): clean up go.mod

* fix(net): refactor message version format code to separate packages

* feat(net): activate v2 network as default

* fix(src): build error

* chore: remove GraphSyncMessage#Loggable

Ref: #332 (comment)

* chore: remove intermediate v1.1 pb protocol message type

v1.1.0 was introduced to start the transition to UUID RequestIDs. That
change has since been combined with the switch to DAG-CBOR messaging format
for a v2.0.0 protocol. Thus, this interim v1.1.0 format is no longer needed
and has not been used at all in a released version of go-graphsync.

Fixes: filecoin-project/lightning-planning#14

* fix: clarify comments re dag-cbor extension data

As per dission in #338, we are going
to be erroring on extension data that is not properly dag-cbor encoded from now
on

* feat: new LinkMetadata iface, integrate metadata into Response type (#342)

* feat(metadata): new LinkMetadata iface, integrate metadata into Response type

* LinkMetadata wrapper around existing metadata type to allow for easier
  backward-compat upgrade path
* integrate metadata directly into GraphSyncResponse type, moving it from an
  optional extension
* still deal with metadata as an extension for now—further work for v2 protocol
  will move it into the core message schema

Ref: #335

* feat(metadata): move metadata to core protocol, only use extension in v1 proto

* fix(metadata): bindnode expects Go enum strings to be at the type level

* fix(metadata): minor fixes, tidy up naming

* fix(metadata): make gofmt and staticcheck happy

* fix(metadata): docs and minor tweaks after review

Co-authored-by: Daniel Martí <[email protected]>

* fix: avoid double-encode for extension size estimation

Closes: filecoin-project/lightning-planning#15

* feat(requesttype): introduce RequestType enum to replace cancel&update bools (#352)

Closes: #345

* fix(metadata): extend round-trip tests to byte representation (#350)

* feat!(messagev2): tweak dag-cbor message schema (#354)

* feat!(messagev2): tweak dag-cbor message schema

For:

1. Efficiency: compacting the noisy structures into tuples representations and
   making top-level components of a message optional.
2. Migrations: providing a secondary mechanism to lean on for versioning if we
   want a gentler upgrade path than libp2p protocol versioning.

Closes: #351

* fix(messagev2): adjust schema per feedback

* feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID (#355)

* feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

Closes: #349

* fixup! feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

* fixup! feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

when using error type T, use *T with As, rather than **T

* fixup! feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

* fixup! feat(graphsync): unify req & resp Pause, Unpause & Cancel by RequestID

Co-authored-by: Daniel Martí <[email protected]>

* feat: SendUpdates() API to send only extension data to via existing request

* fix(responsemanager): send update while completing

If request has finished selector traversal but is still sending blocks,
I think it should be possible to send updates. As a side effect, this
fixes our race.

Logically, this makes sense, cause our external indicator that we're
done (completed response listener) has not been called.

* fix(requestmanager): revert change to pointer type

* Refactor async loading for simplicity and correctness (#356)

* feat(reconciledloader): first working version of reconciled loader

* feat(traversalrecorder): add better recorder for traversals

* feat(reconciledloader): pipe reconciled loader through code

style(lint): fix static checks

* Update requestmanager/reconciledloader/injest.go

Co-authored-by: Rod Vagg <[email protected]>

* feat(reconciledloader): respond to PR comments

Co-authored-by: Rod Vagg <[email protected]>

* fix(requestmanager): update test for rebase

Co-authored-by: Daniel Martí <[email protected]>
Co-authored-by: hannahhoward <[email protected]>
  • Loading branch information
3 people authored Feb 18, 2022
1 parent e8cdcb2 commit 8c66ab9
Show file tree
Hide file tree
Showing 84 changed files with 5,118 additions and 3,322 deletions.
3 changes: 2 additions & 1 deletion benchmarks/testnet/virtual.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"google.golang.org/protobuf/proto"

gsmsg "github.com/ipfs/go-graphsync/message"
gsmsgv1 "github.com/ipfs/go-graphsync/message/v1"
gsnet "github.com/ipfs/go-graphsync/network"
)

Expand Down Expand Up @@ -137,7 +138,7 @@ func (n *network) SendMessage(
rateLimiters[to] = rateLimiter
}

pbMsg, err := mes.ToProto()
pbMsg, err := gsmsgv1.NewMessageHandler().ToProto(peer.ID("foo"), mes)
if err != nil {
return err
}
Expand Down
16 changes: 7 additions & 9 deletions cidset/cidset.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,30 @@ import (
"errors"

"github.com/ipfs/go-cid"
"github.com/ipld/go-ipld-prime/datamodel"
"github.com/ipld/go-ipld-prime/fluent"
cidlink "github.com/ipld/go-ipld-prime/linking/cid"
"github.com/ipld/go-ipld-prime/node/basicnode"

"github.com/ipfs/go-graphsync/ipldutil"
)

// EncodeCidSet encodes a cid set into bytes for the do-no-send-cids extension
func EncodeCidSet(cids *cid.Set) ([]byte, error) {
func EncodeCidSet(cids *cid.Set) datamodel.Node {
list := fluent.MustBuildList(basicnode.Prototype.List, int64(cids.Len()), func(la fluent.ListAssembler) {
_ = cids.ForEach(func(c cid.Cid) error {
la.AssembleValue().AssignLink(cidlink.Link{Cid: c})
return nil
})
})
return ipldutil.EncodeNode(list)
return list
}

// DecodeCidSet decode a cid set from data for the do-no-send-cids extension
func DecodeCidSet(data []byte) (*cid.Set, error) {
list, err := ipldutil.DecodeNode(data)
if err != nil {
return nil, err
func DecodeCidSet(data datamodel.Node) (*cid.Set, error) {
if data.Kind() != datamodel.Kind_List {
return nil, errors.New("did not receive a list of CIDs")
}
set := cid.NewSet()
iter := list.ListIterator()
iter := data.ListIterator()
for !iter.Done() {
_, next, err := iter.Next()
if err != nil {
Expand Down
3 changes: 1 addition & 2 deletions cidset/cidset_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ func TestDecodeEncodeCidSet(t *testing.T) {
for _, c := range cids {
set.Add(c)
}
encoded, err := EncodeCidSet(set)
require.NoError(t, err, "encode errored")
encoded := EncodeCidSet(set)
decodedCidSet, err := DecodeCidSet(encoded)
require.NoError(t, err, "decode errored")
require.Equal(t, decodedCidSet.Len(), set.Len())
Expand Down
16 changes: 5 additions & 11 deletions dedupkey/dedupkey.go
Original file line number Diff line number Diff line change
@@ -1,27 +1,21 @@
package dedupkey

import (
"github.com/ipld/go-ipld-prime/datamodel"
"github.com/ipld/go-ipld-prime/node/basicnode"

"github.com/ipfs/go-graphsync/ipldutil"
)

// EncodeDedupKey returns encoded cbor data for string key
func EncodeDedupKey(key string) ([]byte, error) {
func EncodeDedupKey(key string) (datamodel.Node, error) {
nb := basicnode.Prototype.String.NewBuilder()
err := nb.AssignString(key)
if err != nil {
return nil, err
}
nd := nb.Build()
return ipldutil.EncodeNode(nd)
return nb.Build(), nil
}

// DecodeDedupKey returns a string key decoded from cbor data
func DecodeDedupKey(data []byte) (string, error) {
nd, err := ipldutil.DecodeNode(data)
if err != nil {
return "", err
}
return nd.AsString()
func DecodeDedupKey(data datamodel.Node) (string, error) {
return data.AsString()
}
Binary file removed docs/async-loading.png
Binary file not shown.
75 changes: 0 additions & 75 deletions docs/async-loading.puml

This file was deleted.

Binary file modified docs/processes.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
21 changes: 11 additions & 10 deletions docs/processes.puml
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,17 @@ partition "Top Level Interface" {
if (operation type) then (outgoing request or incoming response)
partition "Graphsync Requestor Implementation" {
:RequestManager;
if (operation type) then (incoming response)
partition "Verifying Queries" {
partition "Executing Requests" {
:TaskQueue;
fork
:ipld.Traverse;
:Executor;
fork again
:ipld.Traverse;
:Executor;
fork again
:ipld.Traverse;
:Executor;
end fork
}
if (operation type) then (verified responses)
partition "Collecting Responses" {
fork
:Response Collector;
Expand All @@ -33,21 +34,21 @@ end fork
}
:Responses returned to client;
stop
else (outgoing request)
else (request messages)
:Send Request To Network;
endif
}
else (incoming request)
partition "Graphsync Responder Implementation" {
:ResponseManager;
partition "Performing Queries" {
:PeerTaskQueue;
:TaskQueue;
fork
:ipld.Traverse;
:QueryExecutor;
fork again
:ipld.Traverse;
:QueryExecutor;
fork again
:ipld.Traverse;
:QueryExecutor;
end fork
}
}
Expand Down
Binary file added docs/request-execution.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
102 changes: 102 additions & 0 deletions docs/request-execution.puml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
@startuml Request Execution
participant "GraphSync\nTop Level\nInterface" as TLI
participant RequestManager
participant TaskQueue
participant RequestExecutor as RE
participant ReconciledLoader
participant TraversalRecord
participant Verifier
participant LocalStorage
participant Traverser
participant Network

== Initialization ==

TLI -> RequestManager ** : Setup
TLI -> RE ** : Setup
TLI -> TaskQueue ** : Setup

== Executing A Request ==

par
note over TLI : Request Initiation
TLI -> RequestManager : New Request
RequestManager -> RequestManager : Create Request Context
RequestManager -> TaskQueue : Push Request
else
note over RE: Request Execution
TaskQueue -> RE : Next Request\nTo Process
RE -> RequestManager : Initiate request execution
RequestManager -> Traverser ** : Create to manage selector traversal
RequestManager -> ReconciledLoader ** : create to manage
RequestManager -> RE : Traverser + ReconciledLoader
note over RE: Local loading phase
loop until traversal complete, request context cancelled, or missing block locally
Traverser -> RE : Request to load blocks\nto perform traversal
RE -> ReconciledLoader : Load next block
ReconciledLoader -> LocalStorage : Load Block
LocalStorage --> ReconciledLoader : Block or missing
ReconciledLoader -> TraversalRecord : Record link traversal
TraversalRecord --> ReconciledLoader
ReconciledLoader --> RE : Block or missing
opt block is present
RE --> Traverser : Next block to load
end
end
RE -> Network : Send Graphsync Request
RE -> ReconciledLoader : remote online
ReconciledLoader -> Verifier ** : Create new from traversal record
ReconciledLoader -> RE
note over RE: Remote loading phase
loop until traversal complete, request context cancelled, or missing block locally
Traverser -> RE : Request to load blocks\nto perform traversal
RE -> ReconciledLoader : Load next block
alt on missing path for remote
ReconciledLoader -> LocalStorage : Load Block
LocalStorage --> ReconciledLoader : Block or missing
else
loop until block loaded, missing, or error
opt new remote responses
alt verification not done
ReconciledLoader -> Verifier : verify next response
alt success
Verifier --> ReconciledLoader : verified
ReconciledLoader -> ReconciledLoader : wait for more responses
else failure
Verifier --> ReconciledLoader : error
end
else verification done
alt next response matches current block load

alt next response contains a block
ReconciledLoader -> LocalStorage : store remote block
LocalStorage --> ReconciledLoader
ReconciledLoader -> ReconciledLoader : block laoded from remote
else next response does not contain block
opt next response is missing
ReconciledLoader -> ReconciledLoader : record missing path
end
ReconciledLoader -> LocalStorage : load block
LocalStorage --> ReconciledLoader : block or missing
end
else next response doesn not match
ReconciledLoader -> ReconciledLoader : error
end
end
end
opt remote goes offline
ReconciledLoader -> LocalStorage : load block
LocalStorage --> ReconciledLoader : block or missing
end
end
ReconciledLoader -> TraversalRecord : Record link traversal
TraversalRecord --> ReconciledLoader
ReconciledLoader --> RE : Block, missing or error
RE -> Traverser : Next block to load
end
end
else
Network -> RequestManager : New Responses
RequestManager -> ReconciledLoader : Ingest Responses
end
@enduml
Loading

0 comments on commit 8c66ab9

Please sign in to comment.