Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Batch JSON-RPC support (rpc client & server) #1503

Merged
merged 25 commits into from
Mar 10, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
4ace7c8
allow non local tls disable for compatibility with private network in…
jakesylvestre Oct 31, 2019
706aff9
Merge pull request #1 from xplorfin/concurrent-write
jakesylvestre Nov 26, 2019
22338ac
new flag
jakesylvestre Dec 4, 2019
c5ecee4
add bulk request changes
jakesylvestre Dec 4, 2019
2b6b35c
server http
jakesylvestre Dec 4, 2019
e4c7407
bulk first pass
jakesylvestre Dec 4, 2019
c25e1e8
tls unwrangle
jakesylvestre Dec 4, 2019
9d2e844
Merge pull request #4 from xplorfin/rpcbulk
jakesylvestre Dec 4, 2019
e740498
Revert "tls unwrangle"
jakesylvestre Dec 4, 2019
a59d23d
add bulk mode
jakesylvestre Dec 4, 2019
fe2192d
batch chan
jakesylvestre Dec 4, 2019
a5d1ff5
add batch method
jakesylvestre Dec 5, 2019
6278ae4
(rpcclient) add batch requests
jakesylvestre Dec 5, 2019
bd91ca2
(rpcclient) dereference fix
jakesylvestre Dec 5, 2019
301229f
Revert "tls unwrangle"
jakesylvestre Dec 30, 2019
935dfb7
fix issues reviewed by @torkelrogstad
jakesylvestre Dec 31, 2019
ef60e30
fix nits
jakesylvestre Dec 31, 2019
d67676a
Merge branch 'master' into rpcbulk
jakesylvestre Jan 21, 2020
b1f9e41
unfalse
jakesylvestre Jan 22, 2020
1d228c1
ci fixes
jakesylvestre Feb 2, 2020
6521f5e
uncache
jakesylvestre Feb 2, 2020
eba3b3f
Revert "uncache"
jakesylvestre Feb 2, 2020
7855b30
bust cache
jakesylvestre Feb 2, 2020
4012d26
remove gocaching as per @pavel in fe003e236663978c5c792e0ed077d5e83b4…
jakesylvestre Feb 9, 2020
69fab28
Merge branch 'rpcbulk' of https://github.com/xplorfin/btcd into rpcbulk
jakesylvestre Feb 9, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ language: go
cache:
directories:
- $GOCACHE
- $GOPATH
- $GOPATH/pkg/mod
- $GOPATH/github.com/golang
- $GOPATH/gopkg.in/alecthomas
Expand Down
4 changes: 2 additions & 2 deletions btcjson/btcdextcmds_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ func TestBtcdExtCmds(t *testing.T) {
for i, test := range tests {
// Marshal the command as created by the new static command
// creation function.
marshalled, err := btcjson.MarshalCmd(testID, test.staticCmd())
marshalled, err := btcjson.MarshalCmd("1.0", testID, test.staticCmd())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you changing from a variable to a hardcoded number here (and all the places below?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fair point, I'll have to refactor. I've held off on squashing till these revisions are complete, but can do this earlier fi it helps your review

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate the feedback

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally I agree about waiting to squash until revisions are done but this needs the rebase before reviewing it for real so you might as well squash at the same time (I hate making people do extra steps if I don't have to).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem

if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand All @@ -217,7 +217,7 @@ func TestBtcdExtCmds(t *testing.T) {

// Marshal the command as created by the generic new command
// creation function.
marshalled, err = btcjson.MarshalCmd(testID, cmd)
marshalled, err = btcjson.MarshalCmd("1.0", testID, cmd)
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand Down
4 changes: 2 additions & 2 deletions btcjson/btcwalletextcmds_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ func TestBtcWalletExtCmds(t *testing.T) {
for i, test := range tests {
// Marshal the command as created by the new static command
// creation function.
marshalled, err := btcjson.MarshalCmd(testID, test.staticCmd())
marshalled, err := btcjson.MarshalCmd("1.0", testID, test.staticCmd())
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand All @@ -169,7 +169,7 @@ func TestBtcWalletExtCmds(t *testing.T) {

// Marshal the command as created by the generic new command
// creation function.
marshalled, err = btcjson.MarshalCmd(testID, cmd)
marshalled, err = btcjson.MarshalCmd("1.0", testID, cmd)
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand Down
4 changes: 2 additions & 2 deletions btcjson/chainsvrcmds_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1092,7 +1092,7 @@ func TestChainSvrCmds(t *testing.T) {
for i, test := range tests {
// Marshal the command as created by the new static command
// creation function.
marshalled, err := btcjson.MarshalCmd(testID, test.staticCmd())
marshalled, err := btcjson.MarshalCmd("1.0", testID, test.staticCmd())
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand All @@ -1117,7 +1117,7 @@ func TestChainSvrCmds(t *testing.T) {

// Marshal the command as created by the generic new command
// creation function.
marshalled, err = btcjson.MarshalCmd(testID, cmd)
marshalled, err = btcjson.MarshalCmd("1.0", testID, cmd)
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand Down
4 changes: 2 additions & 2 deletions btcjson/chainsvrwscmds_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ func TestChainSvrWsCmds(t *testing.T) {
for i, test := range tests {
// Marshal the command as created by the new static command
// creation function.
marshalled, err := btcjson.MarshalCmd(testID, test.staticCmd())
marshalled, err := btcjson.MarshalCmd("1.0", testID, test.staticCmd())
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand All @@ -257,7 +257,7 @@ func TestChainSvrWsCmds(t *testing.T) {

// Marshal the command as created by the generic new command
// creation function.
marshalled, err = btcjson.MarshalCmd(testID, cmd)
marshalled, err = btcjson.MarshalCmd("1.0", testID, cmd)
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand Down
4 changes: 2 additions & 2 deletions btcjson/chainsvrwsntfns_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ func TestChainSvrWsNtfns(t *testing.T) {
for i, test := range tests {
// Marshal the notification as created by the new static
// creation function. The ID is nil for notifications.
marshalled, err := btcjson.MarshalCmd(nil, test.staticNtfn())
marshalled, err := btcjson.MarshalCmd("1.0", nil, test.staticNtfn())
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand All @@ -256,7 +256,7 @@ func TestChainSvrWsNtfns(t *testing.T) {
// Marshal the notification as created by the generic new
// notification creation function. The ID is nil for
// notifications.
marshalled, err = btcjson.MarshalCmd(nil, cmd)
marshalled, err = btcjson.MarshalCmd("1.0", nil, cmd)
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand Down
4 changes: 2 additions & 2 deletions btcjson/cmdparse.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ func makeParams(rt reflect.Type, rv reflect.Value) []interface{} {
// is suitable for transmission to an RPC server. The provided command type
// must be a registered type. All commands provided by this package are
// registered by default.
func MarshalCmd(id interface{}, cmd interface{}) ([]byte, error) {
func MarshalCmd(rpcVersion string, id interface{}, cmd interface{}) ([]byte, error) {
// Look up the cmd type and error out if not registered.
rt := reflect.TypeOf(cmd)
registerLock.RLock()
Expand All @@ -59,7 +59,7 @@ func MarshalCmd(id interface{}, cmd interface{}) ([]byte, error) {
params := makeParams(rt.Elem(), rv.Elem())

// Generate and marshal the final JSON-RPC request.
rawCmd, err := NewRequest(id, method, params)
rawCmd, err := NewRequest(rpcVersion, id, method, params)
if err != nil {
return nil, err
}
Expand Down
2 changes: 1 addition & 1 deletion btcjson/cmdparse_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -433,7 +433,7 @@ func TestMarshalCmdErrors(t *testing.T) {

t.Logf("Running %d tests", len(tests))
for i, test := range tests {
_, err := btcjson.MarshalCmd(test.id, test.cmd)
_, err := btcjson.MarshalCmd("1.0", test.id, test.cmd)
if reflect.TypeOf(err) != reflect.TypeOf(test.err) {
t.Errorf("Test #%d (%s) wrong error - got %T (%v), "+
"want %T", i, test.name, err, err, test.err)
Expand Down
8 changes: 4 additions & 4 deletions btcjson/example_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ func ExampleMarshalCmd() {
// server. Typically the client would increment the id here which is
// request so the response can be identified.
id := 1
marshalledBytes, err := btcjson.MarshalCmd(id, gbCmd)
marshalledBytes, err := btcjson.MarshalCmd("1.0", id, gbCmd)
if err != nil {
fmt.Println(err)
return
Expand Down Expand Up @@ -97,7 +97,7 @@ func ExampleUnmarshalCmd() {
func ExampleMarshalResponse() {
// Marshal a new JSON-RPC response. For example, this is a response
// to a getblockheight request.
marshalledBytes, err := btcjson.MarshalResponse(1, 350001, nil)
marshalledBytes, err := btcjson.MarshalResponse("1.0",1, 350001, nil)
if err != nil {
fmt.Println(err)
return
Expand All @@ -109,7 +109,7 @@ func ExampleMarshalResponse() {
fmt.Printf("%s\n", marshalledBytes)

// Output:
// {"result":350001,"error":null,"id":1}
// {"jsonrpc":"1.0","result":350001,"error":null,"id":1}
}

// This example demonstrates how to unmarshal a JSON-RPC response and then
Expand All @@ -118,7 +118,7 @@ func Example_unmarshalResponse() {
// Ordinarily this would be read from the wire, but for this example,
// it is hard coded here for clarity. This is an example response to a
// getblockheight request.
data := []byte(`{"result":350001,"error":null,"id":1}`)
data := []byte(`{"jsonrpc":"1.0","result":350001,"error":null,"id":1}`)

// Unmarshal the raw bytes from the wire into a JSON-RPC response.
var response btcjson.Response
Expand Down
117 changes: 89 additions & 28 deletions btcjson/jsonrpc.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,64 @@ type Request struct {
ID interface{} `json:"id"`
}

// NewRequest returns a new JSON-RPC 1.0 request object given the provided id,
// method, and parameters. The parameters are marshalled into a json.RawMessage
// for the Params field of the returned request object. This function is only
// provided in case the caller wants to construct raw requests for some reason.
//
// Typically callers will instead want to create a registered concrete command
// type with the NewCmd or New<Foo>Cmd functions and call the MarshalCmd
// function with that command to generate the marshalled JSON-RPC request.
func NewRequest(id interface{}, method string, params []interface{}) (*Request, error) {
// UnmarshalJSON is a custom unmarshal func for the Request struct. The param
// field defaults to an empty json.RawMessage array it is omitted by the request
// or nil if the supplied value is invalid.
func (request *Request) UnmarshalJSON(b []byte) error {
var data map[string]interface{}
err := json.Unmarshal(b, &data)
if err != nil {
return err
}

request.ID = data["id"]
methodValue, hasMethod := data["method"]
if hasMethod {
request.Method = methodValue.(string)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cast here and on line 93 seems a bit dangerous to me, would blow up if the fields are accessed later and it turns out we have the wrong type.
What's your thoughts on this?

metod, ok := methodValue.(string)
if ok {
  request.Method = method
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting point. I don't see an alternative though

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@torkelrogstad I've gone ahead and addressed this on #1583

image

}
jsonrpcValue, hasJsonrpc := data["jsonrpc"]
if hasJsonrpc {
request.Jsonrpc = jsonrpcValue.(string)
}
paramsValue, hasParams := data["params"]
if !hasParams {
// set the request param to an empty array if it is ommited in the request
request.Params = []json.RawMessage{}
// assert the request params is an array of data
} else if params, paramsOk := paramsValue.([]interface{}); paramsOk {
rawParams := make([]json.RawMessage, 0, len(params))
for _, param := range params {
marshalledParam, err := json.Marshal(param)
if err != nil {
return err
}
rawMessage := json.RawMessage(marshalledParam)
rawParams = append(rawParams, rawMessage)
}

request.Params = rawParams
} else {
return Error{Description: "No response received"}
}

return nil
}

// NewRequest returns a new JSON-RPC request object given the provided rpc
// version, id, method, and parameters. The parameters are marshalled into a
// json.RawMessage for the Params field of the returned request object. This
// function is only provided in case the caller wants to construct raw requests
// for some reason. Typically callers will instead want to create a registered
// concrete command type with the NewCmd or New<Foo>Cmd functions and call the
// MarshalCmd function with that command to generate the marshalled JSON-RPC
// request.
func NewRequest(rpcVersion string, id interface{}, method string, params []interface{}) (*Request, error) {
// default to JSON-RPC 1.0 if RPC type is not specified
if rpcVersion != "2.0" && rpcVersion != "1.0" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This strikes me as a bit dangerous. Two potential solutions, could (should, perhaps) do both IMO:

  1. Introduce a type for it
type rpcVersion string
var (
  rpcVersion1 rpcVersion = rpcVersion("1.0")
  rpcVersion2 rpcVersion = rpcVersion("2.0")
)
  1. I think it's better to error out here, instead of silently changing behavior.

str := fmt.Sprintf("rpcversion '%s' is invalid", rpcVersion)
return nil, makeError(ErrInvalidType, str)
}

if !IsValidIDType(id) {
str := fmt.Sprintf("the id of type '%T' is invalid", id)
return nil, makeError(ErrInvalidType, str)
Expand All @@ -98,51 +147,63 @@ func NewRequest(id interface{}, method string, params []interface{}) (*Request,
}

return &Request{
Jsonrpc: "1.0",
Jsonrpc: rpcVersion,
ID: id,
Method: method,
Params: rawParams,
}, nil
}

// Response is the general form of a JSON-RPC response. The type of the Result
// field varies from one command to the next, so it is implemented as an
// interface. The ID field has to be a pointer for Go to put a null in it when
// Response is the general form of a JSON-RPC response. The type of the
// Result field varies from one command to the next, so it is implemented as an
// interface. The ID field has to be a pointer to allow for a nil value when
// empty.
type Response struct {
Result json.RawMessage `json:"result"`
Error *RPCError `json:"error"`
ID *interface{} `json:"id"`
Jsonrpc string `json:"jsonrpc"`
Result json.RawMessage `json:"result"`
Error *RPCError `json:"error"`
ID *interface{} `json:"id"`
}

// NewResponse returns a new JSON-RPC response object given the provided id,
// marshalled result, and RPC error. This function is only provided in case the
// caller wants to construct raw responses for some reason.
//
// NewResponse returns a new JSON-RPC response object given the provided rpc
// version, id, marshalled result, and RPC error. This function is only
// provided in case the caller wants to construct raw responses for some reason.
// Typically callers will instead want to create the fully marshalled JSON-RPC
// response to send over the wire with the MarshalResponse function.
func NewResponse(id interface{}, marshalledResult []byte, rpcErr *RPCError) (*Response, error) {
func NewResponse(rpcVersion string, id interface{}, marshalledResult []byte, rpcErr *RPCError) (*Response, error) {
if rpcVersion != "2.0" && rpcVersion != "1.0" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment about RPC versions from above apply here

str := fmt.Sprintf("rpcversion '%s' is invalid", rpcVersion)
return nil, makeError(ErrInvalidType, str)
}

if !IsValidIDType(id) {
str := fmt.Sprintf("the id of type '%T' is invalid", id)
return nil, makeError(ErrInvalidType, str)
}

pid := &id
return &Response{
Result: marshalledResult,
Error: rpcErr,
ID: pid,
Jsonrpc: rpcVersion,
Result: marshalledResult,
Error: rpcErr,
ID: pid,
}, nil
}

// MarshalResponse marshals the passed id, result, and RPCError to a JSON-RPC
// response byte slice that is suitable for transmission to a JSON-RPC client.
func MarshalResponse(id interface{}, result interface{}, rpcErr *RPCError) ([]byte, error) {
// MarshalResponse marshals the passed rpc version, id, result, and RPCError to
// a JSON-RPC response byte slice that is suitable for transmission to a
// JSON-RPC client.
func MarshalResponse(rpcVersion string, id interface{}, result interface{}, rpcErr *RPCError) ([]byte, error) {
if rpcVersion != "2.0" && rpcVersion != "1.0" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RPC version comment from above

str := fmt.Sprintf("rpcversion '%s' is invalid", rpcVersion)
return nil, makeError(ErrInvalidType, str)
}

marshalledResult, err := json.Marshal(result)
if err != nil {
return nil, err
}
response, err := NewResponse(id, marshalledResult, rpcErr)
response, err := NewResponse(rpcVersion, id, marshalledResult, rpcErr)
if err != nil {
return nil, err
}
Expand Down
12 changes: 6 additions & 6 deletions btcjson/jsonrpc_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,22 +68,22 @@ func TestMarshalResponse(t *testing.T) {
name: "ordinary bool result with no error",
result: true,
jsonErr: nil,
expected: []byte(`{"result":true,"error":null,"id":1}`),
expected: []byte(`{"jsonrpc":"1.0","result":true,"error":null,"id":1}`),
},
{
name: "result with error",
result: nil,
jsonErr: func() *btcjson.RPCError {
return btcjson.NewRPCError(btcjson.ErrRPCBlockNotFound, "123 not found")
}(),
expected: []byte(`{"result":null,"error":{"code":-5,"message":"123 not found"},"id":1}`),
expected: []byte(`{"jsonrpc":"1.0","result":null,"error":{"code":-5,"message":"123 not found"},"id":1}`),
},
}

t.Logf("Running %d tests", len(tests))
for i, test := range tests {
_, _ = i, test
marshalled, err := btcjson.MarshalResponse(testID, test.result, test.jsonErr)
marshalled, err := btcjson.MarshalResponse("1.0", testID, test.result, test.jsonErr)
if err != nil {
t.Errorf("Test #%d (%s) unexpected error: %v", i,
test.name, err)
Expand All @@ -104,7 +104,7 @@ func TestMiscErrors(t *testing.T) {

// Force an error in NewRequest by giving it a parameter type that is
// not supported.
_, err := btcjson.NewRequest(nil, "test", []interface{}{make(chan int)})
_, err := btcjson.NewRequest("1.0", nil, "test", []interface{}{make(chan int)})
if err == nil {
t.Error("NewRequest: did not receive error")
return
Expand All @@ -113,7 +113,7 @@ func TestMiscErrors(t *testing.T) {
// Force an error in MarshalResponse by giving it an id type that is not
// supported.
wantErr := btcjson.Error{ErrorCode: btcjson.ErrInvalidType}
_, err = btcjson.MarshalResponse(make(chan int), nil, nil)
_, err = btcjson.MarshalResponse("1.0", make(chan int), nil, nil)
if jerr, ok := err.(btcjson.Error); !ok || jerr.ErrorCode != wantErr.ErrorCode {
t.Errorf("MarshalResult: did not receive expected error - got "+
"%v (%[1]T), want %v (%[2]T)", err, wantErr)
Expand All @@ -122,7 +122,7 @@ func TestMiscErrors(t *testing.T) {

// Force an error in MarshalResponse by giving it a result type that
// can't be marshalled.
_, err = btcjson.MarshalResponse(1, make(chan int), nil)
_, err = btcjson.MarshalResponse("1.0",1, make(chan int), nil)
if _, ok := err.(*json.UnsupportedTypeError); !ok {
wantErr := &json.UnsupportedTypeError{}
t.Errorf("MarshalResult: did not receive expected error - got "+
Expand Down
4 changes: 2 additions & 2 deletions btcjson/walletsvrcmds_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1210,7 +1210,7 @@ func TestWalletSvrCmds(t *testing.T) {
for i, test := range tests {
// Marshal the command as created by the new static command
// creation function.
marshalled, err := btcjson.MarshalCmd(testID, test.staticCmd())
marshalled, err := btcjson.MarshalCmd("1.0", testID, test.staticCmd())
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand All @@ -1234,7 +1234,7 @@ func TestWalletSvrCmds(t *testing.T) {

// Marshal the command as created by the generic new command
// creation function.
marshalled, err = btcjson.MarshalCmd(testID, cmd)
marshalled, err = btcjson.MarshalCmd("1.0", testID, cmd)
if err != nil {
t.Errorf("MarshalCmd #%d (%s) unexpected error: %v", i,
test.name, err)
Expand Down
Loading