diff --git a/specs/diagram.mermaid b/specs/diagram.mermaid new file mode 100644 index 0000000..ad498a3 --- /dev/null +++ b/specs/diagram.mermaid @@ -0,0 +1,19 @@ +sequenceDiagram + +participant 1 as NODE 1 +participant 2 as NODE 2 +participant 3 as NODE 3 + + +1->>1: RECEIVE NEW BLOCK: 0xCAB +1->>2: GOSSIP: 0xCAB +1->>3: GOSSIP: 0xCAB + +2->>1: RPC: GET_BLOCK_BODIES: 0xCAB +1->>2: RPC: BLOCK_BODIES:[{0xCAB}] + +2->>2: RECEIVE NEW BLOCK: 0xCAB + +2->>3: GOSSIP: 0xCAB + +2->>1: GOSSIP: 0xCAB diff --git a/specs/diagram.png b/specs/diagram.png new file mode 100644 index 0000000..59dc67d Binary files /dev/null and b/specs/diagram.png differ diff --git a/specs/gossip.md b/specs/gossip.md deleted file mode 100644 index 9f22aad..0000000 --- a/specs/gossip.md +++ /dev/null @@ -1,84 +0,0 @@ -# Messages -These messages are used to define a gossip protocol for Ethereum 2.0. These messages define a gossip protocol for clients to replicate information with each other. - -# Serialization and compression - -All messages of the command GOSSIP use SSZ serialization and snappy compression. - -# Envelope -All messages follow the envelope standard to Hobbits as described in [protocol.md]. - -This application protocol is classified under the `GOSSIP` command. - -The message must contain the following headers: - -| Header name | Type | Notes | -|-------------|------|-------| -| method_id | uint8| the method used in this exchange, as described below | -| attributes | string | the type of message being exachanged, timestamp| -| message_hash | bytes32 | a hash uniquely representing the message contents, with a hash function up to the application | -| hash_signature | bytes32 | a signature of the message hash with a public key identifying the node sending data | - -Example: - -```java -EWP 0.2 GOSSIP 24 0 -{ - "method_id": 3, - "attributes": "BLOCK,1560471980" - "message_hash": "0x9D686F6262697473206172652074776F20616E6420666F75722066656574", - "hash_signature": "0x0000000009A4672656E63682070656F706C6520617265207468652062657374" -} -``` - -The message may contain additional headers specified by the application layer. - -# Methods - -## 0x00 GOSSIP - -Nodes use `GOSSIP` methods to send data to other nodes in the network. - -The body of a `GOSSIP` method consists in the data being gossiped. - -The `message_hash` header value must match the hash of the contents of the body according to a predefined hash function defined by the application. - -## 0x01 PRUNE - -Nodes use `PRUNE` messages to inform other nodes that they are removed from the list of peers that will receive data from them. -Instead of sending data, nodes will send attestations as `IHAVE` messages. - -The header may contain the `message_hash` of a message that triggered the pruning. - -## 0x02 GRAFT - -Nodes use `PRUNE` messages to inform other nodes that they are added to the list of peers that will receive data from them. -Instead of sending attestations as `IHAVE` messages, nodes will send data as `GOSSIP` messages. - -No body is present in `GRAFT` messages. - -The header may contain the `message_hash` of a message triggered the graft. - -Targets should reply with a `GOSSIP` message sending the message matching the hash. - -## 0x03 IHAVE - -Nodes use `IHAVE` messages to inform other nodes that they are in possession of data that matches the signature they are sending. - -No body is present in `IHAVE` messages. - -The header must contain the `message_hash` with the value of the hash of the data attested by the peer. - -# Attributes -There are two types of attributes: `message_type` , `unix_timestamp` - -## Message_type -GOSSIP allows to gossip different types of payloads. To differentiate them, it uses a header `message_type`. - -### BLOCK -[Block](https://github.com/ethereum/eth2.0-specs/blob/v0.5.0/specs/core/0_beacon-chain.md#beaconblock) - from v0.5.1 of the BeaconChain spec -### ATTESTATION -[Attestation](https://github.com/ethereum/eth2.0-specs/blob/v0.5.0/specs/core/0_beacon-chain.md#attestation) - from v0.5.1 of the BeaconChain spec - -## unix_timestamp -seconds since Jan 01 1970. (UTC) diff --git a/specs/nodes.graphviz b/specs/nodes.graphviz new file mode 100644 index 0000000..c1bd40d --- /dev/null +++ b/specs/nodes.graphviz @@ -0,0 +1,11 @@ +digraph { + 1 [label="Node 1"] + 2 [label="Node 2"] + 3 [label="Node 3"] + 1 -> 3 + 1 -> 2 + 2 -> 1 + 2 -> 3 + 3 -> 1 + 3 -> 2 +} diff --git a/specs/nodes.png b/specs/nodes.png new file mode 100644 index 0000000..32ff583 Binary files /dev/null and b/specs/nodes.png differ diff --git a/specs/ping.md b/specs/ping.md deleted file mode 100644 index 111fcb7..0000000 --- a/specs/ping.md +++ /dev/null @@ -1,17 +0,0 @@ -# Ping/pong protocol - -The ping/pong protocol is used to test connections between 2 peers and ensure the Hobbits implementation is passing conformance tests. - -When a ping message is received, the hobbits implementer should respond with the body of the ping message as a pong message. - -## Ping - -Headers: `ping` as UTF-8 bytes - -Body: `random 32 bytes` - -## Pong - -Headers: `pong` as UTF-8 bytes - -Body: `32 bytes sent by the ping packet` diff --git a/specs/protocol.md b/specs/protocol.md deleted file mode 100644 index 5812e67..0000000 --- a/specs/protocol.md +++ /dev/null @@ -1,66 +0,0 @@ -# EWP (Ethereum Wire Protocol) - -## Messages - -The message type both defines the `request` and `response` as they are identical. The `message` format is the following: - -``` -EWP -
-``` -Messages are bson encoded and use snappy compression - -A parsed message would look like this: - -```python -{ - 'version': 'string', - 'protocol': 'string' - 'header': 'bytes', - 'body': 'bytes' -} -``` - -### Fields - -| Field | Definition | Validity | -|:------:|----------|:----:| -| `version` | Defines the EWP version number e.g. `0.2`. | `(\d+\.)(\d+)` | -| `protocol` | Defines the communication protocol. | `(RPC\|GOSSIP)` | -| `header` | Defines the header | payload | -| `body` | Defines the body | payload | - -### Examples - -example of a wire protocol message - -#### RPC call example with ping -``` -# Request (RPC call with a body of a RPC ping call) -EWP 0.2 RPC 0 25 -{"id":1,"method_id":0x00} - -# Response -EWP 0.2 RPC 0 25 -{"id":1,"method_id":0x01} -``` - -#### RPC call with payload -``` -# Request -EWP 0.2 RPC 0 1234 -<1234 bytes binary body data> -# Response -EWP 0.2 RPC 321 1234 -<321 bytes of binary header data> -<1234 bytes of binary body data> -``` - -#### Gossip -``` -# Request (Gossip call with a full block) -EWP 0.2 GOSSIP 25 1234 -<25 bytes of binary header data> -<1234 bytes of binary body data> -``` - diff --git a/specs/rpc-messages.md b/specs/rpc-messages.md deleted file mode 100644 index feb6778..0000000 --- a/specs/rpc-messages.md +++ /dev/null @@ -1,150 +0,0 @@ -# Messages - -These messages are used to define an application protocol for Ethereum 2.0. -These messages define a RPC protocol for clients to interact with each other. - -# Envelope - -This application protocol is classified under the `RPC` command. - -All referenced data structures can be found in the [Beacon Chain](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#data-structures) specification and should be serialized as SSZ Containers. - -The body of the RPC calls must conform to: -``` -{ - method_id: uint16 // byte representing the method - id: uint64 // id of the request - body: // body of the request itself -} -``` - -Example: -``` -EWP 0.2 RPC 0 12 -{ - "method_id": 0x01, - "id": 1, - "body": { - "reason": 42 - } -} -``` - -The message may contain additional headers specified by the application layer. - -## `0x00` HELLO - -Nodes can send `HELLO` messages to each other to exchange information on their status. - - -```java -{ - 'network_id': 'uint8' // the ID of the network (1 for mainnet, and some predefined number for a testnet) - 'chain_id': 'uint8' // the ID of the chain (1 for ETH) - 'latest_finalized_root': 'bytes32' // the hash of the latest finalized root - 'latest_finalized_epoch': 'uint64' // the number of the latest finalized epoch - 'best_root': 'bytes32' // the hash of the best root this node can offer - 'best_slot': 'uint64' // the number of the best slot this node can offer -} -``` - -## `0x01` GOODBYE - -Nodes may signal to other nodes that they are going away by sending a `GOODBYE` message. -The reason given is optional. Reason codes are up to each client and should not be trusted. - -```java -{ - 'reason': 'uint64' // an optional reason code up to the client -} - -``` - -## `0x02` GET_STATUS - -Nodes may exchange metadata information using a `GET_STATUS` message. - -This information is useful to identify other nodes and clients and report that information to statistics services. - -```java -{ - 'user_agent': 'bytes' // the human readable name of the client, optionally with its version and other metadata - 'timestamp': 'uint64' // the current time of the node in milliseconds since epoch -} -``` - -## `0x0A` GET_BLOCK_HEADERS - -Nodes may request block headers from other nodes using the `GET_BLOCK_HEADERS` message. - -```java -{ - 'start_root': 'bytes32' // the root hash to start querying from OR - 'start_slot': 'uint64' // the slot number to start querying from - 'max': 'uint64' // the max number of elements to return - 'skip': 'uint64' // the number of elements apart to pick from - 'direction': 'uint8' // 0x01 is ascending, 0x00 is descending direction to query elements -} -``` - -The request should contain either a `start_root` or `start_slot` parameter. - -## `0x0B` BLOCK_HEADERS - -Nodes may provide block roots to other nodes using the `BLOCK_HEADERS` message, usually in response to a `GET_BLOCK_HEADERS` message. - -```java -{ - 'headers': '[]BeaconBlockHeader' -} -``` - -## `0x0C` GET_BLOCK_BODIES - -Nodes may request block bodies from other nodes using the `GET_BLOCK_BODIES` message. - -```java -{ - 'start_root': 'bytes32' // the root hash to start querying from OR - 'start_slot': 'uint64' // the slot number to start querying from - 'max': 'uint64' // the max number of elements to return - 'skip': 'uint64' // the number of elements apart to pick from - 'direction': 'uint8' // 0x01 is ascending, 0x00 is descending direction to query elements -} -``` - -The request should contain either a `start_root` or `start_slot` parameter. - -## `0x0D` BLOCK_BODIES - -Nodes may provide block roots to other nodes using the `BLOCK_BODIES` message, usually in response to a `GET_BLOCK_BODIES` message. - -```java -{ - 'bodies': []BeaconBlock -} -``` - -# Lifecycle and message exchanges - -## Initial hello - -Upon discovering each other, nodes may exchange `HELLO` messages. - -Nodes may send `HELLO` to other peers when they exchange messages for the first time or when their state changes to let them know new blocks are available. - -Upon receiving a `HELLO` message, the node may reply with a `HELLO` message. - -## Status messages - -Any peer may provide information about their status and metadata to any other peer. Other peers may respond on a best effort basis, if at all. - -## Block and header messages - -Peers may request blocks and headers from other peers. - -Other peers may respond on a best effort basis with header and block data. - -There is no SLA for responding. Peers may request blocks repeatedly from the same peers. - - diff --git a/specs/spec.md b/specs/spec.md new file mode 100644 index 0000000..4cd5fe1 --- /dev/null +++ b/specs/spec.md @@ -0,0 +1,329 @@ +# @TODO +- [ ] Link to parsers. +- [ ] attestations +- [ ] what else needs to be defined in the transport header? + +# Hobbits + +*Written by Dean Eigenmann & Rene Nayman * + +## Table of Contents + +1. [Abstract](#abstract) +2. [Flow](#flow) +3. [Wire Protocol](#wire-protocol) + 1. [Messages](#messages) +4. [Transport](#transport) +5. [Protocols](#protocols) + 1. [RPC](#rpc) + 2. [Gossip](#gossip) + 3. [PING](#ping) +6. [Implementations](#implementations) + +## Abstract + +Hobbits is a modular wire protocol which allows implementers to experiment with various application logic over a practical network in an agnostic and expedient manner. + +Hobbits wire protocol was developed so that Eth2.0 clients could begin network-level testing without requiring libp2p. + +## Flow + +We layout a network where the nodes are peered in the following manner: + +![nodes](./nodes.png) + +The following flow dictates how blocks are shared amongst peers: + +![sequence diagram](./diagram.png) + +1. Node 1 receives a new block `0xCAB` and gossips `hash(0xCAB)` to Node 2 and Node 3 +2. Node 2 requests the block bodies for block `0xCAB` from Node 1 in an RPC Command query +3. Node 1 responds to Node 2 with the block bodies for block `0xCAB` in an RPC Command response +4. Node 2 receives a new block `0xCAB` and gossips it to Node 1 and Node 3 + +## Wire Protocol + +### Messages + +The `message` format looks as follows: + +``` +EWP +
+``` + +**NOTE:** Message`body`s along with `header` fields are BSON encoded. + +#### Fields + +Every hobbit message contains the following fields: + +| Field | Definition | Validity | +|:------:|----------|:----:| +| `version` | Defines the EWP version number e.g. `0.2`. | `(\d+\.)(\d+)` | +| `protocol` | Defines the [protocol](#protocols). | `(RPC\|GOSSIP\|PING)` | +| `header` | Defines the header | payload | +| `body` | Defines the body | payload | + +## Transport + +A Hobbits node listens for and sends out messages. + +Hobbits packets are sent as follows: + +``` + +``` + +- **Length**: `big endian` encoded `uint32` that represents the length of the packet, with a length of 4 bytes. +- **Packet**: Byte encoded string of the hobbits message. + +## Protocols + +Hobbits defines 3 protocols that dictate how messages are interpreted and responded to. + +### RPC + + + +#### Envelope + +```python +{ + 'method_id': 'uint16' ## byte representing the method + 'id': 'uint64' ## id of the request + 'body': 'bytes' ## body of the request itself +} +``` + +The `body` field contains an [SSZ](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/simple-serialize.md) encoded byte array. + +##### Example + +Below you will find an example `RPC` request. + +``` +EWP 0.2 RPC 0 65 +{ + "method_id": 0x01, + "id": 1, + "body": { + "bodies": "0x2a00000000000000" + } +} +``` +#### RPC Methods + +There are two types of RPC methods: one that requests data and one that responds with data. + +* Peers may request blocks and headers from other peers, possibly in response to a `GOSSIP` message. +* Peers may request blocks repeatedly from the same peers. +* Other peers may respond on a best effort basis with header and block data. +* There is no SLA for responding. + +##### Handshake + +###### `0x00` HELLO + +Upon discovering each other, nodes may exchange `HELLO` messages. + +Nodes may send `HELLO` to other peers when they exchange messages for the first time or when their state changes to let them know new blocks are available. + +Nodes can send `HELLO` messages to each other to exchange information on their status: + +```python +{ + 'network_id': 'uint8' ## the ID of the network (1 for mainnet, and some predefined number for a testnet) + 'chain_id': 'uint8' ## the ID of the chain (1 for ETH) + 'latest_finalized_root': 'bytes32' ## the hash of the latest finalized root + 'latest_finalized_epoch': 'uint64' ## the number of the latest finalized epoch + 'best_root': 'bytes32' ## the hash of the best root this node can offer + 'best_slot': 'uint64' ## the number of the best slot this node can offer +} +``` + +Upon receiving a `HELLO` message, the node should reply with a `HELLO` message. + +###### `0x01` GOODBYE + +Nodes may signal to other nodes that they are going away by sending a `GOODBYE` message: + +```python +{ + 'reason': 'uint64' ## an optional reason code up to the client +} +``` + +The reason given is optional. Reason codes are up to each client and should not be trusted. + +Upon receiving a `GOODBYE` message, no response is necessary. + +##### `0x02` GET_STATUS + +Nodes may exchange metadata information using a `GET_STATUS` message. + +A `GET_STATUS` request may be sent in response to receiving a `GOSSIP` message: + +```python +{ + 'user_agent': 'bytes' ## the human readable name of the client, optionally with its version and other metadata + 'timestamp': 'uint64' ## the current time of the node in milliseconds since epoch +} +``` + +Any peer may provide information about their status and metadata to any other peer. Other peers may respond on a best effort basis, if at all. + +##### Block Headers + +###### `0x0A` GET_BLOCK_HEADERS + +Nodes may request block headers from other nodes using the `GET_BLOCK_HEADERS` message: + + +```python +{ + 'start_root': 'bytes32' ## the root hash to start querying from OR + 'start_slot': 'uint64' ## the slot number to start querying from + 'max': 'uint64' ## the max number of elements to return + 'skip': 'uint64' ## the number of elements apart to pick from + 'direction': 'uint8' ## 0x01 is ascending, 0x00 is descending direction to query elements +} +``` + +A `GET_BLOCK_HEADERS` request may be sent in response to receiving a `GOSSIP` message. + +###### `0x0B` BLOCK_HEADERS + +Nodes may provide block roots to other nodes using the `BLOCK_HEADERS` message, usually in response to a `GET_BLOCK_HEADERS` message: + +```python +{ + 'headers': '[]BeaconBlockHeader' +} +``` + +##### Block Bodies + +###### `0x0C` GET_BLOCK_BODIES + +Nodes may request block bodies from other nodes using the `GET_BLOCK_BODIES` message: + +```python +{ + 'start_root': 'bytes32' ## the root hash to start querying from OR + 'start_slot': 'uint64' ## the slot number to start querying from + 'max': 'uint64' ## the max number of elements to return + 'skip': 'uint64' ## the number of elements apart to pick from + 'direction': 'uint8' ## `0x01` is ascending, `0x00` is descending direction to query elements +} +``` + +###### `0x0D` BLOCK_BODIES + +Nodes may provide block roots to other nodes using the `BLOCK_BODIES` message, usually in response to a `GET_BLOCK_BODIES` message: + +```python +{ + 'bodies': '[]BeaconBlock' +} +``` + +##### Attestations + +###### GET_ATTESTATION + +Nodes may request an attestation from other nodes using the `GET_ATTESTATION` message: + +```python +{ + 'signature' : 'bytes' +} +``` + +###### ATTESTATION + +```python +{ + 'attestation' : 'Attestation' +} +``` + +### Gossip + + + +#### Envelope + +The message must contain the following header: + +```python +{ + 'method_id': 'uint16' ## the method used in this exchange, as described below + 'topic': 'string' ## the type of message being exchanged + 'timestamp': 'uint32' ## timestamp when the message was sent + 'message_hash': 'bytes32' ## a hash uniquely representing the message contents, with a hash function up to the application + 'hash_signature': 'bytes32' ## a signature of the message hash with a public key identifying the node sending data +} +``` + +##### Example +``` +EWP 0.2 GOSSIP 222 0 +{ + "method_id": 3, + "topic": "BLOCK", + "timestamp": 1560471980, + "message_hash": "0x9D686F6262697473206172652074776F20616E6420666F75722066656574", + "hash_signature": "0x0000000009A4672656E63682070656F706C6520617265207468652062657374" +} +``` + +The message may contain additional headers specified by the application layer. + +#### GOSSIP Methods + + + + +##### `0x00` GOSSIP + +Nodes use `GOSSIP` methods to send data to other nodes in the network. + +The body of a `GOSSIP` method consists in the data being gossiped and + +**@TODO: is the body an ssz encoded byte array?** + +The `message_hash` header value must match the hash of the contents of the body according to a predefined hash function defined by the application. + +### Ping + +The ping/pong protocol is used to test connections between two peers and ensure the Hobbits implementation is passing conformance tests. + +When a `PING` message is received, the node must respond with the body of the `PING` message as a `PONG` message. + +## Ping + +``` +EWP 0.2 PING 4 32 +ping +``` + +Headers: `ping` as UTF-8 bytes + +Body: `random 32 bytes` + +## Pong + +``` +EWP 0.2 PING 4 32 +pong +``` + +Headers: `pong` as UTF-8 bytes + +Body: `32 bytes sent by the ping packet` + +## Implementations + +As a reference, the following implementations exist: + - [go-hobbits](https://github.com/renaynay/go-hobbits)