Documentation #503

lovesh · 2018-01-17T14:03:51Z

No description provided.

Signed-off-by: Lovesh <[email protected]>

ashcherbakov · 2018-01-17T14:08:42Z

docs/catchup.md

@@ -0,0 +1,33 @@
+# Ledger catchup
+
+A node uses a process called `catchup` to sync its ledgers with other nodes. It does this process after starting up 


Please use bullet points to mention all cases where we can start catch-up more explicitly.

ashcherbakov · 2018-01-17T14:09:31Z

docs/catchup.md

+The `LedgerManager` performs catchup of each ledger sequentially, which means unless catchup for one ledger is complete, catchup for other will not start. 
+This is mostly done for simplicity and can be optimised but the pool ledger needs to be caught up first as the pool ledger knows how many nodes are there 
+in the network. Catchup for any ledger involves these steps:
+-   Learn the correct state (how many txns, merkle roots) of the ledger by using `LedgerStatus` messages from other nodes.


Please mention how and when LedgerStatus is sent.

It is mentioned below.

ashcherbakov · 2018-01-17T14:10:20Z

docs/catchup.md

+This is mostly done for simplicity and can be optimised but the pool ledger needs to be caught up first as the pool ledger knows how many nodes are there 
+in the network. Catchup for any ledger involves these steps:
+-   Learn the correct state (how many txns, merkle roots) of the ledger by using `LedgerStatus` messages from other nodes.
+-   Once sufficient consistent `LedgerStatus` messages are received so that the node knows the latest state of the ledger, if it finds it ledger to be 


Please mention the exact number of LedgerStatus messages to be sufficient (see quorums.py)

ashcherbakov · 2018-01-17T14:10:49Z

docs/catchup.md

+Relevant code:
+- LedgerManager: `plenum/common/ledger_manager.py`
+- LedgerInfo: `plenum/common/ledger_info.py`
+- Catchup message types: `plenum/common/messages/node_messages.py`  


We should also mention Quorums (quorums.py) containing all quorum values including catch-up quorums.

ashcherbakov · 2018-01-17T14:12:28Z

docs/catchup.md

+the receiving node B's size is 30 with merkle root `y`, B sends a proof to A that A's ledger with size 20 and root `x` is a subset of B's ledger with size 
+30 and root `y`.
+- After receiving a `ConsistencyProof`, the node verifies it.
+-   Once the node that is catching up has sufficient consistent `ConsistencyProof` messages, it knows how many transactions it needs to catchup and 


Please mention the exact number of ConsistencyProof messages to be sufficient (see quorums.py)

Signed-off-by: Lovesh <[email protected]>

ashcherbakov · 2018-01-18T12:14:55Z

docs/request_handling.md

@@ -0,0 +1,54 @@
+# Request handling.


Please remove the dot at the end

ashcherbakov · 2018-01-18T12:17:14Z

docs/request_handling.md

+#### Request handling
+Below is a description of how a request is processed.
+A node on receiving a client request in  `validateClientMsg`:
+-   The node performs static validation checks (validation that does not require any state, like mandatory fields are present, etc), it calls `RequestHandler`'s `doStaticValidation`


We also have a schema-based validation before static validation in req handlers. This schema-based validation checks that the request is a valid request semantically with expected types and limitations. Probably we should mention it here as well.

ashcherbakov · 2018-01-18T12:17:45Z

docs/request_handling.md

+-   The node performs static validation checks (validation that does not require any state, like mandatory fields are present, etc), it calls `RequestHandler`'s `doStaticValidation`
+-   If the static validation passes, it checks if the signature check is required and does that if needed in `verifySignature`. More on this later.
+-   Checks if it a generic transaction query (`GET_TXN`). If it is then query the ledger for that particular sequence number and return the result. A `REQNACK` might be sent if the query is not correctly constructed. 
+-   Checks if it a specific query (needs a specific `RequestHandler`), if it is then it uses `process_query` that uses the specific `RequestHandler` to respond to the query. A `REQNACK` might be sent if the query is not correctly constructed.


Should be Checks if it is

ashcherbakov · 2018-01-18T12:18:14Z

docs/request_handling.md

+A node on receiving a client request in  `validateClientMsg`:
+-   The node performs static validation checks (validation that does not require any state, like mandatory fields are present, etc), it calls `RequestHandler`'s `doStaticValidation`
+-   If the static validation passes, it checks if the signature check is required and does that if needed in `verifySignature`. More on this later.
+-   Checks if it a generic transaction query (`GET_TXN`). If it is then query the ledger for that particular sequence number and return the result. A `REQNACK` might be sent if the query is not correctly constructed. 


Should be Checks if it is

ashcherbakov · 2018-01-18T12:19:50Z

docs/request_handling.md

+A node has atleast 1 authenticator called `CoreAuthNr` whose `authenticate` method is called over the serialised request data to verify signature.
+
+
+Relevant code: ``


Please use bullet points here

ashcherbakov · 2018-01-18T12:20:31Z

docs/request_handling.md

+Below is a description of how a request is processed.
+A node on receiving a client request in  `validateClientMsg`:
+-   The node performs static validation checks (validation that does not require any state, like mandatory fields are present, etc), it calls `RequestHandler`'s `doStaticValidation`
+-   If the static validation passes, it checks if the signature check is required and does that if needed in `verifySignature`. More on this later.


Please mention that signature is not required for queries.

ashcherbakov · 2018-01-18T12:23:44Z

docs/request_handling.md

+    -   If the `Request` is already in process, then it sends an acknowledgement to the client as a `REQACK`
+    -   If the node has not seen the `Request` before it broadcasts the `Request` to all nodes in a `PROPAGATE`.
+    -   Once a node receives sufficient (`Quorums.propagate`) `PROPAGATE`s for a request, it forwards the request to its replicas.
+    -   A primary replica on receiving a forwarded request does dynamic validation (requiring state, like if violating some unique constraint or doing an unauthorised action) 


Should we mention batching here? That Primary applies dynamic validation for each request in the batch, and that the batch can still be processed if there are invalid requests there. Reject is sent just for invalid requests from the batch,

Batch is an optimisation for consensus, i didn't want to digress, i think a detailed doc explaining our differences in 3PC with RBFT should list batching, merkle roots and checks in 3PC messages, etc

Ok, that's fine. Probably it's sufficient here to just mention this doc so that one can have a look how this all works with batches.

ashcherbakov · 2018-01-18T12:24:57Z

docs/request_handling.md

+
+
+#### Signature verification
+Each node has a `ReqAuthenticator` object which allows to register `ClientAuthNr` objects using `register_authenticator` method. During signature verification, 


Should we mention that we support both single signature and multi-signatures now?

Maybe? Do you?

I think we need to mention this if we have such a section as signature verification.

Signed-off-by: Lovesh <[email protected]>

ashcherbakov · 2018-01-18T12:47:19Z

docs/request_handling.md

+A node has atleast 1 authenticator called `CoreAuthNr` whose `authenticate` method is called over the serialised request data to verify signature.
+
+
+Relevant code: ``


Please remove `` at the end

ashcherbakov · 2018-01-18T12:47:52Z

docs/request_handling.md

+Below is a description of how a request is processed.
+A node on receiving a client request in  `validateClientMsg`: 
+-   The node performs static validation checks (validation that does not require any state, like mandatory fields are present, etc), it uses `ClientMessageValidator` and 
+    `RequestHandler`'s `doStaticValidation` for this


Please put a dot at the end.

Signed-off-by: Lovesh <[email protected]>

Lovesh added 8 commits December 27, 2017 16:28

fix gloassary

4ab19d0

Signed-off-by: Lovesh <[email protected]>

update glossay

d24f08e

Signed-off-by: Lovesh <[email protected]>

documentation for storage

d08dfd6

Signed-off-by: Lovesh <[email protected]>

use bullet points in glossary

a5fedbf

Signed-off-by: Lovesh <[email protected]>

update storage documentation

e5d982f

Signed-off-by: Lovesh <[email protected]>

midcommit

3876012

Signed-off-by: Lovesh <[email protected]>

adding catchup documentation

a3c45e6

Signed-off-by: Lovesh <[email protected]>

reslove merge conflicts

89e194b

Signed-off-by: Lovesh <[email protected]>

ashcherbakov reviewed Jan 17, 2018

View reviewed changes

Lovesh added 2 commits January 18, 2018 15:47

update catchup doc

e27ba42

Signed-off-by: Lovesh <[email protected]>

request handling docs

dfe2413

Signed-off-by: Lovesh <[email protected]>

ashcherbakov reviewed Jan 18, 2018

View reviewed changes

update doc

db0ffba

Signed-off-by: Lovesh <[email protected]>

ashcherbakov reviewed Jan 18, 2018

View reviewed changes

Lovesh added 2 commits January 18, 2018 18:42

remove backticks

328d677

Signed-off-by: Lovesh <[email protected]>

formatting

9cbf440

Signed-off-by: Lovesh <[email protected]>

ashcherbakov approved these changes Jan 18, 2018

View reviewed changes

ashcherbakov merged commit 6ec12c4 into master Jan 18, 2018

lovesh deleted the documentation branch February 13, 2018 21:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Documentation #503

Documentation #503

lovesh commented Jan 17, 2018

ashcherbakov Jan 17, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 17, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 17, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 17, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 17, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

ashcherbakov Jan 18, 2018

lovesh Jan 18, 2018

		@@ -0,0 +1,33 @@
		# Ledger catchup

		A node uses a process called `catchup` to sync its ledgers with other nodes. It does this process after starting up

		A node has atleast 1 authenticator called `CoreAuthNr` whose `authenticate` method is called over the serialised request data to verify signature.


		Relevant code: ``



		#### Signature verification
		Each node has a `ReqAuthenticator` object which allows to register `ClientAuthNr` objects using `register_authenticator` method. During signature verification,

Documentation #503

Documentation #503

Conversation

lovesh commented Jan 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment