[litep2p-milestone]: Improve litep2p robustness

[lexnv]

The Vision

Improve the robustness and fault-tolerance of the litep2p crate and bring the crate to a higher level of maturity.

The goal is to stabilize the litep2p and use it as the default backend of Substrate-based chains.

The Plan

To achieve this, we'll focus our efforts on:

• Feature Completeness
  • Implement missing features (validate ping payloads, generate unique random ping payloads, ensure kad records TTL)
• Robustness
  • Ensure core code logic does not panic upon unexpected/malicious user behaviors, panics, unwraps, undocumented expects, array indexing without length validation etc
  • Fix incorrect functionality and divergences from the libp2p specification
  • Repair bugs and limit the surface of race conditions
• Maintenance: keep the code simple
  • Avoid code duplicates (transport layer)
  • Coherent APIs and errors
  • Adopt rust clippy and best practices
  • Improve and add documentation
• Performance Improvements
  • Avoid unnecessary clones
  • Improve the query engine
  • Analyze memory and CPU consumption
• Testing
  • increase code coverage by unit-testing sensitive components (multiselect / noise, kad etc)
  • improve our integration testing scenarios
  • add fuzzing
  • create a plan for continuous testing and triaging in our test stack and keep track of triage reports

Feature Completeness

Beta Give feedback

1. Implement connection limits #17
   2 of 2
   easy enhancement +2
   
2. quic: Configure timeout for open / accept susbstreams #131
   +0
3. identify: Open inbound substream for unknown peer #103
   bug +1
4. multistream: Lazy negotiated protocol cannot be closed before receiving messages #60
   bug easy +2
   
5. Clone only needed KademliaPeers when yielding closest nodes #53
   easy enhancement +2
6. Add pending nodes to Kademlia k-buckets #48
   easy enhancement +2
   
7. Implement proper peer eviction #37
   easy enhancement +2
   
8. Research the possibility of embedding Peerset into NotificationProtocol #34
   enhancement medium +2
9. Start using V1Lazy when applicable #1
   enhancement +1
10. kad: Revisit FIND_NODE and increase test coverage #100
    enhancement +1
    
11. kad: Implement query timeouts #168
    enhancement +1
12. kad: Implement ADD_PROVIDER command #166
    enhancement +1
13. kad: Implement GET_PROVIDERS command #167
    enhancement +1
14. kad: Implement FindManyNodesContext #80
    enhancement +1
    
15. kad: Add kademlia metrics #171
    enhancement +1
16. transport: Change AddressStore to keep track of more dial addresses #180
    1 of 1
    enhancement +1
    
17. transport/manager: Add ability to accept/reject incoming connections before negotiation #186
    enhancement +1
    
18. request-response: Provide more details into rejected requests for a high request failure #188
    1 of 1
    enhancement +1
    
19. identify: Update local node listen addresses #191
    1 of 1
    enhancement +1
    
20. errors: Add DialError error and ListDialFailures event for better error reporting #206
    +0
    
21. transport: Implement TransportService::local_peer_id() #224
    +0
    
22. kad: Expose all peer records of GET_VALUE query #96
    +0
    

Robustness

Beta Give feedback

1. ping: Report back ping failures #132
   +0
2. crypto/noise: Make code fault tolerant #124
   +0
   
3. ping: Make ping fault tolerant wrt outbound substreams races #133
   +0
4. ping: Verify received "pong" payload #134
   +0
5. kad: Ensure Kademlia Record has TTL #129
   2 of 2
   +0
   
6. kad: Strategy for removing old MemoryStore records #81
   enhancement +1
7. kad: Implement FindManyNodesContext #80
   enhancement +1
   
8. protocol/notif: Fix panic on missing peer state #143
   bug +1
   
9. transport-manager: Unknown connection opened #172
   2 of 2
   bug +1
   
10. kad: Fix substream opening and dialing race #222
    bug +1
    
11. query-executor: Save the task waker on empty futures #219
    +0
    
12. crypto/noise: Generate keypair only for Curve25519 #214
    enhancement +1
    
13. transport: Fix double lock and state overwrite on disconnected peers #179
    +0
    
14. kad: Do not update memory store on incoming GetRecordSuccess #190
    +0
    
15. transport: Reject secondary connections with different connection IDs #176
    +0
    
16. Fix TCP, WebSocket, QUIC leaking connection IDs in reject() #198
    bug +1
    
17. kad: Update connection status of an existing bucket entry #181
    enhancement +1
    

Maintenance

Beta Give feedback

1. transport: Extract common transport code between TCP and WebSocket #70
   2 of 2
   enhancement +1
   
2. error: Merge NegociatedError with SubstreamError #128
   +0
3. clippy: Revisit allowed clippy rules #85
   +0
4. Possible use of libp2p_identity::PeerId as PeerId type #73
   +0
5. kad: Remove unused serde cfg #262
   +0
   
6. transport_service: Improve logs and move code from tokio::select macro #254
   enhancement +1
   

Testing

Beta Give feedback

1. mutlistream_select/tests: Fuzz Message encode / decode #127
   +0
2. mutlistream_select/tests: Add tests for LengthDelimited #126
   +0
3. crypto/noise/tests: Increase test coverage #125
   +0
4. Increase test coverage for Kademlia #39
   easy +1
5. minor: fix tests without websocket feature #215
   bug easy +2
   
6. kad: Refactor GetRecord query and add tests #97
   +0
   
7. ci: Run CI on master and PRs #87
   +0
   
8. ci: Enable dependabot #89
   +0
   

Performance

Beta Give feedback

1. crypto/noise: Investigate the performance impact of creating a new RNG and DH in snow #123
   +0
2. Improve QueryEngine #40
   easy enhancement +2
3. Analyze memory usage #36
   easy enhancement +2
4. substream: Use write_all instead of manually writing bytes #217
   enhancement +1
   
5. find_node: Optimize parallelism factor for slow to respond peers #220
   +0
   

Outstanding

Beta Give feedback

1. webrtc: Panics for non-stun packets #107
   bug +1
2. multistream-select implementation used by WebRTC doesn't conform to the spec #75
   bug +1
3. Improve error handling in WebRTC-related Noise functions #43
   bug easy enhancement +3
4. Add support for fallback protocols #42
   bug enhancement medium +3
5. Implement new substream type for WebRTC #41
   bug enhancement hard +3
6. Implement memory transport #7
   enhancement medium +2
7. Any plan to support WASM target? #99
   +0