Site Reliability Team Weekly Update - 2021-10-15

[frol]

It is the first post dedicated to Sire Reliability Team! Previously, we had their updates included in Public Node Interfaces Team updates, so you may find some references in the previous updates #4975, #4876, ... See more engineering updates posted on gov.near.org

Brief Summary

• RPC-as-a-Service infrastructure prototype is getting shaped

Result

(Acronyms: O - Objective, KR - Key Result)

• O1: Make RPC service reliable for the customers while keeping nodes infrastructure sustainable

  • KR1: Deliver MVP of RPC-as-a-Service with internal services migrated to it

    • setup redis and working on first version of throttling service (@chefsale)
    • Prototype of authorization server for RPC as a service, still left with local database setup via Docker (@mhalambek)
    • Still need to figure out how to deploy this service (@mhalambek)

  • KR2: Developer tools should support RPC-as-a-Service authorization mechanism

    • @miraclx has implemented Basic auth for near-rs-client, now in discussion for integrating that into depending tooling (@chefsale)

• O3: Maintain SLA for incident management

  • KR1: Provide oncall support 24/7 for all major services based on agreed SLA
    • multiple betanet incidents this week caused by the simple sharding rollout, spend a lot of time on hard-forking, reverting, releasing patches, debugging (@chefsale)