Replies: 2 comments
-
Should it be N2C -> N2C, right? Not N2N -> N2C |
Beta Was this translation helpful? Give feedback.
-
Table of Contents
Reusable DiffusionThis document outlines the findings of an investigation into the Cardano diffusion layer and explores how to make it reusable for third-party users. The goal is to transform the current diffusion layer so as to become a library that enables others to build overlay networks, leveraging the existing self-balancing, self-optimizing, and self-healing capabilities of the current Cardano Node network. Users will be able to diffuse any data across their network by running custom protocols, defining their own targets for connectivity, and setting their own governance policies (such as churning and monitoring jobs). Note that this project is not to meant to be a fork of the current Current StateIn the current architecture, the Cardano node initializes the consensus layer, which in turn initializes the diffusion layer. The node parses configuration files and passes diffusion parameters to the consensus layer via Additionally, the consensus layer is responsible for providing the diffusion layer with the versioned applications (mini-protocols) bundle, which manages peer connections and promotions. This is important for protocols like the handshake protocol and codecs. The consensus layer also supplies the Currently the consensus layer only depends on the The goal of a reusable design is to allow third-party users to leverage the Current Design Overview:flowchart TB
subgraph CN[Cardano Node]
C[Consensus]
subgraph Diffusion["Diffusion (ouroboros-network)"]
direction TB
DN2C[N2C]
DN2N[N2N]
end
C -- "LedgerPeerConsensusInterface" --> Diffusion
end
Target Design:flowchart TB
subgraph CN["Cardano Node (Haskell)"]
C[Consensus]
subgraph CNDiff["Diffusion (o-network)"]
CNN2C[N2C]
CNN2N[N2N]
end
C -- "LedgerPeerConsensusInterface" --> CNDiff
end
subgraph CA["Custom App (Haskell)"]
subgraph CADiff["Diffusion (o-network)"]
CAN2C[N2C]
CAN2N[N2N]
end
CNN2C -- "LedgerPeerConsensusInterface RPC" --> CAN2C
end
subgraph CAE["Custom App Logic (X Lang)"]
CAE2C[N2C]
end
CAN2C <--> CAE2C
The main objective is to abstract and extend the Current StakeholdersThis section describes and analyses the current stakeholders (i.e. third party users) that will benefit from using the Mithril NodeMithril is a protocol based on a Stake-based Threshold Multi-signature scheme, which leverages Cardano's Stake Pool Operators (SPOs) to certify data from the Cardano blockchain in a trustless manner. Currently, Mithril is used within the Cardano ecosystem to enable fast bootstrapping of full nodes and secure light wallets. The Mithril protocol coordinates the collection of individual signatures from the signers (run by SPOs), which are then aggregated into multi-signatures and certificates by the aggregators. To achieve full decentralization, Mithril must operate over a decentralized peer-to-peer network. Building such a network from scratch would require substantial time, effort, and investment. Moreover, since SPOs, representing Cardano's active stake, will need to adopt and operate Mithril nodes alongside their Cardano nodes, a more efficient solution is to leverage the existing Cardano network layer. This approach will simplify Mithril's development while minimizing its impact on the Cardano network and reducing the maintenance overhead for SPOs. Mithril will be an early adopter of the proposed design in this document, serving as a use case and illustrative example. Mithril Processes:
architecture-beta
group MithrilNode[Mithril Node]
group OuroborosNetworkMithril[Mithril Ouroboros Network] in MithrilNode
service NodeToClientMithril[Mithril NodeToClient] in OuroborosNetworkMithril
service NodeToNodeMithril[Mithril NodeToNode] in OuroborosNetworkMithril
service NodeToClientCardanoClient[NodeToClient Cardano Client] in OuroborosNetworkMithril
group CardanoNode[Cardano Node]
group OuroborosNetworkCardano[Cardano Ouroboros Network] in CardanoNode
service NodeToClientCardano[Cardano NodeToClient] in OuroborosNetworkCardano
service NodeToNodeCardano[Cardano NodeToNode] in OuroborosNetworkCardano
group MithrilSigner[Mithril Signer]
service NodeToClientMithrilSigner[Mithril NodeToClient] in MithrilSigner
NodeToClientCardanoClient:L -- R:NodeToClientCardano
NodeToClientMithrilSigner:B -- T:NodeToClientMithril
The Mithril node must operate alongside the Cardano node, communicating through UNIX sockets and a custom Node-to-Client (N2C) RPC protocol. This enables the Mithril diffusion layer to access the necessary ledger information to establish a resilient overlay network. Additionally, the Mithril node's diffusion layer will need to support custom protocols to facilitate communication with both other Mithril nodes and the Mithril signer nodes, which provide signatures for diffusion. These signer nodes, or any application-specific logic processes, can be implemented in any suitable programming language, as long as both the Mithril node and the signer node communicate using the same protocol. OthersOther protocols in the Cardano ecosystem, such as Leios and Peras (and probably other protocols in the future), also need the capability to diffuse messages originating from block producers in a decentralized fashion. However, in the Leios and Peras cases, the Cardano node itself is a producer and consumer of these messages. We have taken into consideration this need for a generic solution in the design. Configurable ParametersCurrently, the following diffusion-specific parameters can be configured manually, via configuration files, or programmatically (these belong to the
NOTE: Once we remove non-p2p, we can merge 1. Tracers: Includes tracers for (Local) Mux, (Local) Handshake, and the diffusion tracer. These tracers are likely to remain unchanged until the Non-P2P stack is fully removed. 2. Tracers Extra: Includes tracers for P2P components such as (Local) Public Root Peers, Ledger Peers, Peer Selection, (Churn) Peer Selection Counters, (Local) Connection Manager, (Local) Server, and (Local) Inbound Governor. Third-party users will need to implement their own tracers to monitor the diagnostics of their diffusion layer. 3. Arguments: Includes settings for IPv4/IPv6 addresses, rate limits, and whether the diffusion layer should be initiated in either initiator-only or initiator-responder mode. These configurations are required from third-party users. 4. Arguments Extra: Covers P2P-specific configurations, including Peer Targets, Consensus Mode, Minimum Big Ledger Peers for Trusted State, Peer Sharing, Protocol Idle Timeouts, TIME_WAIT timeouts, Churn Deadline Interval, and Bulk Churn Interval. It also requires STM actions to handle data such as (Local) Public Root Peers, Bootstrap Peers flags, Ledger Peer Snapshots, and whether to use Ledger Peers. Some parameters here are specific to the Cardano node and may not be relevant to third-party applications, so they should be abstracted from the user. 5. Applications: This includes versioned mini-protocol bundles to run based on the connection mode: Initiator Mode, Initiator-Responder Mode, or Local Responder (Non-P2P). These bundles are polymorphic on the N2N/N2C versions and respective data, enabling third-party users to define their own configurations. However, the user must provide the 6. Applications Extra: This includes additional configuration for application behaviors such as the (Local) Rethrow Policy, Return Policy, Peer Metrics TVar for Peer Selection, Block Fetch Mode, and Peer Sharing Registry. As with other parameters, not all of these settings may be relevant for third-party users. The following better illustrates the dependencies between components: flowchart TB
subgraph CN[Cardano Node]
subgraph Diffusion Arguments
Tr[Tracers]
TrE[Tracers Extra]
Args[Arguments]
ArgsE[Arguments Extra]
subgraph Apps[Applications]
LCI[LedgerConsensusInterface]
end
AppsE[Applications Extra]
end
subgraph ON[Ouroboros Network]
OG[Outbound Governor]
CG[Churn Governor]
IG[Inbound Governor]
Etc[Others: Connection Manager, Local Root Peers, Public Root Peers, etc..]
end
OC[Ouroboros Consensus]
end
OC --> LCI
Tr --> Etc
TrE --> OG
TrE --> CG
TrE --> IG
TrE --> Etc
Args --> Etc
Args --> IG
ArgsE --> OG
ArgsE --> CG
ArgsE --> IG
ArgsE --> Etc
Apps --> Etc
AppsE --> OG
AppsE --> CG
AppsE --> Etc
Note that all parameters, except for the Proposed Configuration StructureTo improve the current configuration scheme and provide a cleaner API for third-party users, a new diffusion configuration structure is proposed. The goal is to abstract and hide irrelevant components specific to the Cardano node application (e.g., Block Fetch Mode, Ledger Snapshot, Bootstrap Peers) while allowing users to configure the essential parts of the diffusion layer.
Main P2P ComponentsOutbound Governor (Peer Selection Governor)The peer selection governor manages the discovery and selection of upstream peers. It operates based on a set of targets ( Currently, the Outbound Governor is coupled with Cardano node-specific parameters like data LocalRootPeers extraFlags peeraddr =
LocalRootPeers
-- | Here extraFlags allow 3rd party users to enhance their local peers
(Map peeraddr (PeerAdvertise, PeerTrustable, extraFlags))
[(HotValency, WarmValency, Set peeraddr)]
data PublicRootPeers peeraddr =
PublicRootPeers {
-- | Configuration aka Public Config Peers should not be needed anymore
-- getPublicConfigPeers :: !(Map peeraddr PeerAdvertise)
getLedgerPeers :: !(Set peeraddr)
, getBootstrapPeers :: !(Set peeraddr)
, getBigLedgerPeers :: !(Set peeraddr)
}
-- This moves readUseLedgerPeers from PeerSelectionInterfaces to here
data CardanoPeerSelectionActions m =
CardanoPeerSelectionActions {
cpsaReadLedgerPeerSnapshot :: STM m (Maybe LedgerPeerSnapshot)
, cpsaPeerTargets :: ConsensusModePeerTargets
}
type Config extraLocalRootPeersFlags peeraddr =
[(HotValency, WarmValency, Map peeraddr ( PeerAdvertise, extraLocalRootPeersFlags))]
data PeerSelectionActions extraActions consensusAPI extraLocalRootPeersFlags
peeraddr peerconn m =
PeerSelectionActions {
, readPeerSelectionTargets :: STM m PeerSelectionTargets
, readLocalRootPeers :: STM m (LocalRootPeers.Config extraLocalRootPeersFlags
peeraddr)
, readInboundPeers :: m (Map peeraddr PeerSharing)
, peerSharing :: PeerSharing
, peerConnToPeerSharing :: peerconn -> PeerSharing
, requestPublicRootPeers :: LedgerPeersKind
-> Int
-> m ( PublicRootPeers peeraddr , DiffTime)
, requestPeerShare :: PeerSharingAmount -> peeraddr -> m (PeerSharingResult peeraddr)
, peerStateActions :: PeerStateActions peeraddr peerconn m
, getLedgerStateCtx :: LedgerPeersConsensusInterface consensusAPI m
, readUseBootstrapPeers :: STM m UseBootstrapPeers
, readUseLedgerPeers :: STM m UseLedgerPeers
, getExtraActions :: extraActions
}
-- readUseLedgerPeers was moved to CardanoPeerSelectionActions
data PeerSelectionInterfaces peeraddr peerconn m =
PeerSelectionInterfaces {
countersVar :: StrictTVar m PeerSelectionCounters,
, publicStateVar :: StrictTVar m (PublicPeerSelectionState peeraddr),
, debugStateVar :: StrictTVar m (PeerSelectionState peeraddr peerconn),
}
data CardanoPeerSelectionState =
CardanoPeerSelectionState {
ledgerStateJudgement :: !LedgerStateJudgement
, consensusMode :: !ConsensusMode
, hasOnlyBootstrapPeers :: !Bool
, ledgerPeerSnapshot :: Maybe LedgerPeerSnapshot
}
data PeerSelectionState extraState extraLocalRootPeersFlags
peeraddr peerconn =
PeerSelectionState {
targets :: !PeerSelectionTargets
, localRootPeers :: !(LocalRootPeers extraLocalRootPeersFlags peeraddr)
, publicRootPeers :: !(PublicRootPeers peeraddr)
, knownPeers :: !(KnownPeers peeraddr)
, establishedPeers :: !(EstablishedPeers peeraddr peerconn)
, activePeers :: !(Set peeraddr)
, publicRootBackoffs :: !Int
, publicRootRetryTime :: !Time
, inProgressPublicRootsReq :: !Bool
, bigLedgerPeerBackoffs :: !Int
, bigLedgerPeerRetryTime :: !Time
, inProgressBigLedgerPeersReq :: !Bool
, inProgressPeerShareReqs :: !Int
, inProgressPromoteCold :: !(Set peeraddr)
, inProgressPromoteWarm :: !(Set peeraddr)
, inProgressDemoteWarm :: !(Set peeraddr)
, inProgressDemoteHot :: !(Set peeraddr)
, inProgressDemoteToCold :: !(Set peeraddr)
, stdGen :: !StdGen
, inboundPeersRetryTime :: !Time
, bootstrapPeersTimeout :: !(Maybe Time)
, bootstrapPeersFlag :: !UseBootstrapPeers
, minBigLedgerPeersForTrustedState :: MinBigLedgerPeersForTrustedState
, extraState :: !extraState
}
data CardanoPeerSelectionArguments =
CardanoPeerSelectionArguments {
cnpsaConsensusMode :: ConsensusMode
}
data PeerSelectionArguments extraArgs extraActions consensusAPI
peeraddr peerconn m =
PeerSelectionArguments {
psaPeerSelectionTracer :: Tracer m (TracePeerSelection peeraddr)
, psaDebugPeerSelectionTracer :: Tracer m (DebugPeerSelection peeraddr)
, psaPeerSelectionCountersTracer :: Tracer m PeerSelectionCounters
, psaFuzzRng :: StdGen
, psaPeerSelectionActions :: PeerSelectionActions extraActions consensusAPI
extraLocalRootPeersFlags
peeraddr peerconn m
, psaPeerSelectionPolicy :: PeerSelectionPolicy peeraddr m
, psaPeerSelectionInterfaces :: PeerSelectionInterfaces peeraddr peerconn m
, psaMinBigLedgerPeersForTrustedState :: MinBigLedgerPeersForTrustedState
, psaExtraArgs :: extraArgs
} Perhaps peerSelectionGovernor
:: ( Alternative (STM m)
, MonadAsync m
, MonadDelay m
, MonadLabelledSTM m
, MonadMask m
, MonadTimer m
, Ord peeraddr
, Show peerconn
, Hashable peeraddr
)
=> PeerSelectionArguments extraArgs extraActions consensusAPI
extraLocalRootPeersFlags peeraddr peerconn m
-> m Void
peerSelectionGovernor
PeerSelectionArguments {
psaPeerSelection
, psaDebugPeerSelectionTracer
, psaPeerSelectionCountersTracer
, psaFuzzRng
, psaPeerSelectionActions
, psaPeerSelectionPolicy
, psaPeerSelectionInterfaces
, psaMinBigLedgerPeersForTrustedState
, psaExtraArgs
} =
JobPool.withJobPool $ \jobPool ->
peerSelectionGovernorLoop
psaPeerSelectionTracer
psaDebugPeerSelectionTracer
psaPeerSelectionCountersTracer
psaPeerSelectionActions
psaPeerSelectionPolicy
psaPeerSelectionInterfaces
psaMinBigLedgerPeersForTrustedState
jobPool
(emptyPeerSelectionState psaFuzzRng extraArgs) Monitoring ActionsThe Peer Selection Governor consists of a series of guarded decisions known as monitoring actions. These actions, either blocking or non-blocking, guide the governor's decision-making process. The order of execution is crucial. Although the current governor has Cardano-specific actions, third-party users may need to customize their monitoring actions. Below is a minimal (without Cardano-specific actions) set of monitoring actions, sorted by execution order:
The minimal set of monitoring actions above is not fully decoupled from Cardano node-specific details. Below is a list of actions that depend on Cardano-specific details:
The importance of Since the general case is not obvious, providing a clear/simple outbound governor API for third party users is not easy. One can just do enough to support the Cardano node case which would be only to provide a way to add extra guarding conditions for type MonitoringAction extraState extraActions consensusAPI
extraLocalRootPeersFlags peeraddr peerconn m =
[ PeerSelectionPolicy peeraddr m
-> PeerSelectionActions extraActions consensusAPI extraLocalRootPeersFlags
peeraddr peerconn m
-> PeerSelectionState extraState extraLocalRootPeersFlags peeraddr peerconn
-> Guarded (STM m) (TimedDecision m peeraddr peerconn)]
guardedDecisions :: PeerSelectionState extraState extraLocalRootPeersFlags
peeraddr peerconn
-> MonitoringActions extraState extraActions consensusAPI
extraLocalRootPeersFlags peeraddr peerconn
m
-> Guarded (STM m) (TimedDecision m peeraddr peerconn)
guardedDecisions st actions =
[ Monitor.jobs jobPool st ]
<> foldMap (\a -> a st) actions This API would grant the user full control, with a caveat: they must ensure not to introduce errors, but it would also make debugging easier since the user controls the governor's code. A minimal API could be provided with the essential monitoring actions as a baseline, allowing for customization while avoiding redundancy. Alternatively, an inversion-of-control approach could be implemented. The minimal set of essential operations would still be defined, but users could extend these by providing their own callbacks without compromising the core actions. Although this would complicate the API, it’s unclear how flexible this approach would be for future needs. To extend monitoring actions, users could insert them either before or after the essential blocking and non-blocking actions: data ExtraGuardedDecisions extraState extraActions consensusAPI
extraLocalRootPeersFlags peeraddr peerconn m =
ExtraGuardedDecisions {
-- | This list of guarded decisions will come before all default possibly
-- blocking -- decisions. The order matters, making the first decisions
-- have priority over the later ones.
--
-- Note that these actions should be blocking.
preBlocking :: [MonitoringAction extraState extraActions
consensusAPI extraLocalRootPeersFlags
peeraddr peerconn m]
-- | This list of guarded decisions will come after all possibly preBlocking
-- and default blocking decisions. The order matters, making the first
-- decisions have priority over the later ones.
--
-- Note that these actions should be blocking.
, postBlocking :: [MonitoringAction extraState extraActions
consensusAPI extraLocalRootPeersFlags
peeraddr peerconn m]
-- | This list of guarded decisions will come before all default non-blocking
-- decisions. The order matters, making the first decisions have priority over
-- the later ones.
--
-- Note that these actions should not be blocking.
, preNonBlocking :: [MonitoringAction extraState extraActions
consensusAPI extraLocalRootPeersFlags
peeraddr peerconn m]
-- | This list of guarded decisions will come before all preNonBlocking and
-- default non-blocking decisions. The order matters, making the first
-- decisions have priority over the later ones.
--
-- Note that these actions should not be blocking.
, postNonBlocking :: [MonitoringAction extraState extraActions
consensusAPI extraLocalRootPeersFlags
peeraddr peerconn m]
} An example of Cardano node monitoring actions could look like this: cardanoNodeMonitoringActions
:: ExtraGuardedDecisions extraState extraActions consensusAPI
extraLocalRootPeersFlags peeraddr peerconn m
cardanoNodeMonitoringActions = ExtraGuardedDecisions {
preBlocking = [ \_ psa pst -> monitorBootstrapPeersFlag psa pst
, \_ psa pst -> monitorLedgerStateJudgement psa pst
, \_ _ pst -> waitForSystemToQuiesce pst
]
, postBlocking = [ \_ psa pst -> ledgerPeerSnapshotChange pst psa
]
, preNonBlocking = []
, postNonBlocking = []
}
guardedDecisions :: Time
-> PeerSelectionState extraState extraLocalRootPeersFlags
peeraddr peerconn
-> Map peeraddr PeerSharing
-> MonitoringAction extraState extraActions consensusAPI
extraLocalRootPeersFlags peeraddr peerconn m
-> Guarded (STM m) (TimedDecision m peeraddr peerconn)
guardedDecisions blockedAt st inboundPeers ExtraGuardedDecisions {...} =
-- Make sure preBlocking set is in the right place
foldMap (\a -> a policy actions st) preBlocking
<> Monitor.connections actions st
<> Monitor.jobs jobPool st
<> Monitor.targetPeers actions st
<> Monitor.localRoots actions st
-- Make sure postBlocking set is in the right place
foldMap (\a -> a policy actions st) postBlocking
-- Make sure preNonBlocking set is in the right place
foldMap (\a -> a policy actions st) preNonBlocking
<> BigLedgerPeers.belowTarget actions blockedAt st
<> BigLedgerPeers.aboveTarget policy st
<> RootPeers.belowTarget actions blockedAt st
<> KnownPeers.belowTarget actions blockedAt
inboundPeers policy st
<> KnownPeers.aboveTarget policy st
<> EstablishedPeers.belowTarget actions policy st
<> EstablishedPeers.aboveTarget actions policy st
<> ActivePeers.belowTarget actions policy st
<> ActivePeers.aboveTarget actions policy st
-- Make sure postNonBlocking set is in the right place
foldMap (\a -> a policy actions st) postNonBlocking The options presented in the previous section are not entirely conclusive in determining the best course of action. As the project evolves and more information is gathered—particularly regarding the stakeholders and their specific requirements greater clarity will emerge. With this additional insight, it will become easier to assess the trade-offs and make more informed decisions about the most suitable approach to follow. Churn GovernorThe Churn Governor is responsible for rotating peers in the diffusion layer. This process is not modular and has Cardano node-specific dependencies like Interestingly enough Here's an abstraction of the Cardano-specific data CardanoPeerChurnArgs m =
CardanoPeerChurnArgs {
cpcaModeVar :: StrictTVar m ChurnMode
, cpcaReadFetchMode :: STM m FetchMode
, cpcarTargets :: ConsensusModePeerTargets
, cpcaReadUseBootstrap :: STM m UseBootstrapPeers
, cpcaConsensusMode :: ConsensusMode
}
data PeerChurnArgs' m extraArgs consensusAPI peeraddr =
PeerChurnArgs' {
pcaPeerSelectionTracer :: Tracer m (TracePeerSelection peeraddr)
, pcaChurnTracer :: Tracer m ChurnCounters
, pcaDeadlineInterval :: DiffTime
, pcaBulkInterval :: DiffTime
, pcaPeerRequestTimeout :: DiffTime
-- | This does not need to be here
-- , pcaMetrics :: PeerMetrics m peeraddr
, pcaRng :: StdGen
, pcaPeerSelectionVar :: StrictTVar m PeerSelectionTargets
, pcaReadCounters :: STM m PeerSelectionCounters
, getLocalRootHotTarget :: STM m HotValency
, peerTarget :: PeerSelectionTargets
, getLedgerStateCtx :: LedgerPeersConsensusInterface consensusAPI m
, pcaExtraArgs :: extraArgs
} By abstracting Cardano-specific parameters, we can provide two implementations: one tightly coupled with -- | Promoted data type.
data ChurnType = BasicChurn | CardanoChurn
data PeerChurnArgs (churnType :: ChurnType) m consensusAPI peeraddr =
BasicChurnArgs :: PeerChurnArgs' m () consensusAPI peeraddr
-> PeerChurnArgs BasicChurn m consensusAPI peeraddr
| CardanoChurnArgs :: PeerChurnArgs' m (CardanoPeerChurnArgs m) consensusAPI peeraddr
-> PeerChurnArgs CardanoChurn m consensusAPI peeraddr
peerChurnGovernor :: forall m peeraddr.
( MonadDelay m
, Alternative (STM m)
, MonadTimer m
, MonadCatch m
)
=> PeerChurnArgs churnType m consensusAPI peeraddr
-> m Void
peerChurnGovernor
CardanoChurnArgs (PeerChurnArgs { ... }) = ...
BasicChurnArgs (PeerChurnArgs { ... }) = ... Inbound Governor, Connection Manager, etc.The other components that get initialized by diffusion layer do not depend in any way on Cardano specific details and can be customized all by providing the diffusion arguments at the top level, so these should stay the same. Preliminary PlanBefore fully decoupling the diffusion layer and making it reusable for third-party users, there are a few critical steps that should be considered:
By addressing these preparatory steps, the final implementation will be more robust, maintainable, and easier for third-party developers to adopt and build upon. |
Beta Was this translation helpful? Give feedback.
-
There are three processes:
mithril-node
cardano-node
mithril-signer
(written in Rust by the mithril team)Beta Was this translation helpful? Give feedback.
All reactions