SVM Traces (Light Clients & ZK)

[buffalojoec]

SVM Traces (Light Clients & ZK)

This PR prototypes adding the "traces" feature to SVM.

Traces are bits of transaction processing data that can be used to craft data
structures - such as Merkle trees or Merkle-Patricia Tries - to store
information about the transactions contained in a block.

If a node was to track this information, it could produce proofs for a wide
range of transaction data, including:

• Inclusion within a block (SPV).
• Transaction result (receipt).
• Account state changes (STF).

Ideally, the roots of such data structures would be stored somewhere on-chain,
like an account or a block header. This way, a completely minified light client
could request proofs from one or more full nodes and verify those proofs against
the roots stored on the chain.

About This Demo

This demo is a prototype implementation for SVM but also comes complete with:

• Benchmarking suite for profiling performance costs of enabling SVM traces.
• Unit test suites to demonstrate working with the primitives.
• Full light client example.

Disclaimer: I did not attempt to optimize or use more optimized existing data
structures for trees and/or hashing. The benchmarks can definitely be improved.

SVM Trace Hooks

I've implemented traces as callback functions within SVM's
TransactionProcessingCallback. They are intentionally redundant (all of them
take a reference to the transaction, for example), to ensure maximum downstream
composability. It's also just a prototype for now.

The idea behind including them as callback functions is the timing by which they
are invoked. Projects could simply consume the return type of the SVM's batch
processor post-processing, but the timing of capturing this trace may be
important. Futhermore, pre-account state would be difficult to capture without
some form of caching.

/// Hook for digesting a processed transaction during batch processing.
/// Only called when a transaction is processed. Transactions that fail
/// validation are not passed to digest.
///
/// Designed for transaction inclusion proof generation (light clients).
fn digest_processed_transaction(&self, _transaction: &impl SVMTransaction) {}

/// Hook for digesting a processed transaction receipt during batch
/// processing.
/// ...
fn digest_processed_receipt(
    &self,
    _transaction: &impl SVMTransaction,
    _receipt: &SVMTransactionReceipt,
) {
}

/// Hook for digesting a processed transactions STF trace during batch
/// processing.
/// ...
fn digest_processed_stf_trace(&self, _trace: &STFTrace<impl SVMTransaction>) {}

The Trie Structure

As mentioned, I did not go through the trouble of implementing a hyper-optimized
data structure... yet. I've included a veil struct that could eventually be used
to serve this optimization - perhaps through a Merkle-Patricia Trie or something
custom - when the time comes.

For now, imagine it's a really fast dynamic Merkle tree structure.

Benchmarks

The benchmark suite executes 0, 10, 1,000, and 100,000 transactions against an
SVM batch processor instance with each trace enabled as well as the control (no
traces).

Transactions	Control	With Transaction Hashing	With Receipt Hashing	With STF Trace Hashing
0	599.91 ns	586.73 ns	594.99 ns	598.24 ns
10	1.4980 µs	1.2528 µs	1.2775 µs	1.2770 µs
1,000	88.883 µs	67.763 µs	69.120 µs	69.905 µs
100,000	11.111 ms	9.9574 ms	9.9986 ms	9.8757 ms

See more at svm/benches/trace.rs.

Light Client Example

The light client example at svm/examples/light-client has more information in
its lib.rs file, however below is an overview.

On any given SVM-based chain, if full nodes enable SVM traces, they can store
these traces in Merkle trees. Then they can serve proofs to light clients.

The example demonstrates a layer 2 SVM-based blockchain - Blitz - which does
impose this Merkle tree storage requirement on its full nodes. As a result, they
can serve proofs for transaction inclusion, receipts, and STF traces.

Everything within the blitz module is what the example suggest a full node
would run, while everything within the light_client module is what a light
client would run. The interaction between the two is simply the light client
requesting (in practice over some network/RPC request) proofs to validate
transaction data.

Following a similar pattern in production, an SVM-based layer 2 or rollup could
enable light clients quite simply by enabling traces within SVM.

SVM traces also open the door to more developing more advanced tracing that
could enable ZK proofs to be generated by the SVM API.

Possible Roadmap for Solana L1

If we wanted to pursue development on SVM traces further for the mainnet Solana
protocol, we could fine-tune the implementation and then consider gating the SVM
feature behind a validator startup flag. We could bootstrap separate data stores
for tree structures that are not included in consensus. However, this would
leave the main piece out: the roots.

In order to have a proper light client on Solana L1, we need the tree roots (for
these data stores each node would run separate from its ledger) to exist
on-chain, in order to be validated by consensus. I propose as one idea to do so
with accounts.

We could introduce a new type of account - a Trace account - that can not be
write-locked and is updated by the runtime. However, all nodes would then have
to run the feature.

If this example is interesting to you, it's worth exploring how we could bring
this to mainnet-beta...