[Preview - historical] Frontend-Backend split

[ed255]

This is a historical preview of the Frontend-Backend halo2 split

This PR is not yet ready for merge, but it's ready for reviewing and discussing the approach. See this issue for more details: #239

Note that many names of functions and structs contain v2, Backend and other suffixes. This is temporary.

What has been done in this PR:

• Define a clear interface for the backend. In particular there's the CompiledCircuit struct which is what the frontend should output. This struct contains preprocessing data (fixed columns and copy constraints data) and a simplified ConstraintSystem.
  • The simplified ConstraintSystem has the following data removed: selectors, queries, constants, minimum_degree. This should make it easier for a frontend to output a circuit without caring about proving system details, and at the same time removes abstractions that the backend shouldn't care about.
  • A new Expression type has been added which removes selectors and query indices.
  • There's no grouping of constraints into gates, instead, a simplified Gate type is added which holds a single constraint (polynomial identity). The simplified ConstraintSystem still has a list of gates.
• The key generation builds rebuilds the legacy ConstraintSystem by collecting all queries that appear in the circuit and converts the expressions to the legacy ones which contain the query indices, required for verification evaluation.
• The prover function has been turned into a struct that is called several times to commit each phase
• A witness generation struct has been added to use the legacy frontend with the new prover API, and allows generating the witnesses at each phase manually.
• The verification process is kept the same. Nevertheless the verification key uses legacy structs that contain unnecessary data like selectors; this will be removed in the future.
• Benchmark of the new API: both CPU times and memory footprint is kept the same. See below for details.
  • A new test circuit has been introduced with benchmarking and testing purposes. This circuit tries to do everything that is possible with halo2 (please let me know if you think something is missing). This circuit is also parametrizable in the number of rows and number of columns to allow more flexible benchmarks.
• Add a new error case Error::Other that contains a string used to report fine grained errors. This is not required by the frontend-backend split but I believe improves a lot the ux, and I believe this was needed for a long time (specially to avoid having opaque Synthesis errors during witness generation).

What's missing:

• Currently some of the structs/functions are duplicated for the old/new API; this is to allow benchmarking of both approaches. The old functions/structs will be removed.
• There's debug comments and prints, they will be cleaned up before merging this PR.
• The witness gen + commit per phase can be automated with some helper functions that works on a new Prover / ProofSystem trait.
• A lightweight layer of functions to expose the legacy halo2 functionality to make an easy transition: the goal would be to only change the imported crate to update to the new version (and this crate would contain this lightweight layer following the legacy API and necessary re-exports)
• Crate split: the goal is to have the following crates in the halo2 monorepo:
  • legacy halo2 frontend (the library to write circuits)
  • halo2 backend (the library that implements the proof system)
  • halo2 middleware? (pending name) (the traits and structs used between frontend and backend)
  • halo2 compat? (pending name) (lightweight layer + re-exports to make it easy to update a halo2 dependency to this new version)
• Remove the type Polynomial<Assigned<F>> from the frontend-backend interface, in favor of just Vec<F>.

---

Benchmarks

To reproduce the benchmarks you need to build the tests in release with cargo test --release build and then figure out the path of the generated binary in ../target/release/deps/frontend_backend_split-.... The time program is GNU time, and the max memory used is reported under Maximum resident set size (kbytes).

These results have the following parameters (commit 84954577c98cf2822c24ce44f64d2962f3579f4a)

const K: u32 = 16;
const WIDTH_FACTOR: usize = 4;

Legacy

/bin/time -v ../target/release/deps/frontend_backend_split-3c2559a1fbc64c67 test_mycircuit_full_legacy --nocapture

Keygen: 17.404483048s
Prove: 21.266722309s
Verify: 10.441061ms
test test_mycircuit_full_legacy ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 51.45s

	Command being timed: "../target/release/deps/frontend_backend_split-3c2559a1fbc64c67 test_mycircuit_full_legacy --nocapture"
	User time (seconds): 314.58
	System time (seconds): 2.41
	Percent of CPU this job got: 615%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:51.48
	[...]
	Maximum resident set size (kbytes): 1723364

Fe-Be split

/bin/time -v ../target/release/deps/frontend_backend_split-3c2559a1fbc64c67 test_mycircuit_full_split --nocapture

Keygen: 17.254586855s
Prove: 21.28084802s
Verify: 11.558502ms
test test_mycircuit_full_split ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 50.95s

	Command being timed: "../target/release/deps/frontend_backend_split-3c2559a1fbc64c67 test_mycircuit_full_split --nocapture"
	User time (seconds): 311.39
	System time (seconds): 2.57
	Percent of CPU this job got: 615%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:50.99
	[...]
	Maximum resident set size (kbytes): 1728520

[CPerezz]

First pass.

Missing:

• halo2_proofs/src/plonk/prover.rs
• halo2_proofs/src/plonk/shuffle.rs
• halo2_proofs/src/plonk/verifier.rs
• halo2_proofs/src/poly.rs
• halo2_proofs/src/poly/commitment.rs
• halo2_proofs/tests/frontend_backend_split.rs
• halo2_proofs/tests/plonk_api.rs

So far this is an amazing work! I'm super excited! Happy to have forward discussions over improvements and paths to follow!