feat(prover): WG refactoring #3 #2942
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Artemka374
changed the title
EmilLuta
approved these changes
Sep 25, 2024
prover/crates/bin/witness_generator/src/rounds/basic_circuits/artifacts.rs
Show resolved
Hide resolved
juan518munoz
added a commit
to lambdaclass/zksync-era
that referenced
this pull request
Oct 8, 2024
* feat(ci): Deprecate use of zk tool in docker-build process (#2901) ## What ❔ Remove use of ZK tool in docker-build process ## Why ❔ Improve readability of CI, speeds up building process ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * test: Investigate L1 "nonce too low" errors (#2848) ## What ❔ - Resolves "nonce too low" errors on L1 by retrying transaction requests. This is a temporary measure until a new version of `reth` is released. - Reduces spammy logs for integration tests. Enables verbose output for integration tests in CI. - Fixes some other issues with tests (e.g., a data race in the snapshot recovery test). ## Why ❔ Makes CI more stable. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): vm perf to prometheus (#2909) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * feat(ci): Add building base docker image (#2913) ## What ❔ Workflow to build base-image ## Why ❔ Added ability to push to ghcr.io ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): Fix multiarch for build-base CI (#2914) ## What ❔ Fix multiarch command for build-base-image CI ## Why ❔ ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): Switch build-base to ghcr.io (#2912) ## What ❔ Change build-base image location from dockerhub to ghcr.io ## Why ❔ Workaround for dockerhub rate-limiting ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat: make `to` address optional for transaction data (#2852) ## What ❔ This PR is the first part of [this one](https://github.com/matter-labs/zksync-era/pull/2631). We decided to separate it into smaller PRs to make it easier to read and review. This PR contains only the changes to the `TransactionData` structs for the different VM versions, modifying them for an optional field. The other parts are currently available in our repo fork, as they are based on this branch. The other PRs are: - [feat: evm simulator as base system contract](https://github.com/lambdaclass/zksync-era/pull/256): integrates the EVM interpreter contract as a base system contract and includes all necessary updates for zksync-era to run with it. - [feat: make evm simulator optional for the server](https://github.com/lambdaclass/zksync-era/pull/257): makes the evm simulator optional to run using a config file. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(prover): WitnessGenerator refactoring #2 (#2899) ## What ❔ Introduce WitnessGenerator trait Rename some methods ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(tee): use hex serialization for RPC responses (#2887) ## What ❔ Following Anton's suggestion, we have switched to hex serialization for API/RPC requests and responses. Previously, we used default JSON serialization for Vec<u8>, which resulted in a lengthy comma-separated list of integers. This change standardizes serialization, making it more efficient and reducing the size of the responses. The previous format, with a series of comma-separated integers for pubkey-like fields, looked odd. Then: ``` curl -X POST\ -H "Content-Type: application/json" \ --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [491882, "sgx"] }' \ https://mainnet.era.zksync.io {"jsonrpc":"2.0","result":[{"attestation":[3,0,2,0,0,0,0,0,10,<dozens of comma-separated integers here> ``` Now: ``` $ curl -X POST \ -H "Content-Type: application/json" \ --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [1, "sgx"] }' \ http://localhost:3050 {"jsonrpc":"2.0","result":[{"l1BatchNumber":1,"teeType":"sgx","pubkey":"0506070809","signature":"0001020304","proof":"0a0b0c0d0e","provedAt":"2024-09-16T11:53:38.253033Z","attestation":"0403020100"}],"id":1} ``` This change needs to be deployed in lockstep with: https://github.com/matter-labs/teepot/pull/209. ## Why ❔ To improve user experience (UX) and optimize the RPC response size. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. Co-authored-by: Artem Makhortov <[email protected]> * feat(ci): Add runtime-base image (#2919) ## What ❔ Adding image which will be used as runtime-base for services ## Why ❔ As a workaround for dockerhub ratelimits ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): Rename workflows for runner and builder base (#2921) ## What ❔ Rename workflows for runner and builder base ## Why ❔ Easier to find =) ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): zksync-runtime-base proper platforms names (#2923) ## What ❔ Proper platform names for docker build action ## Why ❔ Docker documentation (which is linked in docker-build-action docs) [points to non-working reference](https://docs.docker.com/reference/cli/docker/buildx/build/#platform) ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(ci): Proper type of var for platforms in docker build action (#2924) ## What ❔ Proper type of var for platforms in docker build action ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk_toolbox): show readable log (#2911) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(ci): Migrate to vendored image as a zksync-runtime-base (#2926) ## What ❔ Move building images to zksync-runtime-base ## Why ❔ Workaround for dockerhub rate limits ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(main): release core 24.25.0 (#2816) :robot: I have created a release *beep* *boop* --- ## [24.25.0](https://github.com/matter-labs/zksync-era/compare/core-v24.24.0...core-v24.25.0) (2024-09-19) ### Features * (DB migration) Rename recursion_scheduler_level_vk_hash to snark_wrapper_vk_hash ([#2809](https://github.com/matter-labs/zksync-era/issues/2809)) ([64f9551](https://github.com/matter-labs/zksync-era/commit/64f95514c99f95da2a19a97ff064c29a97efc22f)) * add da clients ([#2743](https://github.com/matter-labs/zksync-era/issues/2743)) ([9218612](https://github.com/matter-labs/zksync-era/commit/9218612fdb2b63c20841e2e2e5a45bbd23c01fbc)) * attester committees data extractor (BFT-434) ([#2684](https://github.com/matter-labs/zksync-era/issues/2684)) ([92dde03](https://github.com/matter-labs/zksync-era/commit/92dde039ee8a0bc08e2019b7fa6f243a34d9816f)) * emit errors in prover API metrics ([#2890](https://github.com/matter-labs/zksync-era/issues/2890)) ([2ac7cc5](https://github.com/matter-labs/zksync-era/commit/2ac7cc5836e69fc82c98df2005fedee01c1084e1)) * **en:** Resume incomplete snapshot in snapshot creator in more cases ([#2886](https://github.com/matter-labs/zksync-era/issues/2886)) ([f095b4a](https://github.com/matter-labs/zksync-era/commit/f095b4a3223222ac712de53592fe1e68f766600f)) * make `to` address optional for transaction data ([#2852](https://github.com/matter-labs/zksync-era/issues/2852)) ([8363c1d](https://github.com/matter-labs/zksync-era/commit/8363c1d8697ad9bd2fe5d326218476bc3dad38af)) * **prover:** Optimize setup keys loading ([#2847](https://github.com/matter-labs/zksync-era/issues/2847)) ([19887ef](https://github.com/matter-labs/zksync-era/commit/19887ef21a8bbd26977353f8ee277b711850dfd2)) * Selector generator tool ([#2844](https://github.com/matter-labs/zksync-era/issues/2844)) ([b359b08](https://github.com/matter-labs/zksync-era/commit/b359b085895da6582f1d28722107bc5b25f1232c)) * **tee:** use hex serialization for RPC responses ([#2887](https://github.com/matter-labs/zksync-era/issues/2887)) ([abe0440](https://github.com/matter-labs/zksync-era/commit/abe0440811ae4daf4a0f307922a282e9664308e0)) * **utils:** Rework locate_workspace, introduce Workspace type ([#2830](https://github.com/matter-labs/zksync-era/issues/2830)) ([d256092](https://github.com/matter-labs/zksync-era/commit/d2560928cc67b40a97a5497ac8542915bf6f91a9)) * **zk_toolbox:** Add external_node consensus support ([#2821](https://github.com/matter-labs/zksync-era/issues/2821)) ([4a10d7d](https://github.com/matter-labs/zksync-era/commit/4a10d7d9554d6c1aa2f4fc46557d40baaad8ff2f)) ### Bug Fixes * count SECP256 precompile to account validation gas limit as well ([#2859](https://github.com/matter-labs/zksync-era/issues/2859)) ([fee0c2a](https://github.com/matter-labs/zksync-era/commit/fee0c2ad08a5ab4a04252765b367eb9fbb1f3db7)) * **en:** Fix connection starvation during snapshot recovery ([#2836](https://github.com/matter-labs/zksync-era/issues/2836)) ([52f4f76](https://github.com/matter-labs/zksync-era/commit/52f4f763674d25f8a5e7f3a111354a559f798d52)) * **eth_watch:** fix `get_events_inner` ([#2882](https://github.com/matter-labs/zksync-era/issues/2882)) ([c957dd8](https://github.com/matter-labs/zksync-era/commit/c957dd8011213e0e95fa5962e2310321b29a0d16)) * handling of HTTP 403 thrown by proxyd ([#2835](https://github.com/matter-labs/zksync-era/issues/2835)) ([2d71c74](https://github.com/matter-labs/zksync-era/commit/2d71c7408a0eed3662fc51f70fa9f525d66e4c6f)) * **state-keeper:** Restore processed tx metrics in state keeper ([#2815](https://github.com/matter-labs/zksync-era/issues/2815)) ([4d8862b](https://github.com/matter-labs/zksync-era/commit/4d8862b76a55ac78edd481694fefd2107736ffd9)) * **tee-prover:** fix deserialization of `std::time::Duration` in `envy` config ([#2817](https://github.com/matter-labs/zksync-era/issues/2817)) ([df8641a](https://github.com/matter-labs/zksync-era/commit/df8641a912a8d480ceecff58b0bfaef05e04f0c8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]> * docs: Update system requirements (#2894) ## What ❔ - 30 GB of free disk space are no longer sufficient to run a testnet EN. Upped the system requirements to 50 GB. - State growth is normally used to refer to the growth of the state tree, not of the historical state. What we have is historical state pruning, not state pruning. * chore(vm): Bump vm2 repo revision (#2883) ## What ❔ Bumps the `vm2` crate revision and updates new VM usage correspondingly. ## Why ❔ To keep the `vm2` dependency up to date and track possible regressions. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(nix): use precompiled `rocksdb` and `snappy` (#2925) ## What ❔ Use precompiled `rocksdb` and `snappy` in the nix build. ## Why ❔ Speeds up compilation and is usually better maintained. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Harald Hoyer <[email protected]> * feat(prover): Add endpoint to PJM to get queue reports (#2918) ## What ❔ Add `/queue_report` endpoint, which will get the data about queue and send it. ## Why ❔ To work with new autoscaler ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: EmilLuta <[email protected]> * fix(eth-sender): print better error message in case of missing blob prices (#2927) Signed-off-by: tomg10 <[email protected]> Co-authored-by: perekopskiy <[email protected]> * feat: added seed_peers to consensus global config (#2920) It will allow us to announce the recommended default list of peers to all ENs without manual intervention. Fixes BFT-509. --------- Co-authored-by: Bruno França <[email protected]> * feat(vm): Do not panic on VM divergence (#2705) ## What ❔ - Allows to continue batch execution on divergence via new `ShadowLenient` VM mode. - Dumps VM state to logs and optionally a file on divergence. ## Why ❔ Allows to detect divergencies in multiple batches w/o blockers. The dumped VM state will hopefully allow investigating divergencies locally, although this logic isn't implemented yet. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(prover): Use query macro instead string literals for queries (#2930) ## What ❔ In some places our sqlx queries are using string literals instead of query macros. This PR changes this behaviour in places it is possible. ## Why ❔ To prevent possible SQL injections. It also will cache the queries, which should make them faster. ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(circuit_prover): Add circuit prover (#2908) ### Summary This PR introduces a new component `circuit_prover`, which is multiple WVGs & GPU prover running together, groupless. The changes are non-destructive, old setup and new setup must work together in-tandem. ### What? Circuit prover is a component that runs X WVGs alongside a GPU prover. Making full use of CPUs on the GPU machine, WVGs as a component can be removed altogether. Prover groups are not needed anymore. Based on empiric testing we can (almost -- there will be follow-up fixes to make it efficient) fully run everything on a single machine. The current implementation can sunset the old setup. Current metrics show that circuit prover is > 60% efficient than old one (but quirks are needed for node proofs to unlock it -- will be treated as a follow-up). The purpose is to have the `circuit_prover` deprecate old `prover_fri` & `witness_vector_generator`. ### Why? The changes will allow us to reduce our infrastructure footprint by ~2x and fix plenty of issues we had in the past. Namely: - fully decoupled of GCP - better resource utilization & reduce costs - reduce overall infrastructure needs (which solves the GPU unavailability we've been facing) - reduce complexity & other inefficiencies (no more prover groups!) - and more ### Ask We want to unblock folks running on AWS. This PR is done as is to speed up release process on DevOps side, as it's the longest pole. NOTE: This is the first PR out of a longer set of PRs. Comments are more than welcome, but the following concerns will be addressed in follow-up PRs and are out of scope for this PR: - tracing implementation is subpar; in fact, I'm confident that most metrics could be done via traces - there's a lot of code duplication (both old/new prover, but runner interface between new WVG & CP) - tests - concern separation between job scheduling & job execution - job priority based on resource consumption - other nits (such as no README, constants being hard-coded instead of configuration, etc.) ### Reviewer suggestion This is basically a merge between `prover_fri`, `witness_vector_generation` and `JobProcessor`. Checking code alongside should give you a better view of what's going on. Sorry for making this hard. :/ * refactor(vm): Refactor L1 batch params provider (#2891) ## What ❔ Refactors `L1BatchParamsProvider`, in particular its construction. ## Why ❔ To have more intuitive DevEx. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(vm): Extract oneshot VM executor – environment types (#2885) ## What ❔ Continues oneshot VM executor extraction by moving generic environment types to the `zksync_vm_executor` crate. ## Why ❔ Makes it possible to use the VM executor outside the API server. E.g., it's now used in `zksync_node_consensus`. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(api_server): fix blob_gas length (#2673) Discovered during testing Gateway integration (our own asserts from query.rs started being triggered by our eth api implementation :D ) --------- Co-authored-by: perekopskiy <[email protected]> * refactor: Improve EN Grafana dashboards (#2931) ## What ❔ Adding more information to the EN Grafana dashboards, which barely had any. ## Why ❔ Better observability. Fixes BFT-507 * chore(main): release core 24.26.0 (#2928) :robot: I have created a release *beep* *boop* --- ## [24.26.0](https://github.com/matter-labs/zksync-era/compare/core-v24.25.0...core-v24.26.0) (2024-09-23) ### Features * added seed_peers to consensus global config ([#2920](https://github.com/matter-labs/zksync-era/issues/2920)) ([e9d1d90](https://github.com/matter-labs/zksync-era/commit/e9d1d905f1ce86f9de2cf39d79be4b5aada4a81d)) * **circuit_prover:** Add circuit prover ([#2908](https://github.com/matter-labs/zksync-era/issues/2908)) ([48317e6](https://github.com/matter-labs/zksync-era/commit/48317e640a00b016bf7bf782cc94fccaf077ed6d)) * **prover:** Add endpoint to PJM to get queue reports ([#2918](https://github.com/matter-labs/zksync-era/issues/2918)) ([2cec83f](https://github.com/matter-labs/zksync-era/commit/2cec83f26e0b9309387135ca43718af4fcd6f6b1)) * **vm:** Do not panic on VM divergence ([#2705](https://github.com/matter-labs/zksync-era/issues/2705)) ([7aa5721](https://github.com/matter-labs/zksync-era/commit/7aa5721d22e253d05d369a60d5bcacbf52021c48)) * **vm:** Extract oneshot VM executor – environment types ([#2885](https://github.com/matter-labs/zksync-era/issues/2885)) ([a2d4126](https://github.com/matter-labs/zksync-era/commit/a2d4126f9e0c9dc46f49a861549d076fdbcf66d3)) ### Bug Fixes * **api_server:** fix blob_gas length ([#2673](https://github.com/matter-labs/zksync-era/issues/2673)) ([44a8f79](https://github.com/matter-labs/zksync-era/commit/44a8f79739704e1216a7b34a0580ad7ba1cce3bd)) * **eth-sender:** print better error message in case of missing blob prices ([#2927](https://github.com/matter-labs/zksync-era/issues/2927)) ([38fc824](https://github.com/matter-labs/zksync-era/commit/38fc824f75e8b0e84f10348d1502fc8a26d12015)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]> * chore(ci): Fix CI building (#2936) After new prover was introduced, a new flag is needed for code that touches GPU related code that has no CUDA available. This PR adds the flag to CI pipelines that were missed. [Sample](https://github.com/matter-labs/zksync-era/actions/runs/10958933706/job/30430256489) broken CI. * feat(vm): Split old and new VM implementations (#2915) ## What ❔ Splits old and new VM implementations in the `multivm` crate: - Old VMs are encapsulated in the `LegacyVmInstance` enum, while new ones in the `FastVmInstance` enum (which includes plain and shadowed VM variants). - Fast VM and `FastVmInstance` now expose a tracer type. - Usage of the Fast VM in the batch executor are updated correspondingly. ## Why ❔ It seems infeasible to unify the tracer model for old and new VMs, so keeping them all in a single enum makes little sense. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Add options for selective installation of zk_inception and zk_supervisor (#2934) ## What ❔ - Introduced `--inception` and `--supervisor` options to allow users to specify which binaries to install. - Removed the `ZKUP_SKIP_ZK_SUPERVISOR` option. ## Why ❔ Make it clear what to install, focusing on what to include rather what to exclude * feat(zk_toolbox): Add zk toolbox unit tests to zks test (#2935) ## What ❔ Add zk toolbox unit tests to zks test * feat(circuit-prover): Add circuit_prover Dockerfile and build rules (#2939) This will enable us to build images for circuit proving (new version). Not tested. --------- Co-authored-by: Yury Akudovich <[email protected]> * feat(contract-verifier): add new compilers (#2947) * ci: Copy setup_keys to docker/circuit-prover-gpu-gar (#2951) ## What ❔ Copy setup keys to be available for docker/circuit-prover-gpu-gar. <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ To build circuit-prover-gpu image with keys. <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * docs: fix invalid links to "namespaces/eth.rs" and "execution_sandbox.rs" (#2945) ## What ❔ This PR fixes 2 invalid links in the documents which was caused by refactor (refactor: Split the rest of the zksync_core (https://github.com/matter-labs/zksync-era/pull/1940)) ## Why ❔ Unclarity in documentation ## Checklist - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(prover): Fixes broken non-GPU setup (#2953) Previously, this was done via a compile flag. This was deemed poor taste and we've moved a specific env-var - `ZKSYNC_USE_CUDA_STUBS`. This is wired deep within crypto dependencies. This PR updates the dependencies, adds documentation on how to work with the flag and updates all workflows (fixing the broken ones from RUSTFLAGS time). --------- Co-authored-by: Igor Aleksanov <[email protected]> * chore(prover-gateway): Speed up polling (#2957) 16 minutes is just ridiculous. This should've been done a long time ago, but has been avoided due to other priorities and always put on the back-burner/forgotten. Today we change it. * fix(api): Return correct flat call tracer (#2917) ## What ❔ It was an inconsistency between call tracer results ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * chore(main): release core 24.27.0 (#2940) :robot: I have created a release *beep* *boop* --- ## [24.27.0](https://github.com/matter-labs/zksync-era/compare/core-v24.26.0...core-v24.27.0) (2024-09-25) ### Features * **vm:** Split old and new VM implementations ([#2915](https://github.com/matter-labs/zksync-era/issues/2915)) ([93bc66f](https://github.com/matter-labs/zksync-era/commit/93bc66f21f9f67a440f06f1c4402e0d687698741)) ### Bug Fixes * **api:** Return correct flat call tracer ([#2917](https://github.com/matter-labs/zksync-era/issues/2917)) ([218646a](https://github.com/matter-labs/zksync-era/commit/218646aa1c56200f4ffee99b7f83366e2689354f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]> * feat(eth-watch): redesign to support multiple chains (#2867) This PR contains code from sync-layer-stable + I removed old upgrades processor and updated unit-tests to use the new one --------- Signed-off-by: tomg10 <[email protected]> * feat(prover): WG refactoring #3 (#2942) ## What ❔ Create WG struct with generic round param. Implement generic JobProcessor for all types of WG. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(eth-sender): remove deprecated proof loading (#2958) ## What ❔ Removes deprecated proof loading for eth-sender ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(eth-watch): add missing check that from_block is not larger than finalized_block (#2969) Signed-off-by: tomg10 <[email protected]> * refactor: Deleted unnecessary files (#2959) ## What ❔ - ~`.cargo/config.toml` is a remnant from when we used private repos. It should no longer be necessary.~ - ~`configs/` doesn't seem to be used at all.~ - `building-from-scratch` just contains an old docker file that was used for Coinbase. No longer used. * fix: cargo-deny advisory due to structopt (#2972) ## What ❔ Adds https://rustsec.org/advisories/RUSTSEC-2024-0375 to `deny.toml`, which is due to `StructOpt`. <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ * Currently it's impossible to migrate to clap as [zkevm_test_harness](https://github.com/matter-labs/era-zkevm_test_harness/blob/v1.5.0/Cargo.toml#L46) still depends on structopt. * This [unblocks](https://github.com/matter-labs/zksync-era/actions/runs/11054218994/job/30710479095) the CI <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix: ignore unknown fields in rpc json response (#2962) In some previous pr, I've added a field to json RPC response (consensusGlobalConfig) and it turned out to be incompatible due to the default parsing behavior: an error was returned when unknown fields were present. In general, for programmatic communications the recommended behaviour is to silently ignore unknown fields, so that adding new fields is compatible. I've also weakened the restriction on accepting new consensus genesis by external nodes - when the unknown fields are silently dropped, it is possible that the node will perceive genesis with the same fork number differently after a binary update (because it receives more fields). * fix(zk_toolbox): Do not panic if mint is not successful (#2973) ## What ❔ Use governor for minting process and do not panic if minting is not successful ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * initial commit * feat: zk_toolbox command for setting attester committee (#2900) zk_toolbox command for setting attester committee in consensus registry contract. Fixes BFT-505 --------- Co-authored-by: Bruno França <[email protected]> * fix: chainstack block limit exceeded (#2974) ## What ❔ Added support for chainstack developer and growth plan. ## Why ❔ I was getting the below error with chainstack's growth plan when I tried run self deployed zksync hyperchain. To fix this, I have added appropriate changes to the code. ``` ERROR zksync_eth_watch: Failed to process new blocks: Eth client error: JSON-RPC request get_logs(filter=Filter { from_block: Some(Number(6732838)), to_block: Some(Number(6757429)), block_hash: None, address: Some(ValueOrArray([0xadd82b0c015be3fb7e8410061b4f15d577b4207f, 0xd6ee28b24f87c1d634540699b00e9c37fdc7b3bd, 0x41b75786db38a486c99972c8322fcb14c4f000a2, 0x5d300908338b85bee3f30f2e0ad196fc1587b70d])), topics: Some([Some(ValueOrArray([0x4531cd5795773d7101c17bdeb9f5ab7f47d7056017506f937083be5d6e77a382, 0x23bc9f5dc037eb49c162fd08c2a4d43dfe70063149e140d502273168da0a0625, 0xd50ef21701c8ef211433b140724b8d6de471e7d822c8a616c3d424fe2d0e98a9]))]), limit: None }) failed: ErrorObject { code: InvalidParams, message: "Block range limit exceeded. See more details at https://docs.chainstack.com/docs/limits#evm-range-limits", data: None } ``` ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(eth-sender): add a cap to time_in_mempool (#2978) Signed-off-by: tomg10 <[email protected]> * chore: update derive_more to stable version (#2961) ## What ❔ Updates `derive_more` to a stable version. <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ We're currently on beta and this is a requisite for supporting [foundry-zksync](https://github.com/matter-labs/foundry-zksync). <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * use Notify for a more deterministic approach * feat(da-clients): add secrets (#2954) ## What ❔ Add an opportunity to store the sensitive information of DA clients in the secrets. ## Why ❔ Some sensitive info is currently stored as plaintext, which is not secure, we should use secrets for those values ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: Danil <[email protected]> * fix(api): Fix batch fee input for `debug` namespace (#2948) ## What ❔ Changes the `debug_traceCall` handler to use the fee provider for recent blocks (pending / latest / committed). ## Why ❔ Previously, the `debug` namespace used a static batch fee input retrieved at the node initialization. This looks incorrect. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): add fees integration test to toolbox (#2898) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Add fees integration test to toolbox and CI. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> Co-authored-by: Danil <[email protected]> * feat: Expose http debug page (#2952) ## What ❔ Exposes the consensus http debug on port 5000. Fixes BFT-508 * feat(zk_inception): Add ecosystem ports scanner (#2849) ## What ❔ Added `EcosystemPortsScanner` tool to scan ecosystem directory config and find all values for ports. Returns `EcosystemPorts` object with ports found and custom description of a service. `EcosystemPorts` also contains methods to: check whether the port is assigned, allocate a new port in range, etc. ## Why ❔ With multiple chains and number of services (like explorer, etc.) growing - it is useful to have a tool (`EcosystemPorts`) that can be used to allocate new ports, thus, lowering the chance of port conflicts within created configuration. ## Checklist - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: Matías Ignacio González <[email protected]> * build: Set consensus and pruning as default for EN (#2956) ## What ❔ ENs, started from docker compose, now have consensus and pruning enabled as default. Also, the EN docs were updated. * fix(ci): Remove deprecated artifacts dir in contracts (#2989) ## What ❔ Removing deprecated `artifacts` dir from build process ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * chore(zk_toolbox): Use load current chain function (#2990) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. Signed-off-by: Danil <[email protected]> * perf(api): More efficient gas estimation (#2937) ## What ❔ Makes gas estimation more efficient by using heuristics similar to ones used by Ethereum clients: - Defines the lower gas limit bound as the gas consumed with an effectively infinite gas limit. - Defines an initial binary search pivot as a slightly larger value so that in the best / average case, most of the search space is immediately discarded. ## Why ❔ Improves gas estimation latency in the best and average case. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * test: Base token price client, fix: ratio calculation multiplicative inverse error (#2970) ## What ❔ - I introduce tests to external_price_api crate. Some work was taken from https://github.com/matter-labs/zksync-era/pull/2315 - fix multiplicative inverse ratio calculation bug - fix missing checks in get_fraction ## Why ❔ - We want to make sure the flow of data from API response to token ratio is correct ## Checklist - [X] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [X] Tests for the changes have been added / updated. - [X] Documentation comments have been added / updated. - [X] Code has been formatted via `zk fmt` and `zk lint`. * chore(githooks): Use zk_toolbox in githooks (#2895) ## What ❔ Switch to zk_supervisor fmt in our githooks ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. --------- Signed-off-by: Danil <[email protected]> * feat(zk_toolbox): Add SQL format for zk supervisor (#2950) ## What ❔ Add SQL format for zk supervisor * feat(ci): Add Makefile for easy local build (#2985) ## What ❔ Add Makefile for easier local build of all components ## Why ❔ ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk_toolbox): wrong configs path for prover binaries (#2922) ## What ❔ Right now, while running prover binaries inside of zksync-era, config paths are specified as path to config from chain config(which is `./chain/configs` - relative path, but not absolute). This breaks, because to run prover binaries, shell enters prover directory of era. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(zk_toolbox): Migrate db during update (#2995) ## What ❔ Update databases during update process, now it's a universal receipt how to pull changes ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. Signed-off-by: Danil <[email protected]> * fix(ci): Bring back hack for contracts build till full migration to foundry (#3000) ## What ❔ Creation of empty dirs, to support building from both new (foundry-built) and old contracts ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * chore(main): release core 24.28.0 (#2968) :robot: I have created a release *beep* *boop* --- ## [24.28.0](https://github.com/matter-labs/zksync-era/compare/core-v24.27.0...core-v24.28.0) (2024-10-02) ### Features * **da-clients:** add secrets ([#2954](https://github.com/matter-labs/zksync-era/issues/2954)) ([f4631e4](https://github.com/matter-labs/zksync-era/commit/f4631e4466de620cc1401b326d864cdb8b48a05d)) * **eth-sender:** add a cap to time_in_mempool ([#2978](https://github.com/matter-labs/zksync-era/issues/2978)) ([650d42f](https://github.com/matter-labs/zksync-era/commit/650d42fea6124d80b60a8270a303d72ad6ac741e)) * **eth-watch:** redesign to support multiple chains ([#2867](https://github.com/matter-labs/zksync-era/issues/2867)) ([aa72d84](https://github.com/matter-labs/zksync-era/commit/aa72d849c24a664acd083eba73795ddc5d31d55f)) * Expose http debug page ([#2952](https://github.com/matter-labs/zksync-era/issues/2952)) ([e0b6488](https://github.com/matter-labs/zksync-era/commit/e0b64888aae7324aec2d40fa0cd51ea7e1450cd9)) * **zk_toolbox:** add fees integration test to toolbox ([#2898](https://github.com/matter-labs/zksync-era/issues/2898)) ([e7ead76](https://github.com/matter-labs/zksync-era/commit/e7ead760ce0417dd36af3839ac557f7e9ab238a4)) * **zk_toolbox:** Add SQL format for zk supervisor ([#2950](https://github.com/matter-labs/zksync-era/issues/2950)) ([540e5d7](https://github.com/matter-labs/zksync-era/commit/540e5d7554f54e80d52f1bfae37e03ca8f787baf)) ### Bug Fixes * **api:** Fix batch fee input for `debug` namespace ([#2948](https://github.com/matter-labs/zksync-era/issues/2948)) ([79b6fcf](https://github.com/matter-labs/zksync-era/commit/79b6fcf8b5d10a0ccdceb846370dd6870b6a32b5)) * chainstack block limit exceeded ([#2974](https://github.com/matter-labs/zksync-era/issues/2974)) ([4ffbf42](https://github.com/matter-labs/zksync-era/commit/4ffbf426de166c11aaf5d7b5ed7d199644fba229)) * **eth-watch:** add missing check that from_block is not larger than finalized_block ([#2969](https://github.com/matter-labs/zksync-era/issues/2969)) ([3f406c7](https://github.com/matter-labs/zksync-era/commit/3f406c7d0c0e76d798c2d838abde57ca692822c0)) * ignore unknown fields in rpc json response ([#2962](https://github.com/matter-labs/zksync-era/issues/2962)) ([692ea73](https://github.com/matter-labs/zksync-era/commit/692ea73f75a5fb9db2b4ac33ad24d20568638742)) ### Performance Improvements * **api:** More efficient gas estimation ([#2937](https://github.com/matter-labs/zksync-era/issues/2937)) ([3b69e37](https://github.com/matter-labs/zksync-era/commit/3b69e37e470dab859a55787f6cc971e7083de2fd)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]> * feat(zk_toolbox): Add subcommands and flags for chain registration (#2946) ## What ❔ Added subcommands: `zki chain init configs` - just creates configs with an intention to run chain initialization manually via subcommands `zki chain register-chain` - runs steps from `RegisterHyperchain.s.sol` `zki chain accept-chain-ownership` - accepts ownership for `DiamondProxy` `zki chain genesis database` - initializes database only, performs migration (uses values from args or `secrets.yaml`) `zki chain genesis server` - runs server --genesis Added flags: `zki ecosystem init --ecosystem-only` - runs `init` only for ecosystem (skips `init` for chain) Other changes: * Fixed issue with `--wallet_path` value ignored * Nullify database names if `zki ecosystem init` is used for multiple chains * Zeroify some addresses in `contracts.yaml` when copying from ecosystem during init ## Why ❔ These changes allow us to run the chain registration process for externally hosted chains. Not ideal yet, but the process goes like this: L1 side: * Init ecosystem: `zki ecosystem create && zki ecosystem init --ecosystem-only && zki chain init configs` * Fill in wallets * Register chain: `zki chain register-chain` * Deploy L2 contracts: `zki chain deploy-l2-contracts` * Share `contracts.yaml` L2 side: * Init ecosystem: `zki ecosystem create && zki ecosystem init --ecosystem-only && zki chain init configs` * Fill in wallets * Copy `contracts.yaml` * Accept ownership: `zki chain accept-chain-ownership` * Initialize databases: `zki chain genesis database` * Run server genesis: `zki chain genesis server` ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. * fix(zk_toolbox): Correct secrets (#3004) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. Signed-off-by: Danil <[email protected]> * replace atomic for mutex * move const to config * feat(ci): Add external-node build to Makefile (#3002) ## What ❔ Add external-node build to Makefile ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * fix(zk_toolbox): Remove prover db from server init (#3009) ## What ❔ Remove prover db from server init ## Why ❔ prover db is not needed in server init since prover db is initialized with prover init * fix(ci): Build zk-env with CUDA (#3013) ## What ❔ Ability to build zk-env with cuda not only from push to main branch ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * Update contracts * Finalize merge * feat(en): periodically fetch bridge addresses (#2949) ## What ❔ Periodically fetch bridge addresses ## Why ❔ Addresses will be changed during gateway upgrade ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk fmt` and `zk lint`. * feat(eth-watch): catch another reth error (#3026) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * test(api): Add more `TxSender` tests (#3001) ## What ❔ Covers more `TxSender` functionality with unit tests. ## Why ❔ More test coverage is good. Also, it will help with new VM integration (by running the added tests with a shadowed VM). ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * feat(eth-sender): add time_in_mempool_cap config (#3018) Configuration parameter for time_in_mempool_cap, cap for time_in_mempool in eth-sender fee model + default values for parameter_a and parameter_b for GasAdjuster * fix(zkstack_cli): Fix fmt --check (#3027) ## What ❔ `fmt --check` wasn't returning an error code when failing * docs(zk_toolbox): remove generate-sk subcommand references (#3016) ## What ❔ <!-- What are the changes this PR brings about? --> <!-- Example: This PR adds a PR template to the repo. --> <!-- (For bigger PRs adding more context is appreciated) --> Remove `generate-sk` subcommand which doesn’t exist anymore. ## Why ❔ <!-- Why are these changes done? What goal do they contribute to? What are the principles behind them? --> <!-- Example: PR templates ensure PR reviewers, observers, and future iterators are in context about the evolution of repos. --> ## Checklist <!-- Check your PR fulfills the following items. --> <!-- For draft PRs check the boxes as you complete them. --> - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * chore(vm): Remove tests for old VM versions (#3015) ## What ❔ Removes unit test files for old VM versions in the `multivm` crate. ## Why ❔ These tests are not built (not included into the module tree of the crate), so they are just deadweight. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). * Update contracts * refactor(eth): Brush up `eth_signer` crate (#3014) ## What ❔ Brushes up the `eth_signer` crate so that its API is easier to use (e.g., it doesn't require an async runtime). ## Why ❔ Improved DevEx. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`. * Update query jsons * fix(vm): Prepare new VM for use in API server and fix divergences (#2994) ## What ❔ Prepares the new VM for use in the API server (without tracers): - Fills in more statistics - Covers more statistic fields in divergence checks and fixes the corresponding divergence in `pubdata_published` - Fixes a divergence if revert reaches the bootloader call frame (happens in the call execution mode) - Fixes a panic in `ShadowVm` if the transaction hash is not set and makes `impl Debug for Transaction` non-panicking. (A hash is not set if a transaction is converted from a call.) ## Why ❔ Part of preparations for integrating the new VM into the API server. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`. --------- Co-authored-by: Joonatan Saarhelo <[email protected]> * feat: Increased the timeout of waiting for db in tests (#3007) I don't know about other people's setup, but for me it is very flaky with just 3s * feat: add metric to track current cbt ratio (#3020) Add a new metric to track custom base token to ETH ratio * Format md * Refactor secrets --------- Signed-off-by: Danil <[email protected]> Signed-off-by: Harald Hoyer <[email protected]> Signed-off-by: tomg10 <[email protected]> Co-authored-by: Artem Makhortov <[email protected]> Co-authored-by: Alex Ostrovski <[email protected]> Co-authored-by: Danil <[email protected]> Co-authored-by: Nacho Avecilla <[email protected]> …
github-merge-queue bot
pushed a commit
that referenced
this pull request
Oct 31, 2024
🤖 I have created a release *beep* *boop* --- ## [16.6.0](prover-v16.5.0...prover-v16.6.0) (2024-10-31) ### Features * (DB migration) Rename recursion_scheduler_level_vk_hash to snark_wrapper_vk_hash ([#2809](#2809)) ([64f9551](64f9551)) * Add initial version prover_autoscaler ([#2993](#2993)) ([ebf9604](ebf9604)) * added seed_peers to consensus global config ([#2920](#2920)) ([e9d1d90](e9d1d90)) * attester committees data extractor (BFT-434) ([#2684](#2684)) ([92dde03](92dde03)) * Bump crypto and protocol deps ([#2825](#2825)) ([a5ffaf1](a5ffaf1)) * **circuit_prover:** Add circuit prover ([#2908](#2908)) ([48317e6](48317e6)) * **consensus:** Support for syncing blocks before consensus genesis over p2p network ([#3040](#3040)) ([d3edc3d](d3edc3d)) * **da-clients:** add secrets ([#2954](#2954)) ([f4631e4](f4631e4)) * gateway preparation ([#3006](#3006)) ([16f2757](16f2757)) * Integrate tracers and implement circuits tracer in vm2 ([#2653](#2653)) ([87b02e3](87b02e3)) * Move prover data to /home/popzxc/workspace/current/zksync-era/prover/data ([#2778](#2778)) ([62e4d46](62e4d46)) * Prover e2e test ([#2975](#2975)) ([0edd796](0edd796)) * **prover:** add CLI option to run prover with max allocation ([#2794](#2794)) ([35e4cae](35e4cae)) * **prover:** Add endpoint to PJM to get queue reports ([#2918](#2918)) ([2cec83f](2cec83f)) * **prover:** Add error to panic message of prover ([#2807](#2807)) ([6e057eb](6e057eb)) * **prover:** Add min_provers and dry_run features. Improve metrics and test. ([#3129](#3129)) ([7c28964](7c28964)) * **prover:** Add scale failure events watching and pods eviction. ([#3175](#3175)) ([dd166f8](dd166f8)) * **prover:** Add sending scale requests for Scaler targets ([#3194](#3194)) ([767c5bc](767c5bc)) * **prover:** Add support for scaling WGs and compressor ([#3179](#3179)) ([c41db9e](c41db9e)) * **prover:** Autoscaler sends scale request to appropriate agents. ([#3150](#3150)) ([bfedac0](bfedac0)) * **prover:** Extract keystore into a separate crate ([#2797](#2797)) ([e239260](e239260)) * **prover:** Optimize setup keys loading ([#2847](#2847)) ([19887ef](19887ef)) * **prover:** Refactor WitnessGenerator ([#2845](#2845)) ([934634b](934634b)) * **prover:** Update witness generator to zkevm_test_harness 0.150.6 ([#3029](#3029)) ([2151c28](2151c28)) * **prover:** Use query macro instead string literals for queries ([#2930](#2930)) ([1cf959d](1cf959d)) * **prover:** WG refactoring [#3](#3) ([#2942](#2942)) ([df68762](df68762)) * **prover:** WitnessGenerator refactoring [#2](#2) ([#2899](#2899)) ([36e5340](36e5340)) * Refactor metrics/make API use binaries ([#2735](#2735)) ([8ed086a](8ed086a)) * Remove prover db from house keeper ([#2795](#2795)) ([85b7346](85b7346)) * **tee:** use hex serialization for RPC responses ([#2887](#2887)) ([abe0440](abe0440)) * **utils:** Rework locate_workspace, introduce Workspace type ([#2830](#2830)) ([d256092](d256092)) * vm2 tracers can access storage ([#3114](#3114)) ([e466b52](e466b52)) * **vm:** Do not panic on VM divergence ([#2705](#2705)) ([7aa5721](7aa5721)) * **vm:** EVM emulator support – base ([#2979](#2979)) ([deafa46](deafa46)) * **vm:** Extract batch executor to separate crate ([#2702](#2702)) ([b82dfa4](b82dfa4)) * **zk_toolbox:** `zk_supervisor prover` subcommand ([#2820](#2820)) ([3506731](3506731)) * **zk_toolbox:** Add external_node consensus support ([#2821](#2821)) ([4a10d7d](4a10d7d)) * **zk_toolbox:** Add SQL format for zk supervisor ([#2950](#2950)) ([540e5d7](540e5d7)) * **zk_toolbox:** deploy legacy bridge ([#2837](#2837)) ([93b4e08](93b4e08)) * **zk_toolbox:** Redesign zk_toolbox commands ([#3003](#3003)) ([114834f](114834f)) * **zkstack_cli:** Build dependencies at zkstack build time ([#3157](#3157)) ([724d9a9](724d9a9)) ### Bug Fixes * allow compilation under current toolchain ([#3176](#3176)) ([89eadd3](89eadd3)) * **api:** Return correct flat call tracer ([#2917](#2917)) ([218646a](218646a)) * count SECP256 precompile to account validation gas limit as well ([#2859](#2859)) ([fee0c2a](fee0c2a)) * Fix Doc lint. ([#3158](#3158)) ([c79949b](c79949b)) * ignore unknown fields in rpc json response ([#2962](#2962)) ([692ea73](692ea73)) * **prover:** Do not exit on missing watcher data. ([#3119](#3119)) ([76ed6d9](76ed6d9)) * **prover:** fix setup_metadata_to_setup_data_key ([#2875](#2875)) ([4ae5a93](4ae5a93)) * **prover:** Run for zero queue to allow scaling down to 0 ([#3115](#3115)) ([bbe1919](bbe1919)) * **tee_verifier:** correctly initialize storage for re-execution ([#3017](#3017)) ([9d88373](9d88373)) * **vm:** Prepare new VM for use in API server and fix divergences ([#2994](#2994)) ([741b77e](741b77e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
Create WG struct with generic round param.
Implement generic JobProcessor for all types of WG.
Why ❔
Checklist
zk fmtandzk lint.