* feat(datastream): proto stream write

* feat(datastream): read proto stream

* fixing up some stream related things post rebase

* updating to latest proto for data stream

* some fixes for syncing on the new proto format

* only recover the data stream if execution has done any work

* simplify looking up fork id by batch

* fixing up datastream write

* fixing up tests related to data stream proto changes

* post merge fixes from zkevm branch

* more progress on proto datastream

* hack file for hosting a local datastream without erigon attached

* keep track of the fork id within the stream client rather than the batches stage

* handle ger updates in the new proto format for writing the stream

* fix stream related tests

* remove accidentally committed debug files

* datastream related code tidy up

* fix post zkevm merge problems in proto datastream

* fix(witness): closed growing number of cursors (0xPolygonHermez#416)

* add more detail to proto datastream

* add batch number to batch end proto

* add more details for proto format datastream

* fix(merge): hermez db merge fix

* fix for updating chain config with fork id changes

* fixing up tests after changing the fork lookup code for chain config

* do not write the batch end during genesis block write

this will be handled by the writing of block 1 (injected batch) once it detects that the batch number has changed h

* fix bad build on proto branch

* feat(datastream): l1 cache, debug fix

* feat(docs): cache headers, documentation

---------

Co-authored-by: Scott Fairclough <[email protected]>

* FIX: use actual forkId in handleInjectedBatch

* FIX: use initial forkid from config

…ez#487)

* changes to proto stream block hash and local exit root

* block hash in stream to be the actual block hash

* skip intermediate roots in the stream for post etrog

…ez#490)

* add spec for the datastrea

* review fixes

* review fixes

…ce (0xPolygonHermez#500)

deletes here are also being handled in a transparent way to save any headaches there with mismatches

0xPolygonHermez#503)

* fix for duplicating blocks in the stream and writing too far ahead when no verifications available as RPC

* force min timestamp to be 0 for injected batch

…ez#498)

…onHermez#497)

* feat(l1): configure highest l1 block type, default finalized

* docs(readme): l1 syncer highest block config

…xPolygonHermez#507)

…0xPolygonHermez#510)

…xPolygonHermez#512)

…rmez#515)

…xPolygonHermez#521)

* adding some docs for L1 info tree information from a sequencer perspective

* update to l1 info tree docs

* Set DynamicConfigs path to config directory

Instead of hardcoding dynamic configs path to a directory in user's home path, assume dynamic config files for a given network will be found along the main configuration file.

* Update README

* Typo

…ermez#529)

* handling batch start and end for the executor data stream bytes

* decrease write size for datastream catchup

previous value had the node mysteriously just stop, no panic, nothing, just stop.  Maybe OOM killed by the OS or something

* close state reader

was causing a memory leak over time as the underlying cursors were never closed during stream catchup

* stop looking for LER when we find one during stream generation

* fix for debug tool after proto executor changes

* fix(nightly): correct path for block comparison

* fix(nightly): network5

0xPolygonHermez#528)

* wip

add modes in acl db

refactored code

* acl cli

* Uts

* wip

* downgrade the lib version

* downgrade the lib version

* fix

* update makefile

* downgrade libs

* downgrade lib

* fix go mod

* update acl cmd tests

* fix code smells

* remove code smells

---------

Co-authored-by: Goran Rojovic <[email protected]>

* fix

* fix

* comments fix

* comments fix

* fix

…nt to failure (0xPolygonHermez#546)

)

* phase 1 of l1 info root checks in l1 recovery

* phase 2 in handling special index 0 cases for L1 recovery

* fixing up comment about l1 info tree indexes

* fixup for l1 info tree related tests

…xPolygonHermez#561)

…z#531)

* setup the endpoint

* add endpoint and  openrpc

* colector even not for sequencer

* fix test

* remove missing code

* added trace with counters

* add smtLevels to the trace result

* added smtDepth setting to counter tracer

* a bit of code quality

* comments

* write stream nil on err

* readme

…ez#565)

…ygonHermez#565)" (0xPolygonHermez#574)

This reverts commit dd21966.

* faster population of the data stream for consumers

* tune ups on changes for proto stream speed up

…am one (…" (0xPolygonHermez#575)

This reverts commit 55846ff.

* Support validium mode in L1 recovery

* Retry data fetching from DA when the response error is 429 (too many requests)

* Unit tests and minor fixes

* Test decode validium batch data

* Revert unrelated changes

* Address CR feedbacks

* Remove unused functions

* Update zk/da/client.go

Co-authored-by: Rachit Sonthalia <[email protected]>

---------

Co-authored-by: Igor Mandrigin <[email protected]>
Co-authored-by: Rachit Sonthalia <[email protected]>

* test(nightly): tweak bridge test output

* tweak(nightly): remove token addr from bridge tests

* handle data stream disconnection

* centralise the streaming atomic to a single place for data stream

without it the witness can't be generated

…olygonHermez#583)

* test(nightly): tweak bridge test output

* tweak(nightly): remove token addr from bridge tests

* test(nightly): ansible wire up, fixes

* Update nightly-ansible.yml

* use same private key

* fixups for tests

* fixup

…ez#587)

* inline the launch script

* refactor to use matrix better

* add sepolia endpoing

* fix a typo

* fix secret name for Sepolia

* fix typo

* use env vars

* debug

* fix typos, no debug

* fix more typos

* add bridge tests

* fixups

* fix typo

* fix typos

* fixups

* add bridge tests

* fix

* bridge first

…nHermez#591)

* fix(bug): implement bug for multiple words

* fix(bug): ensure bug logic applies to last word

* fix(bug): implement bug for multiple words

* fix(bug): ensure bug logic applies to last word

* fix(bug): length check

* fix

* do not write genesis block if we already have data in the stream

* correct handling of limit timestamp during l1 recovery

…d shows used counters (0xPolygonHermez#573)

* setup the endpoint

* add endpoint and  openrpc

* colector even not for sequencer

* fix test

* remove missing code

* added trace with counters

* add smtLevels to the trace result

* added smtDepth setting to counter tracer

* a bit of code quality

* comments

* write stream nil on err

* readme

* added endpoint

* separate a bit

* fixed bug, added gas used and a test

* minor fix

* fix oom on stage smt

* update

* Upload kurtosis logs on failure

* Fix yml

…Hermez#603)

…dbx (0xPolygonHermez#607)

* add gitaction nightly-bridge-erc20

* add gitaction nightly-bridge-erc20

* set initial forkid to datastream client

* fix tests

* add gitaction nightly-bridge-erc20

* add gitaction nightly-bridge-erc20

* fix bridge ERC20

* fix bridge ERC20

* fix bridge ERC20

* fix bridge ERC20

…ermez#614)

…ez#620)

Co-authored-by: Igor Mandrigin <[email protected]>

* tweak(l1recovery): panic on startup if stream host not set

* test(nightly): l1 recovery

…xPolygonHermez#629)

* create inner context for pool.flush

* remove debug files

* fix(nightly): l1 recovery set working dir

* ci(mac): upgrade macos deprecated version

…z#642)

* fix(test): policy test temp dir creation

* fix(test): increase timeout of test in makefile

This fixes bug where timestamp limit is always 0 in validium recovery mode

* wip - bug - passing some cases

* wip

* fix(bug): logic matching as closely the zkevm bug

---------

Co-authored-by: Kamen Stoykov <[email protected]>

…xPolygonHermez#660)

* added the test smart contract

* found global exit root

* added counter contract deployer

* don't print memory if it is empty in trace

…ermez#666)

* feat(smt): regen smt in ram flag (default false - slower)

* feat(docs): update to include zkevm.smt-regenerate-in-memory flag

* fix(smt): open batch on config, close tx