feat(zk_toolbox): Add block explorer support to zk_toolbox (#2768)

## What ❔
New `zk_inception explorer` command for easy block explorer setup.

### Usage:
`zk_inception explorer init` - initializes explorer database and creates
config files (executed for all chains, unless `--chain` is passed)
`zk_inception explorer backend` - runs backend
[services](https://github.com/matter-labs/block-explorer?tab=readme-ov-file#-architecture)
(api, data_fetcher, worker) required for block explorer app for a single
chain (uses default chain, unless `--chain` is passed)
`zk_inception explorer run` - runs block-explorer-app (displays all
chains, unless `--chain` is passed)

### Config structure:
* Ecosystem level apps configs:
* `ecosystem/configs/apps.yaml` - ecosystem-level configuration for
apps, edit that if you want to customize the port for portal and
explorer apps.
* `ecosystem/configs/apps/portal.config.json` - ecosystem-level
configuration for portal app, edit that if you want to customize display
names, tokens list, URLs, etc. for any chain for portal. Refer to the
[format](https://github.com/matter-labs/dapp-portal/blob/main/types/index.d.ts#L137-L149)
and documentation from the
[dapp-portal](https://github.com/matter-labs/dapp-portal) repository.
* `ecosystem/configs/apps/explorer.config.json` - ecosystem-level
configuration for explorer app, edit that if you want to customize
display names, URLs, etc. for any chain for explorer. Refer to the
[format](https://github.com/matter-labs/block-explorer/blob/main/packages/app/src/configs/index.ts#L23)
from [block-explorer](https://github.com/matter-labs/block-explorer)
repository.
* `ecosystem/configs/.generated/explorer.config.js` - this file is
auto-generated on every `explorer run` and injected as a runtime config
to block-explorer-app docker image for run.
* `ecosystem/configs/.generated/portal.config.js` - this file is
auto-generated on every `portal` run and injected as a runtime config to
dapp-portal docker image for run.

* Chain level apps configs:
* `chain/configs/explorer-docker-compose.yml` - configures required
explorer backend services as a docker compose file, edit that if you
want to customize ports, parameters like batches polling interval. It's
user responsibility to adjust corresponding JSON app configs if ports
are changed in this file.

## Why ❔
Currently, running the block-explorer requires users to manually pull
the repository, install all dependencies, prepare database, modify
configurations, build the project, and then run it. This PR simplifies
the process, allowing users to run the explorer effortlessly with a few
commands.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

---------

Co-authored-by: Manuel Mauro <[email protected]>

zk_toolbox/README.md

@@ -247,6 +247,53 @@ Run the external node:
 zk_inception en run
 ```
 
+### Portal
+
+Once you have at least one chain initialized, you can run the [portal](https://github.com/matter-labs/dapp-portal) - a
+web-app to bridge tokens between L1 and L2 and more:
+
+```bash
+zk_inception portal
+```
+
+This command will start the dockerized portal app using configuration from `apps/portal.config.json` file inside your
+ecosystem directory. You can edit this file to configure the portal app if needed. By default, portal starts on
+`http://localhost:3030`, you can configure the port in `apps.yaml` file.
+
+### Explorer
+
+For better understanding of the blockchain data, you can use the
+[explorer](https://github.com/matter-labs/block-explorer) - a web-app to view and inspect transactions, blocks,
+contracts and more.
+
+First, each chain should be initialized:
+
+```bash
+zk_inception explorer init
+```
+
+This command creates a database to store explorer data and generatesdocker compose file with explorer services
+(`explorer-docker-compose.yml`).
+
+Next, for each chain you want to have an explorer, you need to start its backend services:
+
+```bash
+zk_inception explorer backend --chain <chain_name>
+```
+
+This command uses previously created docker compose file to start the services (api, data fetcher, worker) required for
+the explorer.
+
+Finally, you can run the explorer app:
+
+```bash
+zk_inception explorer run
+```
+
+This command will start the dockerized explorer app using configuration from `apps/explorer.config.json` file inside
+your ecosystem directory. You can edit this file to configure the app if needed. By default, explorer starts on
+`http://localhost:3010`, you can configure the port in `apps.yaml` file.
+
 ### Update
 
 To update your node:

zk_toolbox/crates/common/src/docker.rs

@@ -1,26 +1,33 @@
-use std::collections::HashMap;
-
+use url::Url;
 use xshell::{cmd, Shell};
 
 use crate::cmd::Cmd;
 
-pub fn up(shell: &Shell, docker_compose_file: &str) -> anyhow::Result<()> {
-    Ok(Cmd::new(cmd!(shell, "docker compose -f {docker_compose_file} up -d")).run()?)
+pub fn up(shell: &Shell, docker_compose_file: &str, detach: bool) -> anyhow::Result<()> {
+    let args = if detach { vec!["-d"] } else { vec![] };
+    let mut cmd = Cmd::new(cmd!(
+        shell,
+        "docker compose -f {docker_compose_file} up {args...}"
+    ));
+    cmd = if !detach { cmd.with_force_run() } else { cmd };
+    Ok(cmd.run()?)
 }
 
 pub fn down(shell: &Shell, docker_compose_file: &str) -> anyhow::Result<()> {
     Ok(Cmd::new(cmd!(shell, "docker compose -f {docker_compose_file} down")).run()?)
 }
 
-pub fn run(
-    shell: &Shell,
-    docker_image: &str,
-    docker_args: HashMap<String, String>,
-) -> anyhow::Result<()> {
-    let mut args = vec![];
-    for (key, value) in docker_args.iter() {
-        args.push(key);
-        args.push(value);
+pub fn run(shell: &Shell, docker_image: &str, docker_args: Vec<String>) -> anyhow::Result<()> {
+    Ok(Cmd::new(cmd!(shell, "docker run {docker_args...} {docker_image}")).run()?)
+}
+
+pub fn adjust_localhost_for_docker(mut url: Url) -> anyhow::Result<Url> {
+    if let Some(host) = url.host_str() {
+        if host == "localhost" || host == "127.0.0.1" {
+            url.set_host(Some("host.docker.internal"))?;
+        }
+    } else {
+        anyhow::bail!("Failed to parse: no host");
     }
-    Ok(Cmd::new(cmd!(shell, "docker run {args...} {docker_image}")).run()?)
+    Ok(url)
 }

zk_toolbox/crates/config/src/apps.rs

@@ -0,0 +1,59 @@
+use std::path::{Path, PathBuf};
+
+use serde::{Deserialize, Serialize};
+use xshell::Shell;
+
+use crate::{
+    consts::{APPS_CONFIG_FILE, DEFAULT_EXPLORER_PORT, DEFAULT_PORTAL_PORT, LOCAL_CONFIGS_PATH},
+    traits::{FileConfigWithDefaultName, ReadConfig, SaveConfig, ZkToolboxConfig},
+};
+
+/// Ecosystem level configuration for the apps (portal and explorer).
+#[derive(Debug, Serialize, Deserialize, Clone)]
+pub struct AppsEcosystemConfig {
+    pub portal: AppEcosystemConfig,
+    pub explorer: AppEcosystemConfig,
+}
+
+#[derive(Debug, Serialize, Deserialize, Clone)]
+pub struct AppEcosystemConfig {
+    pub http_port: u16,
+}
+
+impl ZkToolboxConfig for AppsEcosystemConfig {}
+impl FileConfigWithDefaultName for AppsEcosystemConfig {
+    const FILE_NAME: &'static str = APPS_CONFIG_FILE;
+}
+
+impl AppsEcosystemConfig {
+    pub fn get_config_path(ecosystem_base_path: &Path) -> PathBuf {
+        ecosystem_base_path
+            .join(LOCAL_CONFIGS_PATH)
+            .join(APPS_CONFIG_FILE)
+    }
+
+    pub fn read_or_create_default(shell: &Shell) -> anyhow::Result<Self> {
+        let config_path = Self::get_config_path(&shell.current_dir());
+        match Self::read(shell, &config_path) {
+            Ok(config) => Ok(config),
+            Err(_) => {
+                let config = Self::default();
+                config.save(shell, &config_path)?;
+                Ok(config)
+            }
+        }
+    }
+}
+
+impl Default for AppsEcosystemConfig {
+    fn default() -> Self {
+        AppsEcosystemConfig {
+            portal: AppEcosystemConfig {
+                http_port: DEFAULT_PORTAL_PORT,
+            },
+            explorer: AppEcosystemConfig {
+                http_port: DEFAULT_EXPLORER_PORT,
+            },
+        }
+    }
+}

zk_toolbox/crates/config/src/consts.rs

@@ -30,12 +30,43 @@ pub const ERA_OBSERVABILITY_COMPOSE_FILE: &str = "era-observability/docker-compo
 pub const ERA_OBSERBAVILITY_DIR: &str = "era-observability";
 /// Era observability repo link
 pub const ERA_OBSERBAVILITY_GIT_REPO: &str = "https://github.com/matter-labs/era-observability";
+pub(crate) const LOCAL_APPS_PATH: &str = "apps/";
+pub(crate) const LOCAL_CHAINS_PATH: &str = "chains/";
 pub(crate) const LOCAL_CONFIGS_PATH: &str = "configs/";
+pub(crate) const LOCAL_GENERATED_PATH: &str = ".generated/";
 pub(crate) const LOCAL_DB_PATH: &str = "db/";
 pub(crate) const LOCAL_ARTIFACTS_PATH: &str = "artifacts/";
 
-/// Name of portal config file
-pub const PORTAL_CONFIG_FILE: &str = "portal.config.js";
+/// Name of apps config file
+pub const APPS_CONFIG_FILE: &str = "apps.yaml";
+/// Name of portal runtime config file (auto-generated)
+pub const PORTAL_JS_CONFIG_FILE: &str = "portal.config.js";
+/// Name of portal config JSON file
+pub const PORTAL_CONFIG_FILE: &str = "portal.config.json";
+/// Name of explorer runtime config file (auto-generated)
+pub const EXPLORER_JS_CONFIG_FILE: &str = "explorer.config.js";
+/// Name of explorer config JSON file
+pub const EXPLORER_CONFIG_FILE: &str = "explorer.config.json";
+/// Name of explorer docker compose file
+pub const EXPLORER_DOCKER_COMPOSE_FILE: &str = "explorer-docker-compose.yml";
+
+/// Default port for the explorer app
+pub const DEFAULT_EXPLORER_PORT: u16 = 3010;
+/// Default port for the portal app
+pub const DEFAULT_PORTAL_PORT: u16 = 3030;
+/// Default port for the explorer worker service
+pub const DEFAULT_EXPLORER_WORKER_PORT: u16 = 3001;
+/// Default port for the explorer API service
+pub const DEFAULT_EXPLORER_API_PORT: u16 = 3002;
+/// Default port for the explorer data fetcher service
+pub const DEFAULT_EXPLORER_DATA_FETCHER_PORT: u16 = 3040;
+
+pub const EXPLORER_API_DOCKER_IMAGE: &str = "matterlabs/block-explorer-api";
+pub const EXPLORER_DATA_FETCHER_DOCKER_IMAGE: &str = "matterlabs/block-explorer-data-fetcher";
+pub const EXPLORER_WORKER_DOCKER_IMAGE: &str = "matterlabs/block-explorer-worker";
+
+/// Interval (in milliseconds) for polling new batches to process in explorer app
+pub const EXPLORER_BATCHES_PROCESSING_POLLING_INTERVAL: u64 = 1000;
 
 /// Path to ecosystem contacts
 pub(crate) const ECOSYSTEM_PATH: &str = "etc/env/ecosystems";

zk_toolbox/crates/config/src/docker_compose.rs

@@ -0,0 +1,43 @@
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+
+use crate::traits::ZkToolboxConfig;
+
+#[derive(Debug, Default, Serialize, Deserialize, Clone)]
+pub struct DockerComposeConfig {
+    pub services: HashMap<String, DockerComposeService>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub name: Option<String>,
+    #[serde(flatten)]
+    pub other: serde_json::Value,
+}
+
+#[derive(Debug, Serialize, Deserialize, Clone)]
+pub struct DockerComposeService {
+    pub image: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub platform: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub ports: Option<Vec<String>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub environment: Option<HashMap<String, String>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub volumes: Option<Vec<String>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub depends_on: Option<Vec<String>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub restart: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub extra_hosts: Option<Vec<String>>,
+    #[serde(flatten)]
+    pub other: serde_json::Value,
+}
+
+impl ZkToolboxConfig for DockerComposeConfig {}
+
+impl DockerComposeConfig {
+    pub fn add_service(&mut self, name: &str, service: DockerComposeService) {
+        self.services.insert(name.to_string(), service);
+    }
+}

zk_toolbox/crates/config/src/explorer.rs

@@ -0,0 +1,147 @@
+use std::path::{Path, PathBuf};
+
+use serde::{Deserialize, Serialize};
+use xshell::Shell;
+
+use crate::{
+    consts::{
+        EXPLORER_CONFIG_FILE, EXPLORER_JS_CONFIG_FILE, LOCAL_APPS_PATH, LOCAL_CONFIGS_PATH,
+        LOCAL_GENERATED_PATH,
+    },
+    traits::{ReadConfig, SaveConfig, ZkToolboxConfig},
+};
+
+/// Explorer JSON configuration file. This file contains configuration for the explorer app.
+#[derive(Serialize, Deserialize, Debug, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct ExplorerConfig {
+    pub app_environment: String,
+    pub environment_config: EnvironmentConfig,
+    #[serde(flatten)]
+    pub other: serde_json::Value,
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone)]
+pub struct EnvironmentConfig {
+    pub networks: Vec<ExplorerChainConfig>,
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct ExplorerChainConfig {
+    pub name: String, // L2 network chain name (the one used during the chain initialization)
+    pub l2_network_name: String, // How the network is displayed in the app dropdown
+    pub l2_chain_id: u64,
+    pub rpc_url: String,            // L2 RPC URL
+    pub api_url: String,            // L2 API URL
+    pub base_token_address: String, // L2 base token address (currently always 0x800A)
+    pub hostnames: Vec<String>,     // Custom domain to use when switched to this chain in the app
+    pub icon: String,               // Icon to show in the explorer dropdown
+    pub maintenance: bool,          // Maintenance warning
+    pub published: bool, // If false, the chain will not be shown in the explorer dropdown
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub bridge_url: Option<String>, // Link to the portal bridge
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub l1_explorer_url: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub verification_api_url: Option<String>, // L2 verification API URL
+    #[serde(flatten)]
+    pub other: serde_json::Value,
+}
+
+impl ExplorerConfig {
+    /// Returns the path to the explorer configuration file.
+    pub fn get_config_path(ecosystem_base_path: &Path) -> PathBuf {
+        ecosystem_base_path
+            .join(LOCAL_CONFIGS_PATH)
+            .join(LOCAL_APPS_PATH)
+            .join(EXPLORER_CONFIG_FILE)
+    }
+
+    /// Reads the existing config or creates a default one if it doesn't exist.
+    pub fn read_or_create_default(shell: &Shell) -> anyhow::Result<Self> {
+        let config_path = Self::get_config_path(&shell.current_dir());
+        match Self::read(shell, &config_path) {
+            Ok(config) => Ok(config),
+            Err(_) => {
+                let config = Self::default();
+                config.save(shell, &config_path)?;
+                Ok(config)
+            }
+        }
+    }
+
+    /// Adds or updates a given chain configuration.
+    pub fn add_chain_config(&mut self, config: &ExplorerChainConfig) {
+        // Replace if config with the same network name already exists
+        if let Some(index) = self
+            .environment_config
+            .networks
+            .iter()
+            .position(|c| c.name == config.name)
+        {
+            self.environment_config.networks[index] = config.clone();
+            return;
+        }
+        self.environment_config.networks.push(config.clone());
+    }
+
+    /// Retains only the chains whose names are present in the given vector.
+    pub fn filter(&mut self, chain_names: &[String]) {
+        self.environment_config
+            .networks
+            .retain(|config| chain_names.contains(&config.name));
+    }
+
+    /// Hides all chains except those specified in the given vector.
+    pub fn hide_except(&mut self, chain_names: &[String]) {
+        for network in &mut self.environment_config.networks {
+            network.published = chain_names.contains(&network.name);
+        }
+    }
+
+    /// Checks if a chain with the given name exists in the configuration.
+    pub fn contains(&self, chain_name: &String) -> bool {
+        self.environment_config
+            .networks
+            .iter()
+            .any(|config| &config.name == chain_name)
+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.environment_config.networks.is_empty()
+    }
+
+    pub fn save_as_js(&self, shell: &Shell) -> anyhow::Result<PathBuf> {
+        // The block-explorer-app is served as a pre-built static app in a Docker image.
+        // It uses a JavaScript file (config.js) that injects the configuration at runtime
+        // by overwriting the '##runtimeConfig' property of the window object.
+        // This file will be mounted to the Docker image when it runs.
+        let path = Self::get_generated_js_config_path(&shell.current_dir());
+        let json = serde_json::to_string_pretty(&self)?;
+        let config_js_content = format!("window['##runtimeConfig'] = {};", json);
+        shell.write_file(path.clone(), config_js_content.as_bytes())?;
+        Ok(path)
+    }
+
+    fn get_generated_js_config_path(ecosystem_base_path: &Path) -> PathBuf {
+        ecosystem_base_path
+            .join(LOCAL_CONFIGS_PATH)
+            .join(LOCAL_GENERATED_PATH)
+            .join(EXPLORER_JS_CONFIG_FILE)
+    }
+}
+
+impl Default for ExplorerConfig {
+    fn default() -> Self {
+        ExplorerConfig {
+            app_environment: "default".to_string(),
+            environment_config: EnvironmentConfig {
+                networks: Vec::new(),
+            },
+            other: serde_json::Value::Null,
+        }
+    }
+}
+
+impl ZkToolboxConfig for ExplorerConfig {}