Research on Light Client for Mobile

[Danie0918]

Nowadays, mobile has become the main use of the user's scene, we need to complete the light client mobile adaptation to develop mobile Neuron.

[yanguoyu]

There may be two approaches to running a light client on Mobile

1. Build a Mobile arch target to run it.
2. Depend on a virtual environment to run computer arch target.

Here is the reference:
Build the ios target: https://doc.rust-lang.org/rustc/platform-support/arm64e-apple-ios.html
A virtual environment on ios: https://github.com/utmapp/UTM This will start a new OS, so it does not belong to run a node at Mobile.

I will try to see which approach works best.

Update 06-11:
Build target x86_64-apple-ios failed with ckb-rocksdb. I will check why the ckb-rocksdb build failed.

[yanguoyu]

Recording the compilation process:

1. Encountered an issue with not finding the openssl library corresponding to iOS, requiring manual compilation of the openssl library or using a precompiled version from someone else. https://github.com/x2on/OpenSSL-for-iPhone

export OPENSSL_LIB_DIR=OpenSSL-for-iPhone/bin/iPhoneSimulator17.4-x86_64.sdk/lib
export OPENSSL_INCLUDE_DIR=OpenSSL-for-iPhone/bin/iPhoneSimulator17.4-x86_64.sdk/include

2. ckb-rocksdb does not support iOS compilation targets, requiring modification of the compilation script. Here is the fork repo's PR
3. Currently, iOS does not support the thread_local keyword. I have removed this keyword for now, and further research is needed to determine if it can be replaced with another method.
4. Currently, it can be compiled into a library and used in an iOS project. The build is successful, but the call fails due to the subCommand not being passed. Reference: https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html

[Keith-CY]

Please share the solution of the compilation for iOS, because there's a community team interested in running the light client on mobile. Maybe we can push it together.

[yanguoyu]

Please share the solution of the compilation for iOS, because there's a community team interested in running the light client on mobile. Maybe we can push it together.

#393 (comment)
By referring to the above, we can build a light client for iOS and invoke it in the iOS application.

[yanguoyu]

Currently, the light client was called by the iOS app, but the light client run failed with some exceptions. It may be caused by the ckb-rocksdb because I have removed the thread_local locally, I will find a way to replace the thread_local .

[yanguoyu]

Major progress: The light client can now run successfully on iOS. Here is the demo

2024-08-01.21.37.36.mov

Here is the full process:

1. Using a precompiled version from here. https://github.com/x2on/OpenSSL-for-iPhone

export OPENSSL_LIB_DIR=OpenSSL-for-iPhone/bin/iPhoneSimulator17.4-x86_64.sdk/lib
export OPENSSL_INCLUDE_DIR=OpenSSL-for-iPhone/bin/iPhoneSimulator17.4-x86_64.sdk/include

2. Fork and edit ckb-rocksdb

• Use https://github.com/wang-bin/ThreadLocal to replace thread_local.

  1. Replace thread_local type to THREAD_LOCAL(type), for example, the thread_local_adapt.h is copy from https://github.com/wang-bin/ThreadLocal/blob/50eb105f3afd2136a27496081b7b24f567f0f30f/ThreadLocal.h

       + #include "util/thread_local_adapt.h"
       - thread_local size_t ConcurrentArena::tls_cpuid = 0;
       + THREAD_LOCAL(size_t) ConcurrentArena::tls_cpuid = 0;

  2. somewhere we need to change point to reference.

       - thread_status_data_->thread_type = ttype;
       - thread_status_data_->thread_id = thread_id;
       + (*thread_status_data_).thread_type = ttype;
       + (*thread_status_data_).thread_id = thread_id;

  3. std::optional is available, but not value() methods of the class, we should change std::optional<size_t>->value() to *std::optional<size_t>. Refer to https://forums.developer.apple.com/forums/thread/120977

       - EncodeFileBoundaries(dst, f, ts_sz.value());
       + EncodeFileBoundaries(dst, f, (*ts_sz));

• Add iOS compilation refer to feat: enable IOS building rust-rocksdb/rust-rocksdb#653

3. Fork and edit https://github.com/nervosnetwork/ckb-light-client

• Edit Cargo.toml; change rocksdb refer to local change repo and add exort for lib

[patch.crates-io]
rocksdb = { package = "ckb-rocksdb", path="the ckb-rocksdk that fork and edit in previous", version ="=0.21.1" }

[lib]
name = "ckb_light_client"
crate-type = ["staticlib", "cdylib"]

• Add a lib.rs to build for lib

#![allow(clippy::mutable_key_type)]

#[cfg(test)]
#[macro_use]
mod tests;

mod config;
mod error;
mod protocols;
mod service;
mod storage;
mod subcmds;
mod types;
mod utils;
mod verify;

use ckb_app_config::HeaderMapConfig;
use ckb_app_config::NetworkConfig;
use ckb_app_config::SyncConfig;
use config::RunConfig;
use config::parse_config_file;
use env_logger::{Builder, Env, Target};
use types::RpcConfig;
use types::RunEnv;
use types::StoreConfig;
use std::path::PathBuf;
use std::path::Path;
use ckb_types::U256;
use std::os::raw::c_char;
use std::ffi::{CString, CStr};

#[no_mangle]
pub extern fn run_light(to: *const c_char) -> anyhow::Result<()> {
let mut builder = Builder::from_env(Env::default());
builder.target(Target::Stdout);
builder
    .try_init()
    .expect("env_logger builder init should be ok");


let c_str = unsafe { CStr::from_ptr(to) };
let recipient = match c_str.to_str() {
    Err(_) => "there",
    Ok(string) => string,
};
println!("Starting run light client ..., config path is {}", recipient.to_string());
let run_env = parse_config_file::<RunEnv>(recipient)?;
match (RunConfig{ run_env }).execute() {
  Ok(_) => {
    println!("start light client success");
  }
  Err(e) => {
    println!("start light client failed ERROR: {:?}", e)
  }
};

println!("Done.");

Ok(())
}

• export parse_config_file from config.rs

        pub(crate) fn parse_config_file<T: FromStr>(file: &str) -> Result<T>
        where
            <T as FromStr>::Err: Display,
        {
            OpenOptions::new()
            .read(true)
            .open(file)
            .map_err(|err| Error::config(format!("failed to open {} since {}", file, err)))
            .and_then(|mut f| {
                let mut buffer = String::new();
                f.read_to_string(&mut buffer)
                    .map_err(|err| {
                        Error::config(format!("failed to read {} since {}", file, err))
                    })
                    .map(|_| buffer)
            })
            .and_then(|data| T::from_str(&data).map_err(Error::config))
        }

4. Build the ckb-light-client for ios and use it in the iOS project refer to https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html and here is some code snippet https://github.com/yanguoyu/demo-run-light-iOS

cargo build --target x86_64-apple-ios