Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance wasm with checkpoint and restore support (#2333) #3289

Open
wants to merge 19 commits into
base: dev/checkpoint_and_restore
Choose a base branch
from

Commits on Jun 7, 2024

  1. Enhance wasm with checkpoint and restore support (bytecodealliance#2333)

    - Add wasm_runtime_checkpoint/wasm_runtime_restore API
    - Support AOT and Classic Interpreter mode checkpoint and debug through OS signal, tested on windows/mac/linux aarch64/x64
    - Static instrument the AOT to have the checkpoint and restore switches
    - Add sub extra library folder for implementing the ckpt-restore
    - Include extra dependency of yalantinglib
    
    Co-authored-by: Aibo Hu <[email protected]>
    Co-authored-by: kikispace <[email protected]>
    Co-authored-by: Brian Zhao <[email protected]>
    Signed-off-by: victoryang00 <[email protected]>
    4 people committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    0f7ec2e View commit details
    Browse the repository at this point in the history
  2. Merge branch 'dev/checkpoint_and_restore' into main

    Signed-off-by: victoryang00 <[email protected]>
    Signed-off-by: victoryang00 <[email protected]>
    victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    5070370 View commit details
    Browse the repository at this point in the history
  3. lldb_function_to_function_dbi: A hack to avoid crashing on C++ methods (

    bytecodealliance#3190)
    
    Also, print the function name on argument mismatch.
    
    Signed-off-by: victoryang00 <[email protected]>
    yamt authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    8aee5a7 View commit details
    Browse the repository at this point in the history
  4. Revert PR bytecodealliance#3194 (bytecodealliance#3199)

    - Address values in call stack dump are relative to file beginning
    - If running under fast-interp mode, address values are relative to
      every pre-compiled function beginning, which is not compatible
      with addr2line
    
    Signed-off-by: victoryang00 <[email protected]>
    lum1n0us authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    71cb447 View commit details
    Browse the repository at this point in the history
  5. Get location info from function indexes in addr2line script (bytecode…

    …alliance#3206)
    
    Update the `addr2line` script so that:
    - line info is printed in a more convenient format, e.g.
    ```
    0: c
            at wasm-micro-runtime/test-tools/addr2line/trap.c:5:1
    1: b
            at wasm-micro-runtime/test-tools/addr2line/trap.c:11:12
    2: a
            at wasm-micro-runtime/test-tools/addr2line/trap.c:17:12
    ```
    similar to how Rust prints stack traces when there's a panic. In an IDE, the user
    can conveniently click on the printed path and be redirected to the file line.
    - a new `--no-addr` argument can be provided to the script
    
    It can be used in fast interpreter mode (that is not supported by the script otherwise)
    or with older wamr versions (where the stack trace only had the function index info
    and not the function address). In that case, `wasm-objdump` is used to get the function
    name from the index and `llvm-dwarfdump` to obtain the location info (where the line
    refers to the start of the function).
    
    Signed-off-by: victoryang00 <[email protected]>
    eloparco authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    62a1a18 View commit details
    Browse the repository at this point in the history
  6. Refactor APIs and data structures as preliminary work for Memory64 (b…

    …ytecodealliance#3209)
    
    - (Export)wasm_runtime_module_malloc
      - wasm_module_malloc
        - wasm_module_malloc_internal
      - aot_module_malloc
        - aot_module_malloc_internal
    - wasm_runtime_module_realloc
      - wasm_module_realloc
        - wasm_module_realloc_internal
      - aot_module_realloc
        - aot_module_realloc_internal
    - (Export)wasm_runtime_module_free
      - wasm_module_free
        - wasm_module_free_internal
      - aot_module_malloc
        - aot_module_free_internal
    - (Export)wasm_runtime_module_dup_data
      - wasm_module_dup_data
      - aot_module_dup_data
    - (Export)wasm_runtime_validate_app_addr
    - (Export)wasm_runtime_validate_app_str_addr
    - (Export)wasm_runtime_validate_native_addr
    - (Export)wasm_runtime_addr_app_to_native
    - (Export)wasm_runtime_addr_native_to_app
    - (Export)wasm_runtime_get_app_addr_range
    - aot_set_aux_stack
    - aot_get_aux_stack
    - wasm_set_aux_stack
    - wasm_get_aux_stack
    - aot_check_app_addr_and_convert, wasm_check_app_addr_and_convert
      and jit_check_app_addr_and_convert
    - wasm_exec_env_set_aux_stack
    - wasm_exec_env_get_aux_stack
    - wasm_cluster_create_thread
    - wasm_cluster_allocate_aux_stack
    - wasm_cluster_free_aux_stack
    
    - WASMModule and AOTModule
      - field aux_data_end, aux_heap_base and aux_stack_bottom
    - WASMExecEnv
      - field aux_stack_boundary and aux_stack_bottom
    - AOTCompData
      - field aux_data_end, aux_heap_base and aux_stack_bottom
    - WASMMemoryInstance(AOTMemoryInstance)
      - field memory_data_size and change __padding to is_memory64
    - WASMModuleInstMemConsumption
      - field total_size and memories_size
    - WASMDebugExecutionMemory
      - field start_offset and current_pos
    - WASMCluster
      - field stack_tops
    
    - libc-builtin
    - libc-emcc
    - libc-uvwasi
    - libc-wasi
    - Python and Go Language Embedding
    - Interpreter Debug engine
    - Multi-thread: lib-pthread, wasi-threads and thread manager
    
    Signed-off-by: victoryang00 <[email protected]>
    wenyongh authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    5eb4ebf View commit details
    Browse the repository at this point in the history
  7. trans_wasm_func_name.py: Correct function index during translation (b…

    …ytecodealliance#3232)
    
    Adding the N from "aot_func#N" with the import function count is the correct
    wasm function index.
    
    Signed-off-by: victoryang00 <[email protected]>
    lum1n0us authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    defcae8 View commit details
    Browse the repository at this point in the history
  8. Add CodeQL Workflow for Code Security Analysis (bytecodealliance#2812)

    Add CodeQL Workflow for Code Security Analysis
    
    This pull request introduces a CodeQL workflow to enhance the security analysis of our repository.
    CodeQL is a powerful static analysis tool that helps identify and mitigate security vulnerabilities in
    our codebase. By integrating this workflow into our GitHub Actions, we can proactively identify
    and address potential issues before they become security threats.
    
    We added a new CodeQL workflow file (.github/workflows/codeql.yml) that
    - Runs on nightly-run, and consider runs on every pull request to the main branch in the future.
    - Excludes queries with a high false positive rate or low-severity findings.
    - Does not display results for third-party code, focusing only on our own codebase.
    
    Testing:
    To validate the functionality of this workflow, we have run several test scans on the codebase and
    reviewed the results. The workflow successfully compiles the project, identifies issues, and provides
    actionable insights while reducing noise by excluding certain queries and third-party code.
    
    Deployment:
    Once this pull request is merged, the CodeQL workflow will be active and automatically run on
    every push and pull request to the main branch. To view the results of these code scans, please
    follow these steps:
    1. Under the repository name, click on the Security tab.
    2. In the left sidebar, click Code scanning alerts.
    
    Additional Information:
    - You can further customize the workflow to adapt to your specific needs by modifying the workflow file.
    - For more information on CodeQL and how to interpret its results, refer to the GitHub documentation
    and the CodeQL documentation.
    
    Signed-off-by: Brian <[email protected]>
    Signed-off-by: victoryang00 <[email protected]>
    b4yuan authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    e55babb View commit details
    Browse the repository at this point in the history
  9. CodeQL: Add more build combinations and disable run on PR (bytecodeal…

    …liance#3246)
    
    Enhance CodeQL Code Security Analysis:
    - Add more compilation combinations to build iwasm with different kinds of features
    - Disable run on PR created and keep nightly run, since the whole time is very long,
       and will check how to restore run on PR created in the future
    
    Signed-off-by: victoryang00 <[email protected]>
    wenyongh authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    90a5976 View commit details
    Browse the repository at this point in the history
  10. Implement memory64 for classic interpreter (bytecodealliance#3266)

    Adding a new cmake flag (cache variable) `WAMR_BUILD_MEMORY64` to enable
    the memory64 feature, it can only be enabled on the 64-bit platform/target and
    can only use software boundary check. And when it is enabled, it can support both
    i32 and i64 linear memory types. The main modifications are:
    
    - wasm loader & mini-loader: loading and bytecode validating process
    - wasm runtime: memory instantiating process
    - classic-interpreter: wasm code executing process
    - Support memory64 memory in related runtime APIs
    - Modify main function type check when it's memory64 wasm file
    - Modify `wasm_runtime_invoke_native` and `wasm_runtime_invoke_native_raw` to
      handle registered native function pointer argument when memory64 is enabled
    - memory64 classic-interpreter spec test in `test_wamr.sh` and in CI
    
    Currently, it supports memory64 memory wasm file that uses core spec
    (including bulk memory proposal) opcodes and threads opcodes.
    
    ps.
    bytecodealliance#3091
    bytecodealliance#3240
    bytecodealliance#3260
    
    Signed-off-by: victoryang00 <[email protected]>
    wenyongh authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    de59190 View commit details
    Browse the repository at this point in the history
  11. Revert "lldb_function_to_function_dbi: A hack to avoid crashing on C+…

    …+ methods (bytecodealliance#3190)" (bytecodealliance#3281)
    
    This reverts commit 0e8d949.
    
    Because it doesn't make much sense anymore after we disabled debug info
    processing on C++ functions in:
    "aot debug: process lldb_function_to_function_dbi only for C".
    
    Signed-off-by: victoryang00 <[email protected]>
    yamt authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    96ffe8f View commit details
    Browse the repository at this point in the history
  12. Enhance wasm loading with LoadArgs and support module names (bytecode…

    …alliance#3265)
    
    - Add new API wasm_runtime_load_ex() in wasm_export.h
      and wasm_module_new_ex in wasm_c_api.h
    - Put aot_create_perf_map() into a separated file aot_perf_map.c
    - In perf.map, function names include user specified module name
    - Enhance the script to help flamegraph generations
    
    Signed-off-by: victoryang00 <[email protected]>
    lum1n0us authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    6956e0a View commit details
    Browse the repository at this point in the history
  13. thread mgr: Free aux stack only when it was allocated (bytecodeallian…

    …ce#3282)
    
    When thread manager is enabled, the aux stack of exec_env may be allocated
    by wasm_cluster_allocate_aux_stack or disabled by setting aux_stack_bottom
    as UINTPTR_MAX directly. For the latter, no need to free it.
    
    And fix an issue when paring `--gc-heap-size=n` argument for iwasm, and
    fix a variable shadowed warning in fast-jit.
    
    Signed-off-by: victoryang00 <[email protected]>
    wenyongh authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    ce83313 View commit details
    Browse the repository at this point in the history
  14. Update version number to 2.0.0 and update release notes (bytecodealli…

    …ance#3327)
    
    And enable code format check for wasm_export.h and add '\n' in os_printf
    in sgx platform source files.
    
    Signed-off-by: victoryang00 <[email protected]>
    wenyongh authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    5839425 View commit details
    Browse the repository at this point in the history
  15. Add wasm_runtime_detect_native_stack_overflow_size (bytecodealliance#…

    …3355)
    
    - Add a few API (bytecodealliance#3325)
       ```c
       wasm_runtime_detect_native_stack_overflow_size
       wasm_runtime_detect_native_stack_overflow
       ```
    - Adapt the runtime to use them
    - Adapt samples/native-stack-overflow to use them
    - Add a few missing overflow checks in the interpreters
    - Build and run the sample on the CI
    
    Signed-off-by: victoryang00 <[email protected]>
    yamt authored and victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    1acd123 View commit details
    Browse the repository at this point in the history
  16. Enhance wasm with checkpoint and restore support (bytecodealliance#2333)

    - Add wasm_runtime_checkpoint/wasm_runtime_restore API
    - Support AOT and Classic Interpreter mode checkpoint and debug through OS signal, tested on windows/mac/linux aarch64/x64
    - Static instrument the AOT to have the checkpoint and restore switches
    - Add sub extra library folder for implementing the ckpt-restore
    - Include extra dependency of yalantinglib
    
    Co-authored-by: Aibo Hu <[email protected]>
    Co-authored-by: kikispace <[email protected]>
    Co-authored-by: Brian Zhao <[email protected]>
    Signed-off-by: victoryang00 <[email protected]>
    4 people committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    6952a12 View commit details
    Browse the repository at this point in the history
  17. Merge branch 'dev/checkpoint_and_restore' into main

    Signed-off-by: victoryang00 <[email protected]>
    victoryang00 committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    9096c84 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    beb9858 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. fix staging error

    victoryang00 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    b7a26c2 View commit details
    Browse the repository at this point in the history