diff --git a/docs/Crashes.md b/docs/Crashes.md new file mode 100644 index 0000000000000..3d20b20b85fd5 --- /dev/null +++ b/docs/Crashes.md @@ -0,0 +1,134 @@ +## Symbolicating stack traces for engine crashes + +The easiest way to symbolicate stack traces for Android and iOS is running the [dart_ci symbolizer locally](https://github.com/dart-lang/dart_ci/blob/main/github-label-notifier/symbolizer/README.md#using-this-locally). If that is not an option, the steps below explain how to do it manually. + +### Android + +#### Get the symbols + +1. Get the Flutter Framework or Flutter Engine revision from the report. If you have the Engine revision, skip to step 3. + +2. Get the Engine revision from the Framework (this could be automated). https://github.com/flutter/flutter/blob/main/bin/internal/engine.version is the file which contains the information. Substitute the framework hash for `main` in that url. + +3. With the full engine revision (e.g. cea5ed2b9be42a981eac762af3664e4a17d0a53f), you can now get the proper symbol files: + + To view the available artifacts for a build, visit this URL in a browser (replacing the engine hash with your hash): `https://console.cloud.google.com/storage/browser/flutter_infra_release/flutter/cea5ed2b9be42a981eac762af3664e4a17d0a53f` + + To download the symbols for android-arm, download this URL _using your browser_ (replacing the hash again, and noting that this URL is on a different host, "storage", compared to the one above, which uses "console"): `https://storage.cloud.google.com/flutter_infra_release/flutter/cea5ed2b9be42a981eac762af3664e4a17d0a53f/android-arm/symbols.zip`. + + Please be aware that the type of the symbol must match your Apps release type. In above example, this refers to a android-arm **debug** build. If you work with a **release** or **profile** build, the URLs would look like this: + + * https://storage.cloud.google.com/flutter_infra_release/flutter/cea5ed2b9be42a981eac762af3664e4a17d0a53f/android-arm-release/symbols.zip + * https://storage.cloud.google.com/flutter_infra_release/flutter/cea5ed2b9be42a981eac762af3664e4a17d0a53f/android-arm-profile/symbols.zip + + You have to use your browser because it does authentication. + +#### Symbolicate with ndk-stack + +Once you have the symbols unzipped, you can use ndk-stack from your Android NDK. Suppose `stack.txt` contains the stack (including the leading `*** *** ***` line from the crash): + +```bash +.../ndk/prebuilt/linux-x86_64/bin/ndk-stack -sym .../path/to/downloaded/symbols < stack.txt +``` + +Or on macOS: +```bash +.../ndk/prebuilt/darwin-x86_64/bin/ndk-stack -sym .../path/to/downloaded/symbols < stack.txt +``` + +Some debugging tools, like _pidcat_ may not show the full tombstone logs. In that case, please use `adb logcat` directly and copy the full output. + +#### Symbolicate with addr2line + +A alternative way to symbolicate is by using the addr2line tool. It is bundled with the NDK. +To use it, simply call it with a path to the .so file downloaded above and feed the stack addresses to it manually. For example, on macOS: + +``` +% $ANDROID_HOME/ndk/20.0.5594570/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android-addr2line -e ~/Downloads/libflutter.so +``` +_The tool is now awaiting your input, so let's feed it a memory address_: +``` +0x00000000006a26ec +/b/s/w/ir/cache/builder/src/out/android_release_arm64/../../third_party/dart/runtime/vm/dart_api_impl.cc:1366 +``` +This revealed address `0x00000000006a26ec` to correspond with `dart_api_impl.cc:1366`. + +#### Making sure you got the right libflutter.so + +The build system sets a build id for each `libflutter.so` file. In the tombstones, you would see the ID like so: + +``` +#00 pc 000000000062d6e0 /data/app/com.app-tARy3eLH2Y-QN8J0d0WFog==/lib/arm64/libflutter.so!libflutter.so (offset 0x270000) (BuildId: 34ad5bdf0830d77a) +``` + +This equals to a build id of **34ad5bdf0830d77a**. The `libflutter.so` debug files downloaded as shown above could be verified using the `file` command: + +``` +% file ~/Downloads/libflutter.so +/Users/user/Downloads/libflutter.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[xxHash]=34ad5bdf0830d77a, with debug_info, not stripped +``` + +Ensure the build IDs match, else you will not be able to symbolicate. + +#### Expanding Git Revisions + +Go to a commit page with the short commit as the last fragment of the URL: +(e.g. https://github.com/flutter/flutter/commit/9cb914df1 or https://github.com/flutter/engine/commit/2a13567) and then find the full revision on the page. + + +#### Symbolicating local builds + +If you have made your own builds, you can use ndk-stack directly: + +```bash +# dev/engine is where your engine's .gclient file is +# android_debug_unopt is whatever build of your engine you are using +adb logcat | ~/dev/engine/src/third_party/android_tools/ndk/prebuilt/linux-x86_64/bin/ndk-stack -sym ~/dev/engine/src/out/android_debug_unopt +``` + +### iOS + +The dSYM file for `Flutter.framework` (which is the Flutter Engine) for ios-release builds can be downloaded from Google Cloud Storage. Follow the steps from the Android section in this guide, but in the last step use a download url following this schema: `https://storage.cloud.google.com/flutter_infra_release/flutter/38a646e14cc25f5a56a989c6a5787bf74e0ea386/ios-release/Flutter.dSYM.zip` (replace the engine hash with your hash). + +#### Symbolicating local builds + +If you built your local engine in debug or profile Dart modes, the framework's dylib's symbols aren't stripped and are available by default. + +#### Crashes in Dart AOT code + +If the crash is in AOT Dart code (in `--release` or `--profile` builds) on iOS, and you can build your own engine, these steps will be helpful for the VM team to fix the bug: + +* Prepare a reduced test case. +* Compile the engine in profile mode and disable optimizations for symbolicated traces. + * `sky/tools/gn --ios --unopt --runtime-mode profile; ninja -C out/ios_profile_unopt -j800`. +* Launch the application via the Xcode project and make it crash in the debugger. +* File a bug on [dart-lang/sdk](https://github.com/dart-lang/sdk/issues/new). +* Dump the register state and paste it into the bug. + * In `lldb`, `register read`. +* Copy the backtrace and paste it into the bug. + * In `lldb`, `thread backtrace`. Assumes you are on the thread that crashed. If not, `thread select n`. +* Disassemble the last frame and paste it into the bug. + * In `lldb`, `frame select 0` then `disassemble --frame`. +* Disassemble using the `gen_snapshot` and paste the function into the bug for more detailed information. + * In the backtrace, look for the name of the precompiled function that caused the crash. + * Open `SnapshotterInvoke` from Xcode and to the `RunCommand ... Snapshotter` call, add the `--disassemble` flags. + * Modify the `RunCommand` function to dump to a file. + * Build again. The results should end up in the file. + * Look for the function name (by substring match) in this file and copy out that information to the bug. +* Ping someone on dart-lang/sdk. + + +### Android Local Engine Builds + +When running with a local engine build, the symbolization workflow can be cumbersome and unecessary. Instead, it is possible to build the engine itself with symbols and disable Gradle's automatic symbol stripping. This is also required to see symbol names in Android Studio CPU profiles. + +1. In the android engine configuration, provide a --no-stripped argument to gn. For example: `gn --android --android-cpu=arm64 --unopt --no-stripped` + +2. In the flutter project file `android/app/build.gradle`, add the following line under the `android` block: + +``` + packagingOptions{ + doNotStrip "**/*.so" + } + +``` diff --git a/docs/Custom-Flutter-Engine-Embedders.md b/docs/Custom-Flutter-Engine-Embedders.md new file mode 100644 index 0000000000000..c4e965b9b0804 --- /dev/null +++ b/docs/Custom-Flutter-Engine-Embedders.md @@ -0,0 +1,23 @@ +The Flutter Engine is window toolkit agnostic. If you want to build Flutter embedders on one of the platforms not supported out of the box (i.e, iOS & Android), this page is for you. + +This is a very low level API and is not suitable for beginners. + +* The window toolkit agnostic component of the Flutter engine is available as a dynamic library in the `flutter_engine` GN target in [`//shell/platform/embedder:flutter_engine`](https://github.com/flutter/engine/blob/080fbcb1759e5916f0d6cdcdfd945c053320e6b3/shell/platform/embedder/BUILD.gn#L51). +* That target must be built as part of the host GN build. Such builds are already available for desktop Linux & Mac. If you want to target another platform, you will have to configure a GN toolchain for the same. +* You may build this target yourself or download the same artifacts uploaded by the buildbots on each commit. + * The Mac buildbot uploads the artifacts to a known location. Access it here [`https://storage.googleapis.com/flutter_infra_release/flutter/FLUTTER_ENGINE/darwin-x64/FlutterEmbedder.framework.zip`](https://storage.googleapis.com/flutter_infra_release/flutter/080fbcb1759e5916f0d6cdcdfd945c053320e6b3/darwin-x64/FlutterEmbedder.framework.zip). + * Replace `FLUTTER_ENGINE` with the SHA of the Flutter engine you wish to use. + * The Linux buildbot uploads the artifacts to a known location. Access it here [`https://storage.googleapis.com/flutter_infra_release/flutter/FLUTTER_ENGINE/linux-x64/linux-x64-embedder`](https://storage.googleapis.com/flutter_infra_release/flutter/080fbcb1759e5916f0d6cdcdfd945c053320e6b3/linux-x64/linux-x64-embedder) + * Replace `FLUTTER_ENGINE` with the SHA of the Flutter engine you wish to use. + * The binary is not stripped and contains debug information. Embedders are advised to strip the binary before deployment. + * The Windows buildbot uploads the artifacts to a known location. Access it here [`https://storage.googleapis.com/flutter_infra_release/flutter/FLUTTER_ENGINE/windows-x64/windows-x64-embedder.zip`](https://storage.googleapis.com/flutter_infra_release/flutter/080fbcb1759e5916f0d6cdcdfd945c053320e6b3/windows-x64/windows-x64-embedder.zip) + * Replace `FLUTTER_ENGINE` with the SHA of the Flutter engine you wish to use. + * You can also obtain that SHA from the [`engine.version`](https://github.com/flutter/flutter/blob/main/bin/internal/engine.version) file in your Flutter framework checkout. This allows you to exactly match the engine version with the Flutter framework version. +* The Flutter engine API has no platform specific dependencies, has a stable ABI and is available in its entirety in a [single C header file available here](https://github.com/flutter/engine/blob/main/shell/platform/embedder/embedder.h). +* To use as a guide, you may use [this example embedder that uses GLFW](https://github.com/flutter/engine/blob/main/examples/glfw/FlutterEmbedderGLFW.cc) for window management and rendering. + +While we do not object to teams creating custom builds of the Flutter engine for their purposes, we do not support this configuration. Not supporting it means that we do not commit to any timelines for fixing bugs that may come up in such a configuration, even for customers for which we would usually be willing to make commitments (see the [Issue Hygiene](../contributing/issue_hygiene/README.md) page). It also means that we encourage teams to view such configurations as short-term solutions only and encourage teams to transition away from such configurations at the earliest possible opportunity. + +We do not expect custom engine builds to be long-term sustainable. They are not supported on any platform where we plan to be the publisher of a Flutter runtime distinct from the applications that run on the runtime, and they require significant effort to port to our new target platforms such as Web and desktop. There is also an expensive maintenance burden (for example, if we add new features, a custom engine build would need to be updated to support that feature). + +We would generally recommend using custom engine builds only when porting Flutter to platforms that are not supported out of the box, for example in embedded hardware. \ No newline at end of file diff --git a/docs/Custom-Flutter-Engine-Embedding-in-AOT-Mode.md b/docs/Custom-Flutter-Engine-Embedding-in-AOT-Mode.md new file mode 100644 index 0000000000000..64f181c9a9cc1 --- /dev/null +++ b/docs/Custom-Flutter-Engine-Embedding-in-AOT-Mode.md @@ -0,0 +1,132 @@ +This document is intended for developers of third party embedders who wish to package and ship their Flutter applications for AOT mode operation. A third party embedder uses [the stable C embedder API](https://github.com/flutter/engine/blob/869d9f528503778be1e5ab27ba53502f0cb20de2/shell/platform/embedder/embedder.h) to embed Flutter applications on their platform. + +## Building an AOT Flutter Engine + +By default, the Flutter engine packaged for embedders assumes that the mode of operation is JIT. An engine configured for JIT packages a VM that is incompatible with AOT snapshots. Build an AOT engine using the following invocation. This is also the invocation where custom target, sysroot and toolchain selection flags can be specified. + +``` +./flutter/tools/gn --runtime-mode release +``` + +**Note:** Throughout this document, the `gn` flags mentioned work fine with other non-runtime mode or configuration selection options (stuff like `--no-lto`,` --unoptimized`, etc..). Such options are particularly useful during development and their use is highly encouraged. + +This will produce a Flutter engine configured for AOT mode targeting the host. Note that we are repurposing Flutter’s “release” mode policy to prepare an AOT engine. The application of this policy to third party embedders is optional and a decision the authors of such embedders need to make themselves. + +## Building the Architecture Specific `gen_snapshot` + +The binary that converts Dart code to architecture specific AOT instructions is called `gen_snapshot`. A successful invocation of `gen_snapshot` should produce four binary blobs. These are the Dart VM heap and instructions snapshots as well as the isolate heap and instruction snapshots. Refer to the [wiki article on engine operation in AOT mode](Flutter-engine-operation-in-AOT-Mode.md) for the purpose of these snapshots. + +A specific `gen_snapshot` can only produce AOT instructions for one target architecture. To verify that the `gen_snapshot` you have is producing instructions for the architecture you care about, invoke the same with the `--version` flag. It should produce something like the following. + +``` +Dart VM version: 2.1.1-dev.2.0.flutter-ac1bf656c4 (Thu Jan 17 16:55:19 2019 +0000) on "macos_x64" +``` + +The final string denotes the host/target pair. In the case of the example above, the host is `macos` and the target `x64`. An example of a `gen_snapshot` that produces instructions for `aarch64` would read “macos_simarm64” (the author is on MacOS X). + +It is easiest to just pick the supported Flutter target whose architecture is closest to the one of the embedder. For example, if the target is `armv7`, the following invocation will generate a `gen_snapshot` that is suitably configured for that target. + +``` +./flutter/tools/gn --android --runtime-mode release +``` + +Another useful invocation for aarch64 AOT targeted `gen_snapshot` is: + +``` +./flutter/tools/gn --android --runtime-mode release --android-cpu arm64 +``` + +**Important:** Calling convention and alignment must be the same (in addition to just the target architecture) for the AOT instructions generated by `gen_snapshot` and the target. Repurposing Flutter’s build targets makes sure that all those subtleties are taken care of. But it is still up to the embedder to pick and match the AOT instructions. For example, the `--android` --android-cpu arm64`configured`gen_snapshot`will not generate completely compatible instructions for iOS on`aarch64` (even though the target architectures are the same). A mismatch between the snapshot's architecture/ABI and the device's architecture/ABI will be detected when the snapshot is loaded and clearly reported. + +## Building the AOT Snapshot + +There are two separate steps involved in the generation of AOT instructions for the target architecture. First, a target architecture agnostic kernel snapshot (~AST) of the Flutter Dart application needs to be generated. Then, this snapshot is given to `gen_snaphost` which generates the AOT snapshot in the form of the four blobs (~machine code) listed above (and detailed on the wiki page). + +### The Short and Easy Way + +Both iOS and Android AOT modes need to do this step when they build their artifacts. So, if the embedder targets are similar, the workflow supported by the Flutter tooling can be repurposed for custom AOT embedders. For example, in the case of `aarch64` AOT instructions for Android like targets, the following instructions will generated the four AOT blobs in the intermediates. + +```bash +flutter --local-engine --local-engine-host build aot --target-platform android-arm64 --release +``` + +**Note:** The `--local-engine` (and `--local-engine-host`) flag is technically optional. However, not specifying the flag will make the tools pick the released version on of the Flutter engine. This version may contain subtle version mismatches with the engine you are using to prepare the `gen_snapshot` binary. So it is safer to just make sure the version are the same. +See [running with a local engine](Debugging-the-engine.md#running-a-flutter-app-with-a-local-engine) for more details. + +The result of the invocation will be the generation of the following binary blobs in the `build/aot directory`: + +- `vm_snapshot_data`: The VM snapshot data. +- `vm_snapshot_instr`: The VM snapshot instructions. +- `isolate_snapshot_data`: The Isolate snapshot data. +- `isolate_snapshot_instr`: The isolate snapshot instructions. + +### The Hard Way + +To generate the four AOT snapshot blobs directly, you will have to generate the kernel snapshot and then prepare the AOT snapshot manually. The exact flags to use are rather esoteric but self explanatory. And, when necessary, the `-v` flag can be passed to the `flutter build aot` instruction to dump the exact flags used by Flutter. These can then be modified as necessary for the target architecture. + +#### Generating the Kernel Snapshot + +The following invocation will generate a file called `kernel_snapshot.dill` in the build directory. Make sure you run `flutter packages get` in your project first to fetch all package dependencies. + +``` +$FLUTTER_ENGINE_OUT_DIR/dart \ + $FLUTTER_ENGINE_OUT_DIR/frontend_server.dart.snapshot \ + --sdk-root $FLUTTER_ENGINE_OUT_DIR/flutter_patched_sdk/ \ + --strong \ + --target=flutter \ + --aot \ + --tfa \ + -Ddart.vm.product=true \ + --packages .packages \ + --output-dill build/kernel_snapshot.dill \ + package:flutter_gallery/main.dart +``` + +#### Generating the AOT Snapshot + +Once the `kernel_snapshot.dill` file has been obtained, `gen_snapshot` can be invoked with the following arguments to generate the four blobs that form the AOT snapshot. + +``` +$FLUTTER_ENGINE_OUT_DIR/gen_snapshot \ + --causal_async_stacks \ + --packages=.packages \ + --deterministic \ + --snapshot_kind=app-aot-blobs \ + --vm_snapshot_data=build/vm_snapshot_data \ + --isolate_snapshot_data=build/isolate_snapshot_data \ + --vm_snapshot_instructions=build/vm_snapshot_instr \ + --isolate_snapshot_instructions=build/isolate_snapshot_instr \ + --no-sim-use-hardfp \ + --no-use-integer-division \ + build/kernel_snapshot.dill +``` + +**Note:** The `-no*` flags in the invocation above may not be necessary for all targets and can be skipped. As always, when in doubt, call `flutter build aot -v` after selecting the most similar target and see the flags used by Flutter. + +## Packaging the AOT Blobs + +The four AOT blobs need to be shipped with the application and are necessary for the `FlutterEngineRun` call. Embedders have to make a decision about how best to package and ship these blobs to the target. + +On the target at runtime, these blobs need to be mapped into the address space with the following restrictions: + +- `vm_snapshot_data`: Read-Only. +- `vm_snapshot_instr`: Read-Execute. +- `isolate_snapshot_data`: Read-Only. +- `isolate_snapshot_instr`: Read-Execute. + +The responsibility of keeping the mappings alive is upto the embedder. The mappings must be maintained as long as the `FlutterEngine` is running and alive. + +## Configuring the Engine for AOT Operation + +In the `FlutterProjectArgs` struct given the `FlutterEngineRun` call, provide the following options: + +- `vm_snapshot_data`: Pointer to the read-only VM snapshot mapping. +- `vm_snapshot_data_size`: Size of the VM snapshot mapping. +- `vm_snapshot_instructions`: Pointer to the read-execute VM instructions mapping. +- `vm_snapshot_instructions_size`: Size of the VM instructions mapping. +- `isolate_snapshot_data`: Pointer to the read-only isolate snapshot mapping. +- `isolate_snapshot_data_size`: Size of the isolate snapshot mapping. +- `isolate_snapshot_instructions`: Pointer to the read-execute isolate instructions mapping. +- `isolate_snapshot_instructions_size`: Size of the isolate instructions mapping. + +Flutter Engine is now running in AOT mode! diff --git a/docs/Debugging-the-engine.md b/docs/Debugging-the-engine.md new file mode 100644 index 0000000000000..7dad334547035 --- /dev/null +++ b/docs/Debugging-the-engine.md @@ -0,0 +1,147 @@ +This page has some hints about debugging the engine. + +See also [Crashes](Crashes.md) for advice on handling engine crashes (specifically around obtaining stack traces, and reporting crashes in AOT Dart code). + +## Running a Flutter app with a local engine + +First, make sure the appropriate version of the engine is built (see [Compiling the engine](./contributing/Compiling-the-engine.md)). + +### Using the Flutter tool + +Run your Flutter app with: + +```bash +flutter run --local-engine=XXXX --local-engine-host=YYYY +``` + +to run an app with the local engine where `XXXX` should be replaced with the version you wish to use. For example, use `--local-engine=android_debug_unopt --local-engine-host=host_debug_unopt` to run a debug android engine or `--local-engine=ios_debug_sim_unopt --local-engine-host=host_debug_unopt` to run a debug iOS simulator engine. + +> 💡 **TIP**: When developing on a Mac with ARM (M CPU), use `--local-engine-host=host_debug_unopt_arm64`. +> +> You can continue to use `host_debug_unopt` (required for Intel Macs), but the engine will be run under Rosetta +> which may be slower. See [Developing with Flutter on Apple Silicon](../platforms/desktop/macos/Developing-with-Flutter-on-Apple-Silicon.md) +> for more information. + + +It is important to always have a `host_XXXX` version of the engine built when using a local engine since Flutter uses the host build's version of Dart. + +### Using Visual Studio Code + +You will need to add a new [launch configuration](https://code.visualstudio.com/docs/editor/debugging#_launch-configurations) in the `launch.json` file: + +```json +{ + "version": "0.2.0", + "configurations": [ + { + "name": "Launch (local engine)", + "request": "launch", + "type": "dart", + "args": ["--local-engine", "XXX", "--local-engine-host", "YYY"] + } + + // Other profiles below.. + ] +} +``` + +## Bisecting a roll failure + +If the engine roll is failing (see [Autorollers](../infra/Autorollers.md)), you can use `git bisect` on the engine repo to track down the offending commit, using the `--local-engine` and `--local-engine-host` arguments as described above to run the failing framework test with each version of the engine. + +## Tracing OpenGL calls in Skia + +All OpenGL calls in Skia are guarded by either the `GR_GL_CALL_NOERRCHECK` or `GR_GL_CALL_RET_NOERRCHECK` macros. Trace events may be added in these macros to trace all GL calls made by Skia, for example [in a patch like this](https://gist.github.com/chinmaygarde/607eb86d5447615b9cf2804a4f8fb1ce). + +Due to the number of events traced to the timeline, the trace buffer may be filled up very quickly. Unless you want to see only the traces for the past few frames, use an endless trace buffer (`flutter run --endless-trace-buffer` turns on an endless trace buffer). + +Also, make sure to run your application with the `--trace-skia` flag. + +## Debugging iOS builds with Xcode + +Building with `flutter --local-engine` will set a `LOCAL_ENGINE` Xcode build setting in your Flutter application's `ios/Flutter/Generated.xcconfig` file. This will be set until you run `flutter run` again with either a different `--local-engine` option, or with none at all (which will unset it). + +You can speed up your workflow by adding the `--config-only` flag to set up the Xcode build settings and plugins, but not compile the app. For example: + +```bash +flutter build ios --local-engine ios_debug_unopt --local-engine-host host_debug_unopt --config-only +``` + +To start debugging, open your Flutter app `ios/Runner.xcworkspace` file in Xcode. Ensure **Product > Scheme > Edit Scheme > Run > Build Configuration** matches your engine runtime mode (defaults to `Debug`). + +Product > Scheme > Edit Scheme > Run > Build Configuration + +Add an engine symbol breakpoint via **Debug > Breakpoints > Create Symbolic Breakpoint...**. The **Symbol** field should be the engine symbol you're interested in, like `-[FlutterEngine runWithEntrypoint:]` (note the `-[` prefix has no space). + +You can also set a breakpoint directly with [lldb](https://lldb.llvm.org/tutorial.html) by expanding **Flutter > Runner** in the Runner Project Navigator. Put a breakpoint in `AppDelegate.swift`'s `application(didFinishLaunchingWithOptions:)` (Swift project) or `main.m`'s `main()` (Objective-C project) and start the application by clicking the Run button (CMD + R). Then, set your desired breakpoint in the engine in `lldb` via `breakpoint set -...`. + +## Debugging Android builds with gdb + +See https://github.com/flutter/engine/blob/main/sky/tools/flutter_gdb#L13 + +## Debugging native engine code on Android with Android Studio + +1. Build the local engine with the `--no-stripped` flag. +2. Decide on a Flutter app that you with to debug and run it with `flutter run` and the local engine flags. i.e.: `--debug --local-engine-src-path path/to/my/engine/src --local-engine=android_debug_unopt_arm64` +3. Open Android Studio and use `File > Profile or Debug APK`. The location of the debug build APK should be `build/app/outputs/apk/debug/app-debug.apk` under the Flutter app project. +4. To attach the debugger, use `Run > Attach Debugger to Android Process`. For "Use Android Debugger Settings from" choose `[Use default settings]`, and for "Debug Type" choose `Native Only`. +5. Once attached, you can use Android Studio to open local engine C++ source files and set breakpoints. + +## Debugging Windows builds with Visual Studio + +Compiling the engine creates a Visual Studio solution file. You can use it to debug the engine: + +1. Launch your Flutter app using a locally built engine `flutter run -d windows --local-engine host_debug_unopt --local-engine-host host_debug_unopt` +2. Using Visual Studio, open the engine's solution file `.\out\host_debug_unopt\all.sln` +3. Open `Debug` > `Attach to Process...` (or press `CTRL+ALT+P`) +4. Choose your Flutter app using either `Select Window`, or, the list of available processes. +5. Press the `Attach` button + +Building a Flutter app also creates a Visual Studio solution file. You can use it to debug the +engine, your app's runner, and your app's plugins: + +1. Build your Flutter app using a locally built engine using `flutter build windows --debug --local-engine host_debug_unopt --local-engine-host host_debug_unopt` +2. Using Visual Studio, open the Flutter app's `.\build\windows\.sln` +3. In the `Solution Explorer` pane, right click the project whose name matches your app, and select `Set as Startup Project` + + ![Set as Startup Project example](https://user-images.githubusercontent.com/737941/215009513-d31c59fd-1f54-44d9-a702-6dd3fdf71492.png) + +4. Now run your app by pressing `F5` or `DEBUG` > `Start Debugging`. This will launch your app with Visual Studio's debugger attached. + +Read this guide to [learn how to debug C++ using Visual Studio](https://learn.microsoft.com/visualstudio/debugger/getting-started-with-the-debugger-cpp?view=vs-2022). + +## Debugging with gdb on Linux + +Once you have built the engine, you'll find the unstripped libraries in `out/host_debug_unopt/lib.unstripped`, and the executables in `out/host_debug_unopt/exe.unstripped`. + +So, for instance, to run the unit tests under the debugger you would execute: + +```shell +flutter/tools/gn --runtime-mode=debug --unoptimized +ninja -C out/host_debug_unopt +gdb out/host_debug_unopt/exe.unstripped/flutter_linux_unittests +``` + +And then debug the test normally using GDB commands. + +To debug a Flutter app using GDB, the stripped flutter engine GTK library in the built application needs to be replaced with the unstripped one in the engine build output directory. + +First, in your Flutter project, build your Flutter app using the local engine: + +```shell +flutter build linux --debug --local-engine=host_debug_unopt --local-engine-host=host_debug_unopt lib/main.dart +``` + +Then, replace the library in your Flutter application's build directory: `build/linux/x64/debug/bundle/lib/libflutter_linux_gtk.so` with a copy or symbolic link to the engine build's output file `out/host_debug_unopt/lib.unstripped/libflutter_linux_gtk.so`. + +Then you can open it in the debugger with: + +```shell +gdb build/linux/x64/debug/bundle/your_app_name +``` + +Note that this won't help you debug the Dart portion of the app: this is just for debugging the engine code. If you need to simultaneously debug the Dart portion, you can connect to the observatory port given when you run the app in `gdb`. + +## Logging in the engine + +Flutter tool will by default parse out any non-error output from the engine. Error logs will be displayed. Logging is handled though the FML library's `logging.h`. diff --git a/docs/Engine-disk-footprint.md b/docs/Engine-disk-footprint.md new file mode 100644 index 0000000000000..0ce647128c755 --- /dev/null +++ b/docs/Engine-disk-footprint.md @@ -0,0 +1,33 @@ +## Treemaps + +For each commit to [flutter/engine](https://github.com/flutter/engine) the Chromebots generate treemaps illustrating the sizes of the individual components within release builds of `libflutter.so`. The treemap is uploaded to Google Cloud Storage and linked from the [LUCI](https://ci.chromium.org/p/flutter/g/engine/console) console: Select a "Linux aot" build and search for "Open Treemap". + +Alternatively, a link to a treemap can be constructed as follows: + +`https://storage.googleapis.com/flutter_infra_release/flutter///sizes/index.html` where: +* `` is the git hash from [flutter/engine](https://github.com/flutter/engine) for which you want the treemap, and +* `` can be any android release build, e.g. `android-arm-release` or `android-arm64-release`. + +## Benchmarks + +In [devicelab](https://github.com/flutter/flutter/tree/main/dev/devicelab) we run various benchmarks to track the APK/IPA sizes and various (engine) artifacts contained within. These benchmarks run for every commit to [flutter/flutter](https://github.com/flutter/flutter) and are visible on our [build dashboard](https://flutter-dashboard.appspot.com/). The most relevant benchmarks for engine size are: + +* APK/IPA size of Flutter Gallery + * Android: `flutter_gallery_android__compile/release_size_bytes` + * iOS: `flutter_gallery_ios__compile/release_size_bytes` +* APK/IPA size of minimal hello_world app + * Android: `hello_world_android__compile/release_size_bytes` + * iOS: `hello_world_ios__compile/release_size_bytes` +* Size of bundled `icudtl.dat` + * Compressed in APK: `hello_world_android__compile/icudtl_compressed_bytes` + * Uncompressed: `hello_world_android__compile/icudtl_uncompressed_bytes` +* Size of bundled `libflutter.so` (release mode) + * Compressed in APK: `hello_world_android__compile/libflutter_compressed_bytes` + * Uncompressed: `hello_world_android__compile/libflutter_uncompressed_bytes` +* Size of VM & isolate snapshots (data and instructions) + * Compressed in APK: `hello_world_android__compile/snapshot_compressed_bytes` + * Uncompressed: `hello_world_android__compile/snapshot_uncompressed_bytes` + +## Comparing AOT Snapshot Sizes + +A detailed comparison of AOT snapshot sizes can be performed using the [instructions documented here](./benchmarks/Comparing-AOT-Snapshot-Sizes.md). \ No newline at end of file diff --git a/docs/Engine-specific-Service-Protocol-extensions.md b/docs/Engine-specific-Service-Protocol-extensions.md new file mode 100644 index 0000000000000..73e5b28fcd623 --- /dev/null +++ b/docs/Engine-specific-Service-Protocol-extensions.md @@ -0,0 +1,203 @@ +The Flutter engine adds several extensions to the [Dart VM Service Protocol](https://github.com/dart-lang/sdk/blob/main/runtime/vm/service/service.md). The Flutter Engine specific extensions are documented here. Applications may also choose to register their [own extensions](https://api.dartlang.org/stable/1.24.3/dart-developer/registerExtension.html). + +## List views: `_flutter.listViews` + +Tooling requests this very early in the application lifecycle to ask for the details of the root isolate. + +No arguments. + +Response: + +```json +{ + "type": "FlutterViewList", + "views": [ + { + "type": "FlutterView", + "id": "_flutterView/0x1066096d8", + "isolate": { + "type": "@Isolate", + "fixedId": true, + "id": "isolates/453229818", + "name": "main.dart$main-453229818", + "number": 453229818 + } + } + ] +} +``` + +## Isolate Restart or Cold Reload: `_flutter.runInView` + +The IDE has requested (or the user pressed ‘R’ from the ‘flutter run’ interactive console) a cold reload. For example, this happens when the user presses the green play button. + +Used to "cold reload" a running application where the shell (along with the platform view and its rasterizer bindings) remains the same but the root isolate is torn down and restarted with the new configuration. Only used in the development workflow. + +The previous root isolate is killed and its identifier invalidated after this call. Callers can query the new isolate identifier using the `_flutter.listViews` method. + +Four arguments: + +``` +viewId = _flutterView/0x14ba08c68 +mainScript = /path/to/application/lib/main.dart +packagesFile = /path/to/application/.packages +assetDirectory = /path/to/application/build/flutter_assets +``` + +Response: + +```json +{ + "type": "Success", + "view": { + "type": "FlutterView", + "id": "_flutterView/0x104e0ab58", + "isolate": { + "type": "@Isolate", + "fixedId": true, + "id": "isolates/1056589762", + "name": "main.dart$main-1056589762", + "number": 1056589762 + } + } +} +``` + +The object in the **view** key is constructed in the same way as the views in the **List Views** method. + + +## Flush UI thread tasks: `_flutter.flushUIThreadTasks` + +Does nothing but waits for all pending tasks on the UI thread to be completed before returning success to the service protocol caller. + +No arguments. + +Response: + +```json +{"type": "Success"} +``` + +## Get screenshot of view as PNG: `_flutter.screenshot` + +Get the screenshot as PNG of a random Flutter view on the device. The screenshot data will be base64 encoded in the response body. + +No arguments. + +Response: + +```json +{ + "type": "Screenshot", + "screenshot": "" +} +``` + +## Get screenshot of view as Skia picture: `_flutter.screenshotSkp` + +Get the Skia SKP of a random Flutter view on the device. The SKP data will be base64 encoded in the response body. + +No arguments. + +Response: + +```json +{ + "type": "ScreenshotSkp", + "skp": "" +} +``` + +## Update asset bundle path: `_flutter.setAssetBundlePath` + +In case of a hot-reload, the service protocol handles source code updates. However, there may be changes to assets. The DevFS updates assets in an separate directory that needs to be used by the engine. + +Two arguments: + +``` +viewId = _flutterView/0x15bf057f8 +assetDirectory = /path/to/flutter_assets +``` + +Response: + +```json +{ + "type": "Success", + "view": { + "type": "FlutterView", + "id": "_flutterView/0x104e0ab58", + "isolate": { + "type": "@Isolate", + "fixedId": true, + "id": "isolates/1056589762", + "name": "main.dart$main-1056589762", + "number": 1056589762 + } + } +} +``` + +The object in the **view** key is constructed in the same way as the views in the **List Views** method. + +## Get the display refresh rate: `_flutter.getDisplayRefreshRate` + +Get the display refresh rate of the actual device that runs the Flutter view. For example, most devices would return an fps of 60, while iPad Pro would return an fps of 120. + +One argument: + +``` +viewId = _flutterView/0x15bf057f8 +``` + +Response: + +```json +{ + "type": "DisplayRefreshRate", + "fps": 60.0 +} +``` + +## Get Skia SkSL shader artifacts: `_flutter.getSkSLs` + +Get Skia SkSL shader artifacts from an actual device that runs the Flutter view. Such artifacts can be used to warm up shader compilations and avoid jank. One has to first tell Flutter to prepare SkSL shaders by `flutter run --cache-sksl` or `flutter drive --cache-sksl`, and trigger some shader compilations by going through some animations/transitions. Otherwise, this service protocol extension may return an empty set of SkSLs. + +The key of the returned `SkSLs` map will be Base32 encoded. It should be used directly as the filename of the shader artifact. The value in that map is the Base64 encoded SkSL shader. Once decoded, it should be the content of the shader artifact file. + +One argument: + +``` +viewId = _flutterView/0x15bf057f8 +``` + +Response: + +```json +{ + "type": "GetSkSLs", + "SkSLs": { + "CAZAAAACAAAAAAAAAAABGAABAAOAAFAADQAAGAAQABSQAAAAAAAAAAAAAABAAAAAEAAGGAA": "eQ==" + } +} +``` + +## Estimate raster cache memory usage: `_flutter.estimateRasterCacheMemory` + +Estimate the memory usage of both picture and layer raster cache. For each picture or layer cached, there is a rasterized `SkImage` of that picture or layer to speed up future draws. Only `SkImage`'s memory usage is counted as other objects in the cache system are often much smaller compared to `SkImage`. Function `SkImageInfo::computeMinByteSize` is used to estimate the `SkImage` memory usage. + +One argument + + ``` +viewId = _flutterView/0x15bf057f8 +``` + +Response: + +```json +{ + "type": "EstimateRasterCacheMemory", + "layerBytes": 40000, + "pictureBytes": 400 +} +``` diff --git a/docs/Flutter's-modes.md b/docs/Flutter's-modes.md new file mode 100644 index 0000000000000..5300a72d573ff --- /dev/null +++ b/docs/Flutter's-modes.md @@ -0,0 +1,42 @@ +We aspire to reach a state where developers are able to use the following modes for running Flutter code. Our tools should not expose any other combinations of features or modes. Each mode corresponds to a separate build of the engine that we provide. + +1. **Debug** mode on device (including simulators, emulators): Turns on all the assertions in the world, includes all debugging information, enables all the debugger aids (e.g. observatory) and service extensions. Optimizes for fast develop/run cycles. Does not optimize for execution speed, binary size, or deployment. Used by `flutter run`. Built with `sky/tools/gn --android` or `sky/tools/gn --ios`. Also sometimes called "**checked** mode" or "**slow** mode". + +2. **Release** mode on device (excluding simulators, emulators): Turns off all assertions, strips as much debugging information as possible, turns off all the debugger tools. Optimizes for fast startup, fast execution, small package sizes. Disables any debugging aids. Disables service extensions. Intended for deployment to end-users. Used by `flutter run --release`. Built with `sky/tools/gn --android --runtime-mode=release` or `sky/tools/gn --ios --runtime-mode=release`. + +3. **Profile** mode on device (excluding simulators, emulators): Same as release mode except that profile-mode service extensions (like the one that turns on the performance overlay) is enabled, and tracing is enabled, as well as the minimum required to support using the tracing information (e.g. observatory can probably connect to the process). Used by `flutter run --profile`. Built with `sky/tools/gn --android --runtime-mode=profile` or `sky/tools/gn --ios --runtime-mode=profile`. Not available on simulators or emulators because profiling on simulators is not representative of real performance. + +4. Headless **test** mode on desktop: Same as debug mode except headless and for desktop platforms. Used by `flutter test`. Built with `sky/tools/gn`. + +In addition, for our purposes during development, each of the above should be able to be built in two modes: optimized, which is what end-developers use, and unoptimized, which is what we would use when debugging the engine. Optimized is the default, unoptimized engines are built by adding `--unoptimized` to the arguments. + + +### Artifact differences + +Debug mode produces a script snapshot, which is basically tokenized sources. Comments and whitespace are missing, literals are canonicalized. There is no machine code, tree-shaking or obfuscation. + +Profile and release modes produce app-aot snapshots, either as dylibs (iOS and Fuchsia) or 4-tuples of blobs (Android). Both include machine code for all the compiled functions, code metadata, method dictionaries, class and library structures, types, etc. The machine code is fully position-independent. The dylib additionally has DWARF debug info such as function names and source positions. There is tree-shaking. Obfuscation is opt-in. + +### Matrix + +The following axes, as described above, exist: + +* debug, release, profile +* opt, unopt +* iOS, Android, macOS, Linux, Windows + +In addition, some versions can select alternative graphics backends: + +* iOS can choose between: OpenGL, software +* Android can choose between: Vulkan, OpenGL, software +* macOS can choose between, OpenGL, software, headless (debug only) +* Linux can choose between: OpenGL, software, headless (debug only) +* Windows can choose between: OpenGL, software, headless (debug only) + +Separate from all the above, Fuchsia has the following modes: + +* AOT, JIT, interpreted DBC +* Observatory present, observatory absent +* opt, unopt + +In total therefore there are 3×2×(2+3+2+2+2) + 1×2×3 + 3×2×2 modes, which is 84 modes. diff --git a/docs/Flutter-engine-operation-in-AOT-Mode.md b/docs/Flutter-engine-operation-in-AOT-Mode.md new file mode 100644 index 0000000000000..e232ebf287419 --- /dev/null +++ b/docs/Flutter-engine-operation-in-AOT-Mode.md @@ -0,0 +1,75 @@ +# Overview + +The Flutter Engine in AOT mode requires four artifacts to run any given isolate. These are: + +* **Dart VM Snapshot** + * Represents the initial state of the Dart heap shared between isolates. + * Helps launch Dart isolates faster. + * Does not contain any isolate specific information. + * Mostly predefined Dart strings used by the VM + * Should live in the data segment. + * From the VM's perspective, this just needs to be loaded in memory with READ permissions and does not need WRITE or EXECUTE permissions. Practically this means it should end up in rodata when putting the snapshot in a shared library. +* **Dart VM Instructions** + * Contains AOT instructions for common routines shared between all Dart isolates in the VM. + * This snapshot is typically extremely small and mostly contains stubs. + * Must live in the text segment. +* **Isolate Snapshot** + * Represents the initial state of the Dart heap and includes isolate specific information. + * Along with the VM snapshot, helps in faster launches of the specific isolate. + * Should live in the data segment. +* **Isolate Instructions** + * Contains the AOT code that is executed by the Dart isolate. + * Must live in the text segment. + +The VM snapshot and instructions can be shared between all isolates. These must be available during the initialization of the Dart VM. The Dart VM is initialized when the first Flutter view initializes an instance of the Flutter shell. The Flutter shell manages thread safe initialization of the Dart VM. Once the Dart VM is initialized, multiple instances of the Flutter view reference the same VM to run their isolates. There can only be one VM running in the process at any given time. + +Given this configuration, launching each root isolate (for a Flutter application) requires two artifacts along with two other artifacts that are shared between all the other isolates. + +Isolates launched by Dart code (e.g., Isolate.spawn) inherit the snapshots of their parents. + +# Snapshot Generation + +These artifacts on all platforms are generated by the same binary on the host. This binary is shipped with Flutter tools and is called `gen_snapshot`. The way these artifacts are packed and referenced on the device differs however. + +## iOS Configuration + +`gen_snapshot` is invoked on the host by Xcode to generate the four binary blobs for each artifact. + +Using the native toolchain in Xcode, these artifacts are compiled into a framework bundle that is packaged into the application. This framework is typically called `App.framework` and is present in the `Frameworks/` folder of the application bundle. The name of the framework is configurable and if the embedder wishes to name it anything other than `App.framework`, the embedder can choose a custom name and specify the same in the `FLTLibraryPath` `Info.plist` key for the main application bundle. + +The Flutter engine, itself in a Framework called `Flutter.framework` will dynamically open the resolved application framework and look for four specific symbols in that library. These symbols are called `kDartVmSnapshotData`, `kDartVmSnapshotInstructions`, `kDartIsolateSnapshotData` and `kDartIsolateSnapshotInstructions`. These refer to the four snapshots mentioned above. + +Once snapshot resolution is finalized, the Flutter engine initializes the VM and launched the isolates. + +This configuration was chosen because the Flutter engine is not able to mark pages as executable at runtime. Packaging the instruction in a binary makes sure all instructions are present in a separate dynamic library. This arrangement helps in isolate of code for a specific Flutter view’s root isolate. + +## Android Configuration + +`gen_snapshot` is invoked by Gradle to generate the four binary blobs for each artifact. + +Gradle actually invokes `flutter build aot` under the hood to generate these artifacts. These binary artifacts are packaged into the APK directly. These artifacts are named `vm_snapshot_data`, `vm_snapshot_instr`, `isolate_snapshot_data` and `isolate_snapshot_instr`. They refer to the four snapshots mentioned above. + +The embedder may choose to put these artifacts in custom locations. In such cases, the embedder must place these artifacts in a directory and specify the location using the following 5 flags: + * `aot-snapshot-path`: Path to the directory in the APK assets containing the snapshot. + * `vm-snapshot-data`: Path to the VM snapshot in that directory. + * `vm-snapshot-instr`: Path to the VM instructions in that directory. + * `isolate-snapshot-data`: Path to the isolate snapshot in that directory. + * `isolate-snapshot-instr`: Path to the isolate instructions in that directory. + +Once the Flutter engine, itself packaged as a separate dynamic library called `libflutter.so`, resolves the locations of the required artifacts, it maps them in and makes sure the instructions are executable before launching the VM and isolate. + +The tooling on Android does not require the presence of the Android NDK because the engine can mark pages as executable at runtime. If the embedder wishes to package the snapshots into a single dynamic library (and make the configuration look like the one used on iOS), it can do so and specify the library to the engine via the flag `aot-shared-library-path`. A native toolchain is necessary on the host to achieve this. + +# Notes + +`gen_snapshot` can only generate AOT instructions for a specific architecture. For example, generating AOT instructions for `armv7`, `aarch64`, `i386` and `x86-64` requires four different `gen_snapshot` variants. + +The VM and isolate snapshot data is specific to the `gen_snapshot` variant used and must also be generated along with the AOT instructions. Embedders may not mix and match the AOT instructions and data snapshots from different `gen_snapshot` variants. All four artifacts must typically be generated at the same time. + +Using the current Flutter tools, only an `i386` `gen_snapshot` on the host can generate `armv7` AOT instructions and an `x86-64` `gen_snapshot` on the host can generate `aarch64` instructions. The mac builds take advantage of this quirk and `lipo` the two `gen_snapshot` variants and select the executable architecture correctly when generating AOT instructions for the target architecture. This can be inspected with the `lipo` tool on the host while analyzing the `gen_snapshot` binary. In reality, the word sizes of `gen_snapshot` on host and the engine running on the target must match. Theoretically, an `armv7` `gen_snapshot` running on a Raspberry Pi host could generate an AOT snapshot for `armv7` target. The tools don’t ship this configuration however. + +Packaging multiple Flutter AOT application in the same application bundle will currently result in redundant copies of the VM data and instructions. However, these buffers are extremely small. + +Flags to the Flutter engine are typically specified when the platform launches the underlying Flutter shell (platform agnostic way of interacting with the innards of the Flutter engine). This happens in `FlutterViewController` on iOS and `FlutterNativeView` in Java on Android. + +To list all the flags currently supported by the Flutter engine, embedders may locate the `flutter_tester` binary in the tools distribution and pass the `--help` flag to the same. All currently supported flags along with a short help blob will be dumped to the console. diff --git a/docs/Image-Codecs-in-the-Flutter-Engine.md b/docs/Image-Codecs-in-the-Flutter-Engine.md new file mode 100644 index 0000000000000..4ea514f02e29c --- /dev/null +++ b/docs/Image-Codecs-in-the-Flutter-Engine.md @@ -0,0 +1,11 @@ +# Image Codecs in the Flutter Engine + +Flutter has built-in support for a number of common codecs. Images in the supported formats can be decompressed by the Flutter Engine even if they are not supported by the underlying platform. + +However, each codec supported out-of-the-box by the Flutter Engine adds to the binary size of the engine. Unlike AOT compiled Dart code, this binary size increase cannot be tree shaken away. Every user of the Flutter application will be forced to download the codec irrespective of whether their Flutter applications actually decompress images of that type. Flutter users and developers are extremely sensitive to increases in binary size of the application. + +If the Flutter engine is unable to decompress specific images because it doesn’t support that codec, it will delegate responsibility of image decompression to the platform. This comes in handy if the platform has support for newer or less widely used codecs. The downside of course is that the application author will need to provide an alternative in case both the engine and platform don’t support a specific codec. + +If the application wishes to support a codec that isn’t supported by either the engine or the platform, it will have to ship the codec along with their application and expose its functionality via a plugin. This is more cumbersome but the current recommendation for applications for whom support for specific formats is table stakes. + +Speculatively adding and removing codecs into the Flutter engine may lead to fragmentation in the supported codecs across Flutter engine versions. But, if a format does become widely used, it strengthens the case for adding the codec to the engine in spite of the binary size increase. diff --git a/docs/JIT-Release-Modes.md b/docs/JIT-Release-Modes.md new file mode 100644 index 0000000000000..a6b991b7774a8 --- /dev/null +++ b/docs/JIT-Release-Modes.md @@ -0,0 +1,15 @@ +Normally Flutter runs in JIT for faster compilation/debugging support in `debug` mode and AOT mode for better performance in `profile` and `release` mode. For platforms that Flutter cannot produce AOT artifacts for, such as Android x86 (32 bit), a JIT release build may be used instead. The advantage of this mode over a regular debug build is that it removes debugging support and disables assertions which makes the final artifact smaller and more performant, though less so than a full AOT build. + + +JIT release mode can be used with a local engine configuration. For example, to setup an Android x86 jit_release and host build you can use the GN command below. Both device and host artifacts need to be built, except in cases where they are the same such as the Desktop shells. + +```shell +./flutter/tools/gn --runtime-mode=jit_release --android --android-cpu=x86 +ninja -C out/android_jit_release_x86 +./flutter/tools/gn --runtime-mode=jit_release +ninja -C out/host_jit_release +``` + +This can be used with the flutter tool [via the `--local-engine`](Debugging-the-engine.md#running-a-flutter-app-with-a-local-engine) flag to produce a bundle containing the jit release artifacts using the `flutter assemble` command. By default, flutter.gradle does not know how to package this artifacts so it requires custom integration into a build pipeline. Nevertheless, the artifact structure should be identical to a debug build, but with asserts disabled and product mode enabled. + +jit_release is not supported on iOS devices. Applications built in JIT mode cannot be distributed on the Apple App Store. \ No newline at end of file diff --git a/docs/Life-of-a-Flutter-Frame.md b/docs/Life-of-a-Flutter-Frame.md new file mode 100644 index 0000000000000..e606895351afe --- /dev/null +++ b/docs/Life-of-a-Flutter-Frame.md @@ -0,0 +1,62 @@ +Flutter apps work by transforming the widget tree in an application to a render tree that describes how to graphically render the widgets onscreen, and animating in response to input events or the passage of time. We refer to a single still image within the sequence of images composing an animation as a frame, similar to a frame in a movie filmstrip. In the context of a complete Flutter application these are rendered into a `FlutterView` class which is typically an instance of a platform-specific view class like [SurfaceView][surfaceview] on Android, [UIView][uiview] on iOS, [HWND][hwnd] on Windows, [NSView][nsview] on macOS, or [GtkBox][gtkbox] on Linux. + +This page attempts to describe the life of a Flutter frame in the engine from initial trigger to rasterization, presentation, and finally destruction/recycling. For an overview of the Framework side of this process, see the [Flutter's Rendering Pipeline][renderingPipelineTalk] tech talk. + +[gtkbox]: https://docs.gtk.org/gtk3/class.Box.html +[hwnd]: https://learn.microsoft.com/en-us/windows/win32/winprog/windows-data-types#HWND +[nsview]: https://developer.apple.com/documentation/appkit/nsview +[surfaceview]: https://developer.android.com/reference/android/view/SurfaceView +[uiview]: https://developer.apple.com/documentation/uikit/uiview + +## How a frame begins + +All frames are born with a call to `RequestFrame` in the [Animator][animator]. + +A frame may be requested for a variety of reasons, ranging from resizing the Flutter view, to lifecycle events like backgrounding or foregrounding an app, to requests from either the app (via dart:ui's [PlatformDispatcher.scheduleFrame][scheduleFrame]) or the embedder (via the [embedder API][embedderAPI]'s `FlutterEngineScheduleFrame`). + +Flutter does some minimal housekeeping when a frame is requested, primarily to ignore any duplicate requests to schedule a frame before the frame is actually produced. + +Once a frame is scheduled, Flutter [waits for a vsync][vsyncWaiter] from the operating system to proceed. + + +## Building the frame + +At the heart of Flutter's graphics workflow is the frame [pipeline][pipeline]. The pipeline is responsible for coordinating work between the UI thread, where the application code runs, and the Raster thread, where rasterization and compositing is performed. See the [threading section][engineArchThreading] of the Engine Architecture wiki for more details on threading in the engine. + +When a vsync occurs, Flutter begins the work of producing the frame in [Animator][animator]'s aptly-named `BeginFrame`. At this point, the animator reserves a spot in the pipeline and notifies the framework to begin the process of producing a frame by triggering the [PlatformDispatcher.onBeginFrame][onBeginFrame] dart:ui callback. + +When using the engine with the Flutter framework, `onBeginFrame` is handled by [handleBeginFrame][handleBeginFrame] in the framework, whose job it is to kick off the production of a [Scene][scene] in the framework. A good overview of this process can be found in the documentation of [RendererBinding.drawFrame][drawFrame] and the [Flutter's Rendering Pipeline][renderingPipelineTalk] tech talk. This process ultimately culminates in the production of a `Scene` which is handed back to the engine through a call to [FlutterView.render][flutterViewRender]. + +On the engine side, a `Scene` is represented as a [LayerTree][layerTree]. Calling `FlutterView.render` hands the layer tree to the Animator via a call its `Render` method, which posts the layer tree to the pipeline and notifies the Rasterizer that it's time to start rasterizing the frame. + +## Rasterizing the frame + +Rasterization is the process of converting the in-memory layer tree into pixels on a surface. Rasterization-related code in Flutter is executed on the Raster thread, which coordinates with the GPU. On some platforms, the Raster thread and the Platform thread may be the same thread. + +Rasterization starts with a call to the `Draw` method in the [Rasterizer][rasterizer]. At this point, the recently-produced `LayerTree` is pulled from the pipeline. The rasterizer does a quick check to see whether the app is running in headless mode (e.g. backgrounded) and if so, the frame is discarded; otherwise, rasterization proceeds. + +Rasterization begins with a request for a surface to which the GPU can draw via a call to the `AcquireFrame` method of [Surface][surface]. This delegates platform-specific code implemented in each embedder in response to callbacks configured in `FlutterRendererConfig` which acquires an appropriate Metal, OpenGL, Vulkan, or software surface for use by the rasterizer. + +Once a surface is acquired, the [LayerTree][layerTree] is rasterized to the surface via recursive `Preroll` and `Paint` calls through the layers. Behavior of these calls is specific to each layer type, but in the end, generally resolves to drawing via either either Skia or Impeller. Once the layer tree has been walked and all graphics operations have been collected, the frame is submitted to the GPU, and embedders are provided a callback to perform further platform-specific handling on their part -- typically presenting the surface via the platform-specific view implementation. + +The above process is repeated until the pipeline is empty. + +## Cleaning up frame resources + +TODO(cbracken): write this up using [this patch](https://github.com/flutter/engine/pull/38038) as a reminder. + +[animator]: https://github.com/flutter/engine/blob/main/shell/common/animator.h +[drawFrame]: https://api.flutter.dev/flutter/rendering/RendererBinding/drawFrame.html +[embedderAPI]: https://github.com/flutter/engine/blob/main/shell/platform/embedder/embedder.h +[engineArchThreading]: ../about/The-Engine-architecture.md#threading +[flutterViewRender]: https://api.flutter.dev/flutter/dart-ui/FlutterView/render.html +[handleBeginFrame]: https://api.flutter.dev/flutter/scheduler/SchedulerBinding/handleBeginFrame.html +[layerTree]: https://github.com/flutter/engine/blob/main/flow/layers/layer_tree.h +[onBeginFrame]: https://api.flutter.dev/flutter/dart-ui/PlatformDispatcher/onBeginFrame.html +[pipeline]: https://github.com/flutter/engine/blob/main/shell/common/pipeline.h +[rasterizer]: https://github.com/flutter/engine/blob/main/shell/common/rasterizer.h +[renderingPipelineTalk]: https://www.youtube.com/watch?v=UUfXWzp0-DU +[scene]: https://api.flutter.dev/flutter/dart-ui/Scene-class.html +[surface]: https://github.com/flutter/engine/blob/main/flow/surface.h +[scheduleFrame]: https://api.flutter.dev/flutter/dart-ui/PlatformDispatcher/scheduleFrame.html +[vsyncWaiter]: https://github.com/flutter/engine/blob/main/shell/common/vsync_waiter.h diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000000..5bfd804770cc7 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,32 @@ +This is an index of team-facing documentation for the [flutter/engine repository](https://github.com/flutter/engine/). + +- [Accessibility on Windows](../platforms/desktop/windows/Accessibility-on-Windows.md) +- [Code signing metadata](./release/Code-signing-metadata.md) for engine binaries +- [Compiling the engine](./contributing/Compiling-the-engine.md) +- [Comparing AOT Snapshot Sizes](./benchmarks/Comparing-AOT-Snapshot-Sizes.md) +- [Crashes](./Crashes.md) +- [Custom Flutter engine embedders](./Custom-Flutter-Engine-Embedders.md) +- [Custom Flutter Engine Embedding in AOT Mode](./Custom-Flutter-Engine-Embedding-in-AOT-Mode.md) +- [Debugging the engine](./Debugging-the-engine.md) +- [Flutter engine operation in AOT Mode](./Flutter-engine-operation-in-AOT-Mode.md) +- [Flutter Test Fonts](../contributing/testing/Flutter-Test-Fonts.md) +- [Flutter's modes](./Flutter's-modes.md) +- [Engine Clang Tidy Linter](./ci/Engine-Clang-Tidy-Linter.md) +- [Engine disk footprint](./Engine-disk-footprint.md) +- [Engine-specific Service Protocol extensions](./Engine-specific-Service-Protocol-extensions.md) +- [Engine pre‐submits and post‐submits](./ci/Engine-pre-submits-and-post-submits.md) +- [Image Codecs in the Flutter Engine](Image-Codecs-in-the-Flutter-Engine.md) +- [Impeller](./impeller/README.md) documentation index +- [Life of a Flutter Frame](Life-of-a-Flutter-Frame.md) +- [Reduce Flutter engine size with MLGO](Reduce-Flutter-engine-size-with-MLGO.md) +- [Resolving common build failures](../platforms/android/Resolving-common-build-failures.md) +- [Setting up the Engine development environment](./contributing/Setting-up-the-Engine-development-environment.md) +- [Supporting legacy platforms](Supporting-legacy-platforms.md) +- [Testing Android Changes in the Devicelab on an Emulator](../platforms/android/Testing-Android-Changes-in-the-Devicelab-on-an-Emulator.md) +- [Testing the engine](./testing/Testing-the-engine.md) +- [Testing presubmit Engine PRs with the Flutter framework](Testing-presubmit-Engine-PRs-with-the-Flutter-framework.md) +- [The Engine architecture](../about/The-Engine-architecture.md) +- [Upgrading Engine's Android API version](../platforms/android/Upgrading-Engine's-Android-API-version.md) +- [Using the Dart Development Service (DDS) and Flutter DevTools with a custom Flutter Engine Embedding](./Using-the-Dart-Development-Service-(DDS)-and-Flutter-DevTools-with-a-custom-Flutter-Engine-Embedding.md) +- [Using Sanitizers with the Flutter Engine](./Using-Sanitizers-with-the-Flutter-Engine.md) +- [Why we have a separate engine repo](../about/Why-we-have-a-separate-engine-repo.md) diff --git a/docs/Reduce-Flutter-engine-size-with-MLGO.md b/docs/Reduce-Flutter-engine-size-with-MLGO.md new file mode 100644 index 0000000000000..75ed9b2c02e6e --- /dev/null +++ b/docs/Reduce-Flutter-engine-size-with-MLGO.md @@ -0,0 +1,233 @@ +Mobile apps are very sensitive to their download sizes as every increase in KB +may result in a user number decrease. Flutter engine (`libflutter.so`) has to be +included in every Flutter app and it has a size of several MBs. So we'll try to +reduce its size by using [MLGO][MLGO]. It's different from the previous Flutter +attempt of reducing sizes as MLGO does not require any code or dependency +removals. + +Reducing engine size with MLGO needs to 1) train a model once, 2) apply that +model to compile the Flutter engine. Note that model training does not need to +happen too frequently - the model should 'hold up' to code changes over +weeks/months. On Ubuntu, do the following: + +- **Follow the [Setting-up-the-Engine-development-environment][engine setup]**. + To check if this step is successful, try [compiling for Android][compile + android]. For size comparisons, we recommend using the release Android build + `./flutter/tools/gn --android --runtime-mode=release --no-goma`. (Option + `--no-goma` is needed if you're not a Googler, or if you're using a custom + Clang as we'll do later with MLGO.) +- **Set up [MLGO] LLVM**. (The steps are adapted from [MLGO demo][MLGO demo].) + 1. Prerequisites: `sudo apt-get install cmake ninja-build lld`. + 2. Create a root directory for everything MLGO related `mkdir ~/mlgo && + export MLGO_DIR=~/mlgo` + 3. Clone MLGO repo + `cd $MLGO_DIR && git clone https://github.com/google/ml-compiler-opt.git` + 4. Tensorflow dependencies + ```bash + cd $MLGO_DIR + sudo apt-get install python3-pip + python3 -m pip install --upgrade pip + python3 -m pip install --user -r ml-compiler-opt/requirements.txt + + TF_PIP=$(python3 -m pip show tensorflow | grep Location | cut -d ' ' -f 2) + + export TENSORFLOW_AOT_PATH="${TF_PIP}/tensorflow" + + mkdir $MLGO_DIR/tensorflow + export TENSORFLOW_C_LIB_PATH=$MLGO_DIR/tensorflow + + wget --quiet https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.15.0.tar.gz + + tar xfz libtensorflow-cpu-linux-x86_64-1.15.0.tar.gz -C "${TENSORFLOW_C_LIB_PATH}" + ``` + 5. Clone llvm-project + ```bash + cd $MLGO_DIR && git clone https://github.com/llvm/llvm-project.git + export LLVM_SRCDIR=$MLGO_DIR/llvm-project + export LLVM_INSTALLDIR=$MLGO_DIR/llvm-install + ``` + 6. Build LLVM + ```bash + cd ${LLVM_SRCDIR} + mkdir build + cd build + cmake -G Ninja \ + -DLLVM_ENABLE_LTO=OFF \ + -DCMAKE_INSTALL_PREFIX= \ + -DTENSORFLOW_C_LIB_PATH=${TENSORFLOW_C_LIB_PATH} \ + -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=On \ + -C ${LLVM_SRCDIR}/clang/cmake/caches/Fuchsia-stage2.cmake \ + ${LLVM_SRCDIR}/llvm + + ninja distribution + DESTDIR=${LLVM_INSTALLDIR} ninja install-distribution-stripped + ``` +- **Build Flutter engine for MLGO training** + ```bash + # Set your engine dir appropriately if it's not in the default location + export ENGINE_DIR=~/flutter/engine/src + cd $ENGINE_DIR + + sed -i \ + 's/cflags += lto_flags/cflags += lto_flags + ["-Xclang", "-fembed-bitcode=all"]/' \ + build/config/compiler/BUILD.gn + + sed -i \ + "s/prefix = rebase_path(\"\/\/buildtools\/\$host_dir\/clang\/bin\", root_build_dir)/prefix = \"${LLVM_INSTALLDIR//\//\\/}\/bin\"/" \ + build/toolchain/android/BUILD.gn + + ./flutter/tools/gn --android --runtime-mode=release --no-goma --no-lto + ninja -C out/android_release + ``` +- **Train the model** + ```bash + export CORPUS=$MLGO_DIR/corpus + cd $MLGO_DIR/ml-compiler-opt + python3 compiler_opt/tools/extract_ir.py \ + --cmd_filter="^-Oz$" \ + --input=$ENGINE_DIR/out/compile_commands.json \ + --input_type=json \ + --llvm_objcopy_path=$LLVM_INSTALLDIR/bin/llvm-objcopy \ + --output_dir=$CORPUS + + export DEFAULT_TRACE=$MLGO_DIR/default_trace + export WARMSTART_OUTPUT_DIR=$MLGO_DIR/warmstart + export OUTPUT_DIR=$MLGO_DIR/model + + rm -rf $DEFAULT_TRACE && \ + PYTHONPATH=$PYTHONPATH:. python3 \ + compiler_opt/tools/generate_default_trace.py \ + --data_path=$CORPUS \ + --output_path=$DEFAULT_TRACE \ + --compile_task=inlining \ + --clang_path=$LLVM_INSTALLDIR/bin/clang \ + --llvm_size_path=$LLVM_INSTALLDIR/bin/llvm-size \ + --sampling_rate=0.2 + + rm -rf $WARMSTART_OUTPUT_DIR && \ + PYTHONPATH=$PYTHONPATH:. python3 \ + compiler_opt/rl/train_bc.py \ + --root_dir=$WARMSTART_OUTPUT_DIR \ + --data_path=$DEFAULT_TRACE \ + --gin_files=compiler_opt/rl/inlining/gin_configs/behavioral_cloning_nn_agent.gin + + # The following will take about half a day. + rm -rf $OUTPUT_DIR && \ + PYTHONPATH=$PYTHONPATH:. python3 \ + compiler_opt/rl/train_locally.py \ + --root_dir=$OUTPUT_DIR \ + --data_path=$CORPUS \ + --clang_path=$LLVM_INSTALLDIR/bin/clang \ + --llvm_size_path=$LLVM_INSTALLDIR/bin/llvm-size \ + --num_modules=100 \ + --gin_files=compiler_opt/rl/inlining/gin_configs/ppo_nn_agent.gin \ + --gin_bindings=train_eval.warmstart_policy_dir=\"$WARMSTART_OUTPUT_DIR/saved_policy\" + ``` +- **Build LLVM with the trained model** + ```bash + cd $LLVM_SRCDIR + rm -rf llvm/lib/Analysis/models/inliner/* + cp -rf $OUTPUT_DIR/saved_policy/* llvm/lib/Analysis/models/inliner/ + + mkdir build-release + cd build-release + cmake -G Ninja \ + -DLLVM_ENABLE_LTO=OFF \ + -DCMAKE_INSTALL_PREFIX= \ + -DTENSORFLOW_AOT_PATH=${TENSORFLOW_AOT_PATH} \ + -C ${LLVM_SRCDIR}/clang/cmake/caches/Fuchsia-stage2.cmake \ + ${LLVM_SRCDIR}/llvm + + export LLVM_INSTALLDIR_RELEASE=$LLVM_INSTALLDIR-release + ninja distribution + DESTDIR=${LLVM_INSTALLDIR_RELEASE} ninja install-distribution-stripped + ``` +- **Build Flutter engine using LLVM with the trained model** + ```bash + cd $ENGINE_DIR + git stash # Undo previous changes for model training + + sed -i \ + 's/cflags += lto_flags/cflags += lto_flags + ["-mllvm", "-enable-ml-inliner=release"]/' \ + build/config/compiler/BUILD.gn + + sed -i \ + "s/prefix = rebase_path(\"\/\/buildtools\/\$host_dir\/clang\/bin\", root_build_dir)/prefix = \"${LLVM_INSTALLDIR_RELEASE//\//\\/}\/bin\"/" \ + build/toolchain/android/BUILD.gn + + ./flutter/tools/gn --android --runtime-mode=release --no-goma --no-lto + ninja -C out/android_release libflutter.so + ``` +- **Compare**. To compare the engine size with or without MLGO, one can add or + remove the `["-mllvm", "-enable-ml-inliner=release"]` flags in + `build/config/compiler/BUILD.gn`, compile the engine, and check the size of + `out/android_release/lib.stripped/libflutter.so`. As end-users will download + zipped engine, we also recommend comparing its zipped size. + ```bash + export ENGINE_LIB_DIR=$ENGINE_DIR/out/android_release/lib.stripped + + cd $ENGINE_DIR + ./flutter/tools/gn --android --runtime-mode=release --no-goma --no-lto + ninja -C out/android_release libflutter.so + cd $ENGINE_LIB_DIR + mv libflutter.so libflutter.ml_nolto.so + zip libflutter.ml_nolto.so.zip libflutter.ml_nolto.so + + cd $ENGINE_DIR + ./flutter/tools/gn --android --runtime-mode=release --no-goma + ninja -C out/android_release libflutter.so + cd $ENGINE_LIB_DIR + mv libflutter.so libflutter.ml_lto.so + zip libflutter.ml_lto.so.zip libflutter.ml_lto.so + + # Remove the ML flags to disable ML. + cd $ENGINE_DIR + sed -i \ + 's/cflags += lto_flags + \["-mllvm", "-enable-ml-inliner=release"\]/cflags += lto_flags/' \ + build/config/compiler/BUILD.gn + + cd $ENGINE_DIR + ./flutter/tools/gn --android --runtime-mode=release --no-goma --no-lto + ninja -C out/android_release libflutter.so + cd $ENGINE_LIB_DIR + mv libflutter.so libflutter.noml_nolto.so + zip libflutter.noml_nolto.so.zip libflutter.noml_nolto.so + + cd $ENGINE_DIR + ./flutter/tools/gn --android --runtime-mode=release --no-goma + ninja -C out/android_release libflutter.so + cd $ENGINE_LIB_DIR + mv libflutter.so libflutter.noml_lto.so + zip libflutter.noml_lto.so.zip libflutter.noml_lto.so + + ls -l + ``` + + Here's the table of size comparisons for engine version b9ecd8a. + + | Flutter engine size comparison | ML_LTO | ML_NOLTO | NOML_LTO | NOML_NOLTO | + | --- | --- | --- | --- | --- | + | unzipped size (bytes) | 6270960 | 6338580 | 6312012 | 6577684 | + | zipped size (bytes) | 3586091 | **3577604** | 3606484 | 3689468 | + | unzipped size change over NOML_LTO | -0.65% | 0.4% | 0 | 4.21% | + | zipped size change over NOML_LTO | -0.57% | **-0.80%** | 0 | 2.3% | + | unzipped size change over NOML_NOLTO | -4.66% | -3.64% | -4.04% | 0 | + | zipped size change over NOML_NOLTO | -2.80% | -3.03% | -2.25% | 0 | + +## Conclusion + +As shown in the table above, for the zipped size, the winner here is the +ML_NOLTO version which is even smaller than the ML_LTO version. It has a 0.8% +reduction over our previous art of NOML_LTO. + +The ML_LTO version is not very good because currently the model can only be +trained without LTO. [MLGO][MLGO] is planning to allow ThinLTO in their +training. Hopefully, it will help achieve the MLGO's normal reduction of 3%-5% +(e.g., ML_NOLTO vs NOML_NOLTO) when the training and final build are in the same +condition. + + +[MLGO]: https://github.com/google/ml-compiler-opt +[engine setup]: ./contributing/Setting-up-the-Engine-development-environment +[compile android]: ./contributing/Compiling-the-engine#compiling-for-android-from-macos-or-linux +[MLGO demo]: https://github.com/google/ml-compiler-opt/blob/main/docs/demo/demo.md diff --git a/docs/Supporting-legacy-platforms.md b/docs/Supporting-legacy-platforms.md new file mode 100644 index 0000000000000..632a65613fbba --- /dev/null +++ b/docs/Supporting-legacy-platforms.md @@ -0,0 +1,28 @@ +## Building ARMv7 (iOS) & armeabi v7a (Android) with Xcode10 + +In Xcode10, the i386 architecture is deprecated for macOS, so building the Flutter engine for armv7/armeabi-v7a fails. Specifically, libraries like CoreFoundation contain only code for the x86_64 architecture. + +![iOS ARMv7](https://user-images.githubusercontent.com/817851/45751101-e7a54980-bc43-11e8-833f-b6458c9a4762.png) + +![Android armeabi-v7a](https://user-images.githubusercontent.com/817851/45751099-e70cb300-bc43-11e8-97fa-a877dff5449d.png) + +To address this, get the MacOS 10.13 SDK from Xcode 9.x from [Apple](https://developer.apple.com/download/more/), and extract the SDK components from the `.xip` file. Uncompress the SDK into `/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs` and name the SDK `MacOSX10.13.sdk`: + +![Uncompressed SDK in Xcode10](https://user-images.githubusercontent.com/817851/45752211-47512400-bc47-11e8-88fe-b738ac53831f.png) + +To check if the logic is fine, run command below: + +```bash +python your-flutter-engine-path/engine/src/build/mac/find_sdk.py 10.12 +``` + +When `find_sdk.py` return 10.13, the ninja build will succeed for gen_snapshot (i386), Flutter.framework (ARMv7) and libflutter.so (armeabi-v7a). + +## Build Flutter engine for 32bit iOS simulator on modern Mac(x86_64) + +To build the Flutter engine for iOS simulator on a modern Mac(x86_64), the gn command will generate a `target_cpu` value with x64. Henceforth, the Flutter.framework and gen_snapshot will be x86_64. +However, sometimes you may want to develop Flutter on a 32bit simulator(like iPhone5), you will need both Flutter.framework and gen_snapshot to be i386. + +Follow instruction below to change the default behavior in gn command: +1. Edit your-flutter-engine-path/engine/src/flutter/tools/gn +![Edit gn](https://user-images.githubusercontent.com/817851/49006557-57840300-f1a4-11e8-850a-d019dc854bbd.png) \ No newline at end of file diff --git a/docs/Testing-presubmit-Engine-PRs-with-the-Flutter-framework.md b/docs/Testing-presubmit-Engine-PRs-with-the-Flutter-framework.md new file mode 100644 index 0000000000000..d0961a23885b3 --- /dev/null +++ b/docs/Testing-presubmit-Engine-PRs-with-the-Flutter-framework.md @@ -0,0 +1,48 @@ +# Running Framework presubmit tests on Engine PRs + +This documentation describes how to run flutter/flutter presubmit checks on flutter/engine PRs before submitting them. + +# Overview + +1. Wait for all presubmit checks on your flutter/engine PR to be green. +2. Determine the commit hash for your flutter/engine PR. +3. Create and upload a flutter/flutter PR, (OR run tests locally). +4. Wait for flutter/flutter presubmits/tests to run ☕. + +Step (1) is the usual flutter/engine workflow. + +# 2. The commit hash + +1. Go to the "Commits" tab in the GitHub UI for you Engine PR. +1. Click the button to copy the most recent commit hash to your clipboard. + +Screenshot 2023-08-04 at 12 54 55 PM + +# 3. Create and upload a flutter/flutter PR. + +Edit your flutter/flutter checkout as follows: + +1. `bin/internal/engine.version` should be edited to contain the commit hash from (2). +1. `bin/internal/engine.realm` should be edited to contain the string `flutter_archives_v2`. + +To run flutter/flutter presubmits on CI, you can accomplish these two edits directly in the GitHub editor UI, if desired. Otherwise, upload a flutter/flutter PR with these changes. + +You can also build apps, and run tests locally at this point. + +# 4. Wait for flutter/flutter presubmits to run ☕. + +The flutter/flutter presubmit checks will run. There will be at least two failures: +1. A Flutter CLI test will ensure that a PR with a non-empty `engine.realm` file will fail a presubmit check. +1. The `fuchsia_precache` test will fail because Fuchsia artifacts are not uploaded from Engine presubmit runs. + +Any other failures are possibly due to the changes to flutter/engine, so deflake and examine them carefully. + +# 5. Devicelab tests + +A subset of devicelab tests are available for optionally running in presubmit on flutter/flutter PRs. They are the tests listed in the flutter/flutter [.ci.yaml](https://github.com/flutter/flutter/blob/main/.ci.yaml) file that are prefixed with `Linux_android`, `Mac_android`, and `Mac_ios`. + +To run one of these tests, remove the line `presubmit: false` from the `.ci.yaml` file under the test you'd like to run. For an example, see the PR [here](https://github.com/flutter/flutter/pull/135254). + +Screenshot 2023-09-21 at 3 19 51 PM + +This will trigger the devicelab test to run. The test will show up in the list of presubmit checks, and you can click through to the [LUCI page](https://ci.chromium.org/ui/p/flutter/builders/try/Linux_android%20new_gallery__transition_perf/2/overview) to see the results. diff --git a/docs/Using-Sanitizers-with-the-Flutter-Engine.md b/docs/Using-Sanitizers-with-the-Flutter-Engine.md new file mode 100644 index 0000000000000..9540d12d803c9 --- /dev/null +++ b/docs/Using-Sanitizers-with-the-Flutter-Engine.md @@ -0,0 +1,125 @@ + +All Flutter Engine builds can now enable a combination of the following sanitizers. The different sanitizers are used to isolate specific classes of construction issues. + +* **Thread Sanitizer**: Detects data races. +* **Address Sanitizer**: Detects memory errors like use-after-free, buffer overflows... +* **Memory Sanitizer**: Detects reads on uninitialized memory. +* **Undefined Behavior Sanitizer**: Detects undefined behavior. +* **Leak Sanitizer**: Detects memory leaks. + +While the buildroot is wired up to attempt sanitizer enabled builds for all targets, not all target architectures, host platform and sanitizer combinations are supported. Support for each sanitizer also varies greatly with each toolchain version. For this reason, and due to general constraints in available build/test infrastructure, these sanitizers are not enabled by default on presubmits or build-bots. It is best to work backwards from a problem and choose a sanitizer that has the best shot of isolating the same. + + +## General Usage Guidelines + +Most sanitizer enabled variants will either fail to build at all or fail when running specific unit-test targets. These failures are either because there are false positives that have not been annotated correctly or genuine issues in either the Flutter Engine or Dart VM. However, it is possible to discover further issues without fixing them all. This is done by selectively suppressing known issues. The suppressions and other sanitizer options have to be specified as environment options. These options can be specified in one shot by invoking the following in the current shell: + + +``` +source ./flutter/testing/sanitizer_suppressions.sh +``` + + +This should enable relevant options for all known sanitizers and also print the files containing suppressions for each sanitizer. + +Sample output: + + +``` +Using Thread Sanitizer suppressions in ./flutter/testing/tsan_suppressions.txt +Using Leak Sanitizer suppressions in ./flutter/testing/lsan_suppressions.txt +``` + + +You can run the specific unit-test binaries in the terminal and all sanitizer options and suppressions will take hold. + +If an issue is raised that is not previously known, a suppression may be added and a bug filed to track the same. Instructions on how to add suppressions for the specific sanitizer are specific at the top of the relevant suppressions file. + +Make sure to add suppression that are as detailed as possible. Adding suppressions that are too broad may suppress errors from issues that are actually new and untracked. If you find that the issue you expect is not caught by the sanitizer, make sure to check the list of suppressions to see if any of the suppressed issues look relevant. If so, you might need to selectively disable suppressions: + + +``` +----------------------------------------------------- +Suppressions used: + count bytes template + 1 120 class_createInstance + 5 80 MakeSkSurfaceFromBackingStore + 3 128 _dispatch_once_callout +----------------------------------------------------- +``` + + +Support for sanitizer enabled builds using Goma is spotty. Disable Goma for more reliable builds till these issues are addressed. + +All sanitizer caught issues are tagged with the [sanitizer](https://github.com/flutter/flutter/labels/sanitizer) label. + + +## Leak Sanitizer + +A tool used to detect memory leaks. Official documentation is at [https://clang.llvm.org/docs/LeakSanitizer.html](https://clang.llvm.org/docs/LeakSanitizer.html). Enable using the `--lsan` GN flag on any target. For best results, use this with unoptimized build variants. The buildroot is configured to use the ideal arguments in this mode. Full documentation on how to add suppressions to this file is at [https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer#suppressions](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer#suppressions). This tool can be used either in standalone mode or in conjunction with Address Sanitizer. + + +``` +$ ./flutter/tools/gn --runtime-mode debug --lsan --unoptimized --no-goma +$ autoninja -C out/host_debug_unopt +$ source ./flutter/testing/sanitizer_suppressions.sh +$ ./out/host_debug_unopt/embedder_unittests +``` + + +## Address Sanitizer + +Address sanitizer detects memory errors. Official documentation is at [https://github.com/google/sanitizers/wiki/AddressSanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer). Enable using the `--asan` flag. Enabling address sanitizer also implicitly enables +Leak Sanitizer. The `./flutter/testing/sanitizer_suppressions.sh` script enables leak sanitization in Address Sanitizer builds. + + +``` +$ ./flutter/tools/gn --runtime-mode debug --asan --unoptimized --no-goma +$ autoninja -C out/host_debug_unopt +$ source ./flutter/testing/sanitizer_suppressions.sh +$ ./out/host_debug_unopt/embedder_unittests +``` + + +Address sanitizer has [spotty support on aarch64](https://github.com/google/sanitizers/wiki/AddressSanitizer#introduction) targets. Use it on x64 target for best results. + + +## Undefined Behavior Sanitizer + +Catches undefined behavior. Official documentation at [https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html). Enable using the `--ubsan` flag. The `sanitizer_suppressions.sh` file will specify a suppressions files. [Classes of errors](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html#id4) can be disabled at runtime. + + +``` +$ ./flutter/tools/gn --runtime-mode debug --ubsan --unoptimized --no-goma +$ autoninja -C out/host_debug_unopt +$ source ./flutter/testing/sanitizer_suppressions.sh +$ ./out/host_debug_unopt/embedder_unittests +``` + + + +## Thread Sanitizer + +Catches data races. Official documentation at [https://clang.llvm.org/docs/ThreadSanitizer.html](https://clang.llvm.org/docs/ThreadSanitizer.html). + + +``` +$ ./flutter/tools/gn --runtime-mode debug --tsan --unoptimized --no-goma +$ autoninja -C out/host_debug_unopt +$ source ./flutter/testing/sanitizer_suppressions.sh +$ ./out/host_debug_unopt/embedder_unittests +``` + + + +## Memory Sanitizer + +Detects reads of uninitialized memory. This sanitizer is only available on Linux. Enable using the `--msan` flag. + + +``` +$ ./flutter/tools/gn --runtime-mode debug --msan --unoptimized --no-goma +$ autoninja -C out/host_debug_unopt +$ source ./flutter/testing/sanitizer_suppressions.sh +$ ./out/host_debug_unopt/embedder_unittests +``` diff --git a/docs/Using-the-Dart-Development-Service-(DDS)-and-Flutter-DevTools-with-a-custom-Flutter-Engine-Embedding.md b/docs/Using-the-Dart-Development-Service-(DDS)-and-Flutter-DevTools-with-a-custom-Flutter-Engine-Embedding.md new file mode 100644 index 0000000000000..1f6177c521b18 --- /dev/null +++ b/docs/Using-the-Dart-Development-Service-(DDS)-and-Flutter-DevTools-with-a-custom-Flutter-Engine-Embedding.md @@ -0,0 +1,61 @@ +When attempting to connect to the VM service URI output by the Flutter engine in the context of a custom embedder, users may encounter the following error: + +`This VM does not have a registered Dart Development Service (DDS) instance and is not currently serving Dart DevTools.` + +This happens when a Flutter application is launched without using the `flutter` CLI tool, which typically is responsible for starting a Dart Development Service (DDS) instance. DDS is middleware for the Dart VM Service that provides additional functionality like log and event history, and can be configured to serve the DevTools developer tooling suite. + +Developers working on custom embeddings of the Flutter engine can start a DDS instance in one of two ways: + +1) Using the `dart development-service` command shipped with the Dart SDK (**recommended**). +2) Starting a Flutter DevTools instance using the `dart devtools` shipped with the Dart SDK, providing the VM service URI as an argument (e.g., `dart devtools http://localhost:8181`). + +## Using `dart development-service` + +DDS can be started using the `dart development-service` command. As of writing, the command has the following interface allowing for configuration of the service: + +``` +Start Dart's development service. + +Usage: dart [vm-options] development-service [arguments] +-h, --help Print this usage information. + --vm-service-uri= (mandatory) The VM service URI DDS will connect to. + --bind-address=
The address DDS should bind to. + (defaults to "localhost") + --bind-port= The port DDS should be served on. + (defaults to "0") + --[no-]disable-service-auth-codes Disables authentication codes. + --[no-]serve-devtools If provided, DDS will serve DevTools. If not specified, "--devtools-server-address" is ignored. + --devtools-server-address Redirect to an existing DevTools server. Ignored if "--serve-devtools" is not specified. + --[no-]enable-service-port-fallback Bind to a random port if DDS fails to bind to the provided port. + --cached-user-tags A set of UserTag names used to determine which CPU samples are cached by DDS. + --google3-workspace-root Sets the Google3 workspace root used for google3:// URI resolution. + +Run "dart help" to see global options. +``` + +The `--vm-service-uri` option is required and specifies the URI of the Dart VM service served by the Flutter engine. If provided, the `--serve-devtools` flag will result in the DevTools instance shipped with the Dart SDK from DDS's HTTP server. + +Running the command will result in JSON encoded connection information being output to `stdout`: + +```bash +$ dart development-service --vm-service-uri=http://127.0.0.1:59113/BBPoXnZUWFU=/ --serve-devtools +{"state":"started","ddsUri":"http://127.0.0.1:59123/tbrR0DzW2j8=/","devToolsUri":"http://127.0.0.1:59123/tbrR0DzW2j8=/devtools?uri=ws://127.0.0.1:59123/tbrR0DzW2j8=/ws","dtd":{"uri":"ws://127.0.0.1:59122/R1LbdlhtkUygRWNA"}} +``` + +Once DDS has started, all VM service requests should be made through the URI provided by DDS instead of the original VM service URI. In the context of the standalone Dart VM (i.e., `dart`) and the `flutter` tool, the original VM service URI is hidden and the DDS URI is advertised as the Dart VM service URI to reduce the likelihood of developers accidentally connecting to the VM service directly. Direct connections to a VM service with an active DDS instance attached will be rejected. + +The DDS instance will automatically shutdown when the target application is closed, requiring no manual lifecycle management. + +## Using `dart devtools` + +Developers are also able to start DevTools using the `dart devtools` command. By providing the VM service URI as a positional parameter, the served DevTools instance will automatically connect to the provided target application. However, before connecting directly to the VM service URI, the command first checks that the provided VM service URI points to a DDS instance. If it doesn't, the command will start DDS, print the DDS URI to console, and then launch a DevTools instance that connects to the DDS instance directly instead of to the provided VM service URI. + +```bash +$ dart devtools http://127.0.0.1:59251/2LS6f3Kb2JI=/ +Started the Dart Development Service (DDS) at http://127.0.0.1:59260/38XeuQpIHRE=/ +Serving DevTools at http://127.0.0.1:9101. + + Hit ctrl-c to terminate the server. +``` + +As with the `dart development-service` command, the lifecycle of DDS is tied directly to the lifecycle of the application it's connected to. However, using the `dart devtools` command does not cause DevTools to be served by DDS, meaning that DevTools will not be available if the `dart devtools` process is killed, even though DDS will remain attached to the target application. \ No newline at end of file diff --git a/docs/benchmarks/Comparing-AOT-Snapshot-Sizes.md b/docs/benchmarks/Comparing-AOT-Snapshot-Sizes.md new file mode 100644 index 0000000000000..c397f41fe7ee0 --- /dev/null +++ b/docs/benchmarks/Comparing-AOT-Snapshot-Sizes.md @@ -0,0 +1,90 @@ +These instructions can be used to prepare a tabulated summary of the differences in the sizes of two AOT snapshots. The instructions assume that the Flutter Engine has been [setup](../contributing/Setting-up-the-Engine-development-environment.md) on the host (at `FLUTTER_ENGINE` in these instructions). + +Build the AOT snapshot (`flutter build aot`) for the application but pass in the `--verbose` flag. We need to find the `gen_snapshot` invocation and re-run it with an extra option (`--print-instructions-sizes-to`). If you are instrumenting with a local engine, the `flutter build` [takes a `--local-engine` and `--local-engine-host` flag](../Debugging-the-engine.md#running-a-flutter-app-with-a-local-engine) as well. + +Here is an example of the updated invocation. Specify the path to a JSON file that acts as a summary (`SUMMARY_LOCATION` in these instructions) as follows. + +```bash +$FLUTTER_ENGINE/out/host_debug_unopt/gen_snapshot \ + --print-instructions-sizes-to=$SUMMARY_LOCATION \ + --causal_async_stacks \ + --packages=.packages \ + --deterministic \ + --snapshot_kind=app-aot-blobs \ + --vm_snapshot_data=build/aot/vm_snapshot_data \ + --isolate_snapshot_data=build/aot/isolate_snapshot_data \ + --vm_snapshot_instructions=build/aot/vm_snapshot_instr \ + --isolate_snapshot_instructions=build/aot/isolate_snapshot_instr \ + build/aot/app.dill +``` + +Save the file at `SUMMARY_LOCATION` as `before.json` + +Now, either change the Dart code with the changes you wish to see the effects, or, rebuild the engine to create an updated `gen_snapshot` binary. + +After you have made necessary changes, re-run `flutter build aot` again. This step is important because AOT compilation has a [kernel compilation step](../Custom-Flutter-Engine-Embedding-in-AOT-Mode.md#generating-the-kernel-snapshot) before the `gen_snapshot` invocation. + +Re-run the gen_snapshot invocation and save the resulting file to `after.json`. + +Run the `compare_size.dart` tool and pass in the `before.json` and `after.json` files to generate the summary. + +An example invocation looks as follow. + +``` +$FLUTTER_ENGINE/out/host_debug_unopt/dart-sdk/bin/dart \ + before.json \ + after.json +``` + +This should dump something like the following to the console. + +``` ++------------+----------------------------------------------------------+--------------+ +| Library | Method | Diff (Bytes) | ++------------+----------------------------------------------------------+--------------+ +| dart:async | new ZoneSpecification.from | +2136 | +| dart:async | runZoned | +1488 | +| dart:async | new _CustomZone | +927 | +| dart:async | runZoned. | +881 | +| dart:async | _rootFork | +504 | +| dart:async | _rootCreatePeriodicTimer | +500 | +| dart:async | _rootCreateTimer | +498 | +| dart:async | _rootRegisterUnaryCallback | +485 | +| dart:async | _rootRegisterBinaryCallback | +485 | +| dart:async | _rootRegisterCallback | +485 | +| dart:async | _rootPrint | +453 | +| dart:async | _CustomZone.fork | +396 | +| dart:async | _rootErrorCallback | +389 | +| dart:async | _CustomZone.bindUnaryCallbackGuarded | +368 | +| dart:async | _rootHandleUncaughtError | +342 | +| dart:async | _CustomZone.runBinary | +296 | +| dart:async | _CustomZone.runUnary | +293 | +| dart:async | _CustomZone.[] | +291 | +| dart:async | _CustomZone.registerCallback | +290 | +| dart:async | _CustomZone.run | +290 | +| dart:async | _CustomZone.registerUnaryCallback | +290 | +| dart:async | _CustomZone.registerBinaryCallback | +290 | +| dart:async | _CustomZone.runBinaryGuarded | +289 | +| dart:async | _CustomZone.runUnaryGuarded | +286 | +| dart:async | _RootZone.fork | +283 | +| dart:async | _CustomZone.bindCallback | +259 | +| dart:async | _CustomZone.bindUnaryCallback | +259 | +| dart:async | _CustomZone.bindUnaryCallback. | +248 | +| dart:async | _RootZone.bindUnaryCallback. | +248 | +| dart:async | _CustomZone.bindUnaryCallbackGuarded. | +248 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +| | [Stub] Type Test Type: class 'PopupMenuEntry' | -128 | +| | [Stub] Type Test Type: class '_SyncIterator@0150898' | -128 | +| | [Stub] Type Test Type: class 'PopupMenuItem' | -128 | +| | [Stub] Type Test Type: class 'FormFieldState' | -128 | +| | [Stub] Type Test Type: class 'PopupMenuButton' | -128 | +| | [Stub] Type Test Type: class '_SyncIterator@0150898' | -131 | +| | [Stub] Type Test Type: class '_SplayTreeMapNode@3220832' | -139 | +| | [Stub] Type Test Type: class '_SplayTreeMapNode@3220832' | -165 | +| dart:io | new Directory | -211 | +| dart:io | new Link | -211 | ++------------+----------------------------------------------------------+--------------+ +Total change +24036 bytes. + +``` diff --git a/docs/ci/Engine-Clang-Tidy-Linter.md b/docs/ci/Engine-Clang-Tidy-Linter.md new file mode 100644 index 0000000000000..05879f82181e6 --- /dev/null +++ b/docs/ci/Engine-Clang-Tidy-Linter.md @@ -0,0 +1,56 @@ +# Engine Clang Tidy Linter + +## Description + +In May 2020, [`clang-tidy`](https://clang.llvm.org/extra/clang-tidy/) was added as a CI step to the Flutter Engine. Previously the only lint checks that were happening in the engine were formatting, there were no semantic checks. Now there are, but that means there is work to be done migrating all the code to conform to all the lint checks. + +If a file has `// FLUTTER_NOLINT` at the top, it has issues with the lint that haven't been addressed and the linter will ignore it. As the issues are fixed the comments should be removed. + +You can run the linter locally by running `flutter/ci/lint.sh`. + +## CI background information + +* The clang-tidy ci step is run 4 times: host_debug on mac, ios_debug on mac, host_debug on linux, android_debug_arm64 on linux. +* Before the linter can run, the target must be built in order to generate code. +* Clang-tidy jobs are sharded such that the intersection of files in iOS and macOS are shared, similarly for Linux and Android. + +## FAQs + +### I don't understand this lint error, where do I get help? + +You can ask on the `hackers-engine` discord channel. Ping @gaaclarke or @zanderso if you don't get the response you want. + +### Hey, why are/aren't you checking for X? + +The checks that are enabled are negotiable. If you think we are missing something, please discuss it on `hacker-engine`. + +### Can I just use `NOLINT` to turn off the error? + +You can, but please get explicit approval to do from someone on the team. + +### How do I turn on a large new lint? + +Here are things that can make it easier to land large new lints. + +#### Tips + +* Try to reduce the number of checks you are turning on at a time, they can have cascading effects where fixing one check with "clang-tidy fix" can cause violations with other checks. +* Prefer using NOLINTNEXTLINE over NOLINT since auto formatting can move NOLINT comments and break them. + +#### Clang-tidy fix on CI + +Instead of running clang-tidy fix locally on your machines 4 times you can get the CI bots to print out the fix. Here's steps on how to do that: + +1. Edit `//ci/clang_tidy.sh`: + ```diff + # To run on CI, just uncomment the following line: + -# FLUTTER_LINT_PRINT_FIX=1 + +FLUTTER_LINT_PRINT_FIX=1 + ``` + This will run on _all_ files, and print out the patch generated by `clang-tidy fix` in the CI bots. + +2. Make a draft PR with the check added and FLUTTER_LINT_PRINT_FIX=1 +3. Look at the output of failed clang-tidy runs and make sure that it didn't garble the fix. Sometimes that happens and if it does, you'll have to manually fix where it garbled the fix or use NOLINTNEXTLINE. +4. Copy the patch the CI bot printed out to the clipboard, in the terminal, at the engine repo, use `git apply`, paste the patch, press ctrl+d then enter. Watch out that there can be overlapping patches between the mac runs and the linux runs since we don't shard across platforms. +5. Commit and push that. +6. When the 4 CI runs are green, remove FLUTTER_LINT_PRINT_FIX=1 and put up for review diff --git a/docs/ci/Engine-pre-submits-and-post-submits.md b/docs/ci/Engine-pre-submits-and-post-submits.md new file mode 100644 index 0000000000000..d137d97d616ba --- /dev/null +++ b/docs/ci/Engine-pre-submits-and-post-submits.md @@ -0,0 +1,79 @@ +The Flutter engine repo runs both pre-submit (before merging) and post-submit (after merging) suites of tests and checks, defined in [`.ci.yaml`](https://github.com/flutter/engine/blob/main/.ci.yaml). + +> [!TIP] +> See [Cocoon Scheduler](https://github.com/flutter/cocoon/blob/main/CI_YAML.md) for details. + +Failure to run appropriate tests for changes can (and do) result in the engine tree turning red, which in turn causes an expensive cascade of developers being blocked, investigative work, reverts, and roll-forwards. Where possible, all attempts should be made to run any/all tests _before_ merging a PR. See nuances (below) for exceptional cases. + + + +* [Pre-submit](#pre-submit) +* [Post-submit](#post-submit) + * [Running post-submits eagerly](#running-post-submits-eagerly) + +## Pre-submit + +Presubmits run (and are required to be passing) to merge a PR: + +Checks + +

+ +For example, the `linux_host_engine` target above runs based on the configuration in [`ci/builders/linux_host_engine.json`](https://github.com/flutter/engine/blob/458956228dad9837956aeb78b2988879e764a0b2/ci/builders/linux_host_engine.json). + +### Nuances + +Typically, pre-submits _always_ run on every PR, and don't need any special attention (other than keeping them green). There are two exceptions: + +1. Targets that provide a `runIf: ...` configuration +2. Changes that impact Clang Tidy + +> [!WARNING] +> +> `runIf: ...` is a powerful (but dangerous) feature that trades predictability for speed. +> +> `runIf` will skip certain targets if a particular file (or commonly, sets of files) are not changed in a given PR. +> +> For example, the [`linux_clang_tidy_presubmit`](https://github.com/flutter/engine/blob/991676f3bc9482eaaeb3764b6b835f0e3ff8b3c5/.ci.yaml#L219-L235) target will not run if only markdown (`*.md`) files are changed. + +Clang Tidy, on the other hand, is only run on _files that have changed in a given PR_. For example, if you have: + +```h +// impeller/a.h +struct A {} +``` + +... files that import that header, such as `impeller/foo/bar/baz.cc`, will _not_ be run in pre-submit. This means that changing (or updating the `DEPS` of libraries that provide) headers is _not_ a safe change, and will _not_ be detected in pre-submit. As an example, [#48705](https://github.com/flutter/engine/pull/48705) had to be reverted (despite passing all pre-submit checks), because the Clang Tidy _post-submit_ caught a failure. + +See [post-submit](#post-submit) below for options to run post-submits eagerly (i.e. as a pre-submit). + +## Post-submit + +Some (albeit fewer) targets are configured with the property `presubmit: false`: + +```yaml + - name: Mac mac_clang_tidy + recipe: engine_v2/engine_v2 + presubmit: false +``` + +These targets will _not_ show up during a PR, and will not be executed, but can (and do) turn the tree red. + +### Running post-submits eagerly + +We've intentionally chosen to make it _easier_ to land PRs, at the cost of turning the tree red periodically because post-submit checks catch something that the developer did not intend (or even know about). As a code author (or reviewer), you can optionally turn on post-submits to run eagerly (during pre-submit) by adding the label `test: all` (available only in the `flutter/engine` repo). + +Add the label, and push the PR (or a new commit, **the scheduler will not understand the label being added without a commit**): + +Screenshot 2023-12-07 at 1 55 48 PM + +

+ +For example, [#48158](https://github.com/flutter/engine/pull/48158) ran _all_ of the checks, including what is typically post-submits: + +Screenshot 2023-12-07 at 1 55 39 PM + +

+ +> [!WARNING] +> This increases the use of workers/capacity, and should be discouraged to be used on _all_ PRs. diff --git a/docs/contributing/Compiling-the-engine.md b/docs/contributing/Compiling-the-engine.md new file mode 100644 index 0000000000000..f209d34afad93 --- /dev/null +++ b/docs/contributing/Compiling-the-engine.md @@ -0,0 +1,330 @@ +_If you've never built the engine before, first see [Setting up the Engine development environment](Setting-up-the-Engine-development-environment.md)._ + +# Contents + +Depending on the platform you are making changes for, you may be interested in all or only some of the sections below: + +* [General Compilation Tips](#general-compilation-tips) +* [Using a custom Dart SDK](#using-a-custom-dart-sdk) +* [Compiling for Android](#compiling-for-android-from-macos-or-linux) +* [Compiling for iOS (from macOS)](#compiling-for-ios-from-macos) +* [Compiling for macOS or Linux](#compiling-for-macos-or-linux) +* [Compiling for Windows](#compiling-for-windows) +* [Compiling for Fuchsia](#compiling-for-fuchsia) +* [Compiling for the Web](#compiling-for-the-web) +* [Compiling for testing](#compiling-for-testing) + +## General Compilation Tips + +- For local development and testing, it's generally preferable to use `--unopt` builds. + These builds will have additional logging and checks enabled, and generally use build + and link flags that lead to faster compilation and better debugging symbols. + If you are trying to do performance testing with a local build, do not use the `--unopt` + flag. +- Link Time Optimization: Optimized builds also perform Link Time Optimization of all + binaries. This makes the linker take a lot of time and memory to produce binaries. If + you need optimized binaries but don't want to perform LTO, add the `--no-lto` flag. +- Android and iOS expect both a `host` and `android` (or `ios`) build. It is critical to + recompile the host build after upgrading the Dart SDK (e.g. via a `gclient sync` after + merging up to head), since artifacts from the host build need to be version matched to + artifacts in the Android/iOS build. +- Web, Desktop, and Fuchsia builds have only one build target (i.e. `host` or `fuchsia`). +- Make sure to exclude the `out` directory from any backup scripts, as many large binary + artifacts are generated. This is also generally true for all of the directories outside + of the `engine/src/flutter` directory. + +## Using a custom Dart SDK + +When targeting the host and desktop, on CI we use a pre-built Dart SDK vended by the Dart team. +To build and use the SDK from the Dart sources downloaded by `gclient sync`, after editing those +source files, pass the flag `--no-prebuilt-dart-sdk` to `//flutter/tools/gn`. + +## Compiling for Android (from macOS or Linux) + +These steps build the engine used by `flutter run` for Android devices. + +Run the following steps, from the `src` directory created in [Setting up the Engine development environment](Setting-up-the-Engine-development-environment.md): + +1. `git pull upstream main` in `src/flutter` to update the Flutter Engine repo. + +2. `gclient sync` to update dependencies. + +3. Prepare your build files + * `./flutter/tools/gn --android --unoptimized` for device-side executables. + * `./flutter/tools/gn --android --android-cpu arm64 --unoptimized` for newer 64-bit Android devices. + * `./flutter/tools/gn --android --android-cpu x86 --unoptimized` for x86 emulators. + * `./flutter/tools/gn --android --android-cpu x64 --unoptimized` for x64 emulators. + * `./flutter/tools/gn --unoptimized` for host-side executables, needed to compile the code. + * On Apple Silicon ("M" chips), add `--mac-cpu arm64` to avoid using emulation. This will generate `host_debug_unopt_arm64`. + +> 💡 **TIP**: When developing on a Mac with ARM (M CPU), prefer `host_debug_unopt_arm64`. +> +> You can continue to use `host_debug_unopt` (required for Intel Macs), but the engine will be run under Rosetta +> which may be slower. See [Developing with Flutter on Apple Silicon](../../platforms/desktop/macos/Developing-with-Flutter-on-Apple-Silicon.md) +> for more information. + +4. Build your executables + * `ninja -C out/android_debug_unopt` for device-side executables. + * `ninja -C out/android_debug_unopt_arm64` for newer 64-bit Android devices. + * `ninja -C out/android_debug_unopt_x86` for x86 emulators. + * `ninja -C out/android_debug_unopt_x64` for x64 emulators. + * `ninja -C out/host_debug_unopt` (or `ninja -C out/host_debug_unopt_arm64`, see above) for host-side executables. + * These commands can be combined. Ex: `ninja -C out/android_debug_unopt && ninja -C out/host_debug_unopt` + * For MacOS, you will need older version of XCode(9.4 or below) to compile android_debug_unopt and android_debug_unopt_x86. If you only care about x64, you can ignore this + +This builds a debug-enabled ("unoptimized") binary configured to run Dart in +checked mode ("debug"). There are other versions, see [Flutter's modes](../Flutter's-modes.md). + +If you're going to be debugging crashes in the engine, make sure you add +`android:debuggable="true"` to the `` element in the +`android/AndroidManifest.xml` file for the Flutter app you are using +to test the engine. + +See [The flutter tool](../../tool/README.md) for instructions on how to use the `flutter` tool with a local engine. +You will typically use the `android_debug_unopt` build to debug the engine on a device, and +`android_debug_unopt_x64` to debug in on a simulator. Modifying dart sources in the engine will +require adding a `dependency_override` section in you app's `pubspec.yaml` as detailed +[here](../../tool/README.md#using-a-locally-built-engine-with-the-flutter-tool). + +Note that if you use particular android or ios engine build, you will need to have corresponding +host build available next to it: if you use `android_debug_unopt`, you should have built `host_debug_unopt`, +`android_profile` -> `host_profile`, etc. One caveat concerns cpu-flavored builds like `android_debug_unopt_x86`: you won't be able to build `host_debug_unopt_x86` as that configuration is not supported. What you are expected to do is to build `host_debug_unopt` and symlink `host_debug_unopt_x86` to it. + +### Compiling everything that matters on Linux + +The following script will update all the builds that matter if you're developing on Linux and testing on Android and created the `.gclient` file in `~/dev/engine`: + +```bash +set -ex + +cd ~/dev/engine/src/flutter +git fetch upstream +git rebase upstream/main +gclient sync +cd .. + +flutter/tools/gn --unoptimized --runtime-mode=debug +flutter/tools/gn --android --unoptimized --runtime-mode=debug +flutter/tools/gn --android --runtime-mode=profile +flutter/tools/gn --android --runtime-mode=release + +cd out +find . -mindepth 1 -maxdepth 1 -type d | xargs -n 1 sh -c 'ninja -C $0 || exit 255' +``` +For `--runtime-mode=profile` build, please also consider adding `--no-lto` option to the `gn` command. It will make linking much faster with a small sacrifice on the binary size and memory usage (which probably doesn't matter for debugging or performance benchmark purposes.) + +## Compiling for iOS (from macOS) + +These steps build the engine used by `flutter run` for iOS devices. + +Run the following steps, from the `src` directory created in the steps above: + +1. `git pull upstream main` in `src/flutter` to update the Flutter Engine repo. + +2. `gclient sync` to update dependencies. + +3. `./flutter/tools/gn --ios --unoptimized` to prepare build files for device-side executables (or `--ios --simulator --unoptimized` for simulator). + * This also produces an Xcode project for working with the engine source code at `out/ios_debug_unopt/flutter_engine.xcodeproj` + * For a discussion on the various flags and modes, see [Flutter's modes](../Flutter's-modes.md). + * Add the `--simulator-cpu=arm64` argument for an arm64 Mac simulator to output to `out/ios_debug_sim_unopt_arm64`. + +4. `./flutter/tools/gn --unoptimized` to prepare the build files for host-side executables. + * On Apple Silicon ("M" chips), add `--mac-cpu arm64` to avoid using emulation. This will generate `host_debug_unopt_arm64`. + +5. `ninja -C out/ios_debug_unopt && ninja -C out/host_debug_unopt` to build all artifacts (use `out/ios_debug_sim_unopt` for Simulator). + +See [The flutter tool](../../tool/README.md) for instructions on how to use the `flutter` tool with a local engine. +You will typically use the `ios_debug_unopt` build to debug the engine on a device, and +`ios_debug_sim_unopt` to debug in on a simulator. Modifying dart sources in the engine will +require adding a `dependency_override` section in you app's `pubspec.yaml` as detailed +[here](../../tool/README.md#using-a-locally-built-engine-with-the-flutter-tool). + +See also [instructions for debugging the engine in a Flutter app in Xcode](../Debugging-the-engine.md#debugging-ios-builds-with-xcode). + +## Compiling for macOS or Linux + +These steps build the desktop embedding, and the engine used by `flutter test` on a host workstation. + +1. `git pull upstream main` in `src/flutter` to update the Flutter Engine repo. + +2. `gclient sync` to update your dependencies. + +3. `./flutter/tools/gn --unoptimized` to prepare your build files. + * `--unoptimized` disables C++ compiler optimizations. On macOS, binaries are emitted unstripped; on Linux, unstripped binaries are emitted to an `exe.unstripped` subdirectory of the build. + +4. `ninja -C out/host_debug_unopt` to build a desktop unoptimized binary. + * If you skipped `--unoptimized`, use `ninja -C out/host_debug` instead. + +See [The flutter tool](../../tool/README.md) for instructions on how to use the `flutter` tool with a local engine. +You will typically use the `host_debug_unopt` build in this setup. Modifying dart sources in the engine will +require adding a `dependency_override` section in you app's `pubspec.yaml` as detailed +[here](../../tool/README.md#using-a-locally-built-engine-with-the-flutter-tool). + + +## Compiling for Windows + +> [!WARNING] +> You can only build selected binaries on Windows (mainly `gen_snapshot` and the desktop embedding). + +On Windows, ensure that the engine checkout is not deeply nested. This avoid the issue of the build scripts working with excessively long paths. + +1. Make sure you have Visual Studio installed (non-Googlers only). [Debugging Tools for Windows 10](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools#small-classic-windbg-preview-logo-debugging-tools-for-windows-10-windbg) must be installed. + +2. `git pull upstream main` in `src/flutter` to update the Flutter Engine repo. + +3. Ensure long path support is enabled on your machine. Launch PowerShell as an administrator and run: +``` +Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -Force +``` + +4. If you are not a Google employee, you must set the following environment variables to point the depot tools at Visual Studio: +```shell +DEPOT_TOOLS_WIN_TOOLCHAIN=0 +GYP_MSVS_OVERRIDE_PATH="C:\Program Files (x86)\Microsoft Visual Studio\2019\Community" # (or your location for Visual Studio) +WINDOWSSDKDIR="C:\Program Files (x86)\Windows Kits\10" # (or your location for Windows Kits) +``` +Also, be sure that Python27 is before any other python in your Path. + +5. `gclient sync` to update your dependencies. + +6. switch to `src/` directory. + +7. `python .\flutter\tools\gn --unoptimized` to prepare your build files. + * If you are only building `gen_snapshot`: `python .\flutter\tools\gn [--unoptimized] --runtime-mode=[debug|profile|release] [--android]`. + +8. `ninja -C .\out\

` to build. + * If you used a non-debug configuration, use `ninja -C .\out\ gen_snapshot`. + Release and profile are not yet supported for the desktop shell. + +## Compiling for Fuchsia + +### Build components for Fuchsia + +1. Building fuchsia is only supported on linux. You need to run `gclient config --custom-var=download_fuchsia_deps=True` then `gclient sync`. + +It will set `"download_fuchsia_deps": True` in `"custom_vars"` section in `.gclient` file, and download necessary binaries to build fuchsia components. + +2. If you'd like to run tests locally, also run `gclient config --custom-var=run_fuchsia_emu=True` then `gclient sync`. + +It will set `"run_fuchsia_emu": True` in `"custom_vars"` section in `.gclient` file, and download necessary binaries and images to run tests on fuchsia emulators. +You can set both `custom_vars` and run `gclient sync` only once. + +You will also need kvm enabled, or nested virtualization on the gcloud VMs. Fuchsia and the tests will all be executed on the qemu. + +3. Prepare and build + +``` +./flutter/tools/gn --fuchsia --no-lto +``` + + * It will create a `out/fuchsia_debug_x64`. + * Use `--fuchsia-cpu arm64` to build components for arm64. It will be created in a folder `out/fuchsia_debug_arm64`. + * Use `--runtime-mode=release` or `--runtime-mode=profile` to select other profiles as other platforms. + * Ignore `--no-lto` to use lto or link-time optimization. + +``` +ninja -C out/fuchsia_debug_x64 -k 0 +``` + + * It builds all but ignores known errors. + * Or specify following targets to avoid using `-k 0`. + +``` +flutter/shell/platform/fuchsia:fuchsia \ +flutter/shell/platform/fuchsia/dart_runner:dart_runner_tests \ +fuchsia_tests +``` + + * Use `autoninja` if it's available. + * `-C out/fuchsia_release_x64` for release build; other configurations are similar with a different folder name in `out/`. + +4. Run all tests locally + +``` +python3 flutter/tools/fuchsia/with_envs.py flutter/testing/fuchsia/run_tests.py +``` + + * It runs the tests in `out/fuchsia_debug_x64` by default. According to the configuration, it may take 5 minutes with regular gtest output to the terminal. + * Add `fuchsia_release_x64` at the end of the command for release build; other configurations are similar with a different folder name in `out/`. + +## Compiling for the Web + +For building the engine for the Web we use the [felt](https://github.com/flutter/engine/blob/main/lib/web_ui/README.md) tool. + +To test Flutter with a local build of the Web engine, add `--local-web-sdk=wasm_release` to your `flutter` command, e.g.: + +``` +flutter run --local-web-sdk=wasm_release -d chrome +flutter test --local-web-sdk=wasm_release test/path/to/your_test.dart +``` + +## Compiling for the Web on Windows + +Compiling the web engine might take a few extra steps on Windows. Use cmd.exe and "run as administrator". + +1. Make sure you have Visual Studio installed. Set the following environment variables. For Visual Studio use the path of the version you installed. + * `GYP_MSVS_OVERRIDE_PATH = "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community"` + * `GYP_MSVS_VERSION = 2017` +2. Make sure, depot_tools, ninja and python are installed and added to the path. Also set the following environment variable for depot tools: + * `DEPOT_TOOLS_WIN_TOOLCHAIN = 0` + * Tip: if you get a python error try to use Python 2 instead of 3 +3. `git pull upstream main` in `src/flutter` to update the Flutter Engine repo. +4. `gclient sync` to update your dependencies. + * Tip: If you get a git authentication errors on this step try Git Bash instead +5. `python .\flutter\tools\gn --unoptimized --full-dart-sdk` to prepare your build files. +6. `ninja -C .\out\` to build. + +To test Flutter with a local build of the Web engine, add `--local-web-sdk=wasm_release` to your `flutter` command, e.g.: + +``` +flutter run --local-web-sdk=wasm_release -d chrome +flutter test --local-web-sdk=wasm_release test/path/to/your_test.dart +``` + +For testing the engine again use [felt](https://github.com/flutter/engine/blob/main/lib/web_ui/README.md) tool +this time with felt_windows.bat. + +``` +felt_windows.bat test +``` + +## Compiling for testing + +### Dart tests + +To run dart tests, build the engine: + +``` +flutter/tools/gn --unoptimized +ninja -C out/host_debug_unopt/ +``` + +execute `run_tests` for native: +``` +python3 flutter/testing/run_tests.py --type dart +``` + +and `felt` for web: +``` +cd flutter/lib/web_ui +dev/felt test [test file] +``` + + +## Troubleshooting Compile Errors + +### Version Solving Failed + +From time to time, as the Dart versions increase, you might see dependency errors such as: + +``` +The current Dart SDK version is 2.7.0-dev.0.0.flutter-1ef444139c. + +Because ui depends on 1.0.0 which requires SDK version >=2.7.0 <3.0.0, version solving failed. +``` + +Running `gclient sync` does not update the tags, there are two solutions: +1. under `engine/src/third_party/dart` run `git fetch --tags origin` +2. or run gclient sync with with tags parameter: `gclient sync --with_tags` + +_See also: [Debugging the engine](../Debugging-the-engine.md), which includes instructions on running a Flutter app with a local engine._ diff --git a/docs/contributing/Setting-up-the-Engine-development-environment.md b/docs/contributing/Setting-up-the-Engine-development-environment.md new file mode 100644 index 0000000000000..6bf60feeb019c --- /dev/null +++ b/docs/contributing/Setting-up-the-Engine-development-environment.md @@ -0,0 +1,176 @@ +_If you've already built the engine and have the configuration set up but merely need a refresher on +actually compiling the code, see [Compiling the engine](Compiling-the-engine.md)._ + +_If you are checking these instructions to refresh your memory and your fork of the engine is stale, +make sure to merge up to HEAD before doing a `gclient sync`._ + +# Getting dependencies + +Make sure you have the following dependencies available: + + * A Linux, macOS, or Windows host + * Linux supports cross-compiling artifacts for Android and Fuchsia, but not iOS. + * macOS supports cross-compiling artifacts for Android and iOS. + * Windows doesn't support cross-compiling artifacts for any of Android, Fuchsia, or iOS. + * `git` (used for source version control). + * An ssh client (used to authenticate with GitHub). + * `python3` (used by many of our tools, including `gclient`). + * **Chromium's + [depot_tools](https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up)** (Which includes gclient) + * Add the `depot_tools` directory to the *front* of your `PATH`. + * On macOS and Linux: `curl` and `unzip` (used by `gclient sync`). + * On Linux: The `pkg-config` package. + * On Windows: + - Visual Studio 2017 or later (required for non-Googlers only). + - [Windows 10 SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/) (required for non-Googlers only). Be sure to install the "Debugging Tools for Windows" feature. + * On macOS: + - Install the latest Xcode. + - On Apple Silicon arm64 Macs, install the Rosetta translation environment by running `softwareupdate --install-rosetta`. + - Install Oracle's Java JDK, version 1.8 or later. + +You do not need to install [Dart](https://www.dartlang.org/downloads/linux.html). +A Dart toolchain is automatically downloaded as part of the "Getting the source" +step. Similarly for the Android SDK, it is downloaded by the `gclient sync` step below. + +## Getting the source + +Run the following steps to set up your environment: + +> [!IMPORTANT] +> Non-Googler Windows users should set the following environment variables to point +> `depot_tools` to their Visual Studio installation directory: +> * `DEPOT_TOOLS_WIN_TOOLCHAIN=0` +> * `GYP_MSVS_OVERRIDE_PATH=C:\Program Files\Microsoft Visual Studio\2022\Community` +> * Use the path of your installation. + +Create a new directory to hold the source code and move into it. Here, we are using the "engine" directory. +```sh +mkdir engine; cd engine; +``` + +> [!IMPORTANT] +> On Windows, the following must be run as an Administrator due to [a known issue](https://github.com/flutter/flutter/issues/94580). + +Fetch the Flutter engine sources. This may take a while on a slow connection. Do **not** interrupt this process. Otherwise, a partial checkout cannot be resumed and you'll have to delete all the files including the hidden files in the engine directory and start over. +```sh +fetch flutter +``` +The [Flutter Engine](https://github.com/flutter/engine) repository resides at `src/flutter`. The convention is to refer to this repository as `upstream`. + +```sh +git -C src/flutter remote rename origin upstream +``` + +Optionally, if you are working with a fork of the engine, add that as a Git remote. + +```sh +git -C src/flutter remote add origin +``` + +The "Engine Tool" called `et` is useful when working with the engine. It is located in the `flutter/bin` directory in the source checkout. Add this to your `$PATH` in your `.rc`. + +### Additional Steps for Web Engine + +Amend the generated `.gclient` file in the root of the source directory to add the following: +``` +solutions = [ + { + # Same as above... + "custom_vars": { + "download_emsdk": True, + }, + }, +] +``` + +Now, run + +```sh +gclient sync +``` + +## Next steps: + + * [Compiling the engine](Compiling-the-engine.md) explains how to actually get builds, now that you have the code. + * [The flutter tool](../../tool/README.md) has a section explaining how to use custom engine builds. + * [Signing commits](../../contributing/Signing-commits.md), to configure your environment to securely sign your commits. + +## Editor autocomplete support + +### Xcode [Objective-C++] + +On Mac, you can simply use Xcode (e.g., `open out/host_debug_unopt/products.xcodeproj`). + +### VSCode with C/C++ Intellisense [C/C++] + +VSCode can provide some IDE features using the [C/C++ extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools). It will provide basic support on install without needing any additional configuration. There will probably be some issues, like header not found errors and incorrect jump to definitions. + +Intellisense can also use our `compile_commands.json` for more robust functionality. Either symlink `src/out/compile_commands.json` to the project root at `src` or provide an absolute path to it in the `c_cpp_properties.json` config file. See ["compile commands" in the c_cpp_properties.json reference](https://code.visualstudio.com/docs/cpp/c-cpp-properties-schema-reference). This will likely resolve the basic issues mentioned above. + +For example, in `src/.vscode/settings.json`: + +```json +{ + "clangd.path": "buildtools/mac-arm64/clang/bin/clangd", + "clangd.arguments": [ + "--compile-commands-dir=out/host_debug_unopt_arm64" + ], + "clang-format.executable": "buildtools/mac-arm64/clang/bin/clang-format" +} +``` + +... which is built with: + +```shell +# M1 Mac (host_debug_unopt_arm64) +./tools/gn --unopt --mac-cpu arm64 --enable-impeller-vulkan --enable-impeller-opengles --enable-unittests +``` + +For adding IDE support to the Java code in the engine with VSCode, see ["Using VSCode as an IDE for the Android Embedding"](#using-vscode-as-an-ide-for-the-android-embedding-java). + +### Zed Editor + +[Zed](https://zed.dev/) can be used to edit C++ code in the Engine. To enable analysis and auto-completion, symlink `src/out/compile_commands.json` to the project root at `src`. + +### cquery/ccls (multiple editors) [C/C++/Objective-C++] + +Alternatively, [cquery](https://github.com/cquery-project/cquery) and a derivative [ccls](https://github.com/MaskRay/ccls) are highly scalable C/C++/Objective-C language server that supports IDE features like go-to-definition, call hierarchy, autocomplete, find reference etc that works reasonably well with our engine repo. + +They(https://github.com/cquery-project/cquery/wiki/Editor-configuration) [supports](https://github.com/MaskRay/ccls/wiki/Editor-Configuration) editors like VSCode, emacs, vim etc. + +To set up: +1. Install cquery + 1. `brew install cquery` or `brew install ccls` on osx; or + 1. [Build from source](https://github.com/cquery-project/cquery/wiki/Getting-started) +1. Generate compile_commands.json which our GN tool already does such as via `src/flutter/tools/gn --ios --unoptimized` +1. Install an editor extension such as [VSCode-cquery](https://marketplace.visualstudio.com/items?itemName=cquery-project.cquery) or [vscode-ccls](https://marketplace.visualstudio.com/items?itemName=ccls-project.ccls) + 1. VSCode-query and vscode-ccls requires the compile_commands.json to be at the project root. Copy or symlink `src/out/compile_commands.json` to `src/` or `src/flutter` depending on which folder you want to open. + 1. Follow [Setting up the extension](https://github.com/cquery-project/cquery/wiki/Visual-Studio-Code#setting-up-the-extension) to configure VSCode-query. + +![](https://media.giphy.com/media/xjIrToRDVvMPvjkBcl/giphy.gif) + +### Using VSCode as an IDE for the Android Embedding [Java] + +1. Install the extensions vscjava.vscode-java-pack (Extension Pack for Java) and vscjava.vscode-java-dependency (Project Manager for Java). + +1. Right click on the `shell/platform/android` folder in the engine source and click on `Add Folder to Java Source Path`. This creates an anonymous workspace and turns those files from ["syntax mode"](https://code.visualstudio.com/docs/java/java-project#_syntax-mode) to "compile mode". At this point, you should see a lot of errors since none of the external imports are found. + +1. Find the "Java Dependencies" pane in your Explorer view. Use the "Explorer: Focus on Java Dependencies View" command if hidden. + +1. Refresh the view and find the "flutter_*" project. There should be a "_/shell/platform/android" source folder there. + +1. In the "Referenced Libraries" sibling node, click the + button, navigate to `engine/src/third_party/android_embedding_dependencies` and add the entire folder. This is the equivalent of adding + ``` + "java.project.referencedLibraries": [ + "{path to engine}/src/third_party/android_embedding_dependencies/lib/**/*.jar" + ] + ``` + to your VSCode's settings.json for your user or for your workspace. + +1. If you previously had a `shell/platform/android/.classpath`, delete it. + +## VSCode Additional Useful Configuration + +1. Create [snippets](https://code.visualstudio.com/docs/editor/userdefinedsnippets) for header files with [this configuration](https://github.com/chromium/chromium/blob/master/tools/vscode/settings.json5). This will let you use `hdr` keyboard macro to create the boiler plate header code. Also consider some of [these settings](https://github.com/chromium/chromium/blob/master/tools/vscode/settings.json5) and [more tips](https://chromium.googlesource.com/chromium/src/+show/lkgr/docs/vscode.md). + +2. To format GN files on save, [consider using this extension](https://marketplace.visualstudio.com/items?itemName=persidskiy.vscode-gnformat). \ No newline at end of file diff --git a/docs/impeller/Flutter-GPU.md b/docs/impeller/Flutter-GPU.md new file mode 100644 index 0000000000000..676f0b426b661 --- /dev/null +++ b/docs/impeller/Flutter-GPU.md @@ -0,0 +1,49 @@ +**Flutter GPU** (previously referred to as "Dart GPU" or "Impeller Dart") is an effort to expose a low level graphics API in the Flutter Framework. + +Design doc: https://flutter.dev/go/impeller-dart + +Flutter GPU's runtime is a thin wrapper over [Impeller](README.md)'s HAL, from which custom renderers may be entirely built using Dart. Just like with Impeller, Flutter GPU shader bundles are compiled ahead of time using [impellerc](https://github.com/flutter/engine/tree/main/impeller/compiler). As such, Flutter GPU is only available on platforms that support Impeller. + +## Dart FFI + +Under the hood, the API communicates with Flutter Engine via Dart FFI, calling symbols publicly exported by libflutter and/or embedders. These symbols are prefixed with `InternalFlutterGpu`, and are considered unstable. Direct usage of the exported symbols is not supported and will break without notice; the only supported way to use Flutter GPU is by importing `package:flutter_gpu`. + +## Try out Flutter GPU + +Once released, Flutter GPU will be shipped as part of the Flutter SDK in the form of a Dart package called `flutter_gpu`. An early implementation of the `flutter_gpu` package is being developed under the [`lib/gpu` directory](https://github.com/flutter/engine/tree/main/lib/gpu) of the Flutter Engine repository. + +> [!CAUTION] +> _All_ aspects of Flutter GPU are subject to breakage or removal at any time without prior deprecation notice or viable feature replacement. DO NOT rely on Flutter GPU for production projects at this time, but DO have fun playing with it and sharing your experiments with the community. + +Flutter GPU is currently unfinished, extremely experimental, and not well documented. [bdero](https://github.com/bdero) is actively developing and testing Flutter GPU against the MacOS desktop embedder; shader compilation and import likely don't function correctly on other platforms yet. However, if you wish to experiment with Flutter GPU, it is possible to do so without a custom Engine build: + +1. Update your Flutter checkout to the latest version in the [master channel](https://docs.flutter.dev/release/upgrade#other-channels). +1. Clone [Flutter Engine](https://github.com/flutter/engine) and checkout the Engine commit that the Flutter master channel is currently pinned to. This can be found in the [`bin/internal/engine.version` file](https://github.com/flutter/flutter/blob/main/bin/internal/engine.version) of the main Flutter repository. + ```sh + git clone https://github.com/flutter/engine.git + cd engine + git reset --hard [PINNED_ENGINE_COMMIT] + ``` +1. Create a new Flutter project using the Flutter tool and add `flutter_gpu` as a dependency in `pubspec.yaml` with a local path pointing to the `lib/gpu` directory within the Flutter Engine repository cloned in step 2. For example: + ```yaml + dependencies: + flutter: + sdk: flutter + flutter_gpu: + path: ../engine/src/flutter/lib/gpu + ``` +1. From here, you can import the API and begin using it. + ```dart + import 'package:flutter_gpu/gpu.dart' as gpu; + ``` + Check out this [examples repository](https://github.com/bdero/flutter-gpu-examples), which includes an example of [drawing a triangle](https://github.com/bdero/flutter-gpu-examples/blob/master/lib/triangle.dart), among other things. + +## Reporting bugs + +If you run into issues while using Flutter GPU, please file a bug using the standard [bug report template](https://github.com/flutter/flutter/issues/new?template=2_bug.yml). Additionally, mention "Flutter GPU" in the title, label the bug with the `e: impeller` label, and tag [bdero](https://github.com/bdero) in the issue description. + +## Questions or feedback? + +If you have non-bug report questions surrounding Flutter GPU, there are several ways you can reach out to the developer: +* Create a thread in the #help channel of the [Discord server](../../contributing/Chat.md). Place "Flutter GPU" in the title of the thread and tag @bdero in the message. +* Send a Twitter DM to [@algebrandon](https://twitter.com/algebrandon). diff --git a/docs/impeller/Impeller-Scene.md b/docs/impeller/Impeller-Scene.md new file mode 100644 index 0000000000000..dd09db8c79019 --- /dev/null +++ b/docs/impeller/Impeller-Scene.md @@ -0,0 +1,19 @@ +We are excited to have you tinker on [the Impeller Scene Demo presented at Flutter Forward](https://www.youtube.com/live/zKQYGKAe5W8?feature=share&t=7048). While we spend time learning the use-cases and finalizing the API, the functionality for Impeller Scene is behind a compile-time flag. During this time, there are no guarantees around API stability. + +**Compiling the Engine** + +- Configure your Mac host to compile the Flutter Engine by [following the guidance in wiki](../contributing/Setting-up-the-Engine-development-environment.md). +- Ensure that you are on the [main branch of the Flutter Engine](https://github.com/flutter/engine/tree/main). +- Ensure that you are on the [main branch of the Flutter Framework](https://github.com/flutter/flutter/tree/main). +- Configure the host build: `./flutter/tools/gn --enable-impeller-3d --no-lto` +- Configure the iOS build: `./flutter/tools/gn --enable-impeller-3d --no-lto --ios` + - Add the `--simulator --simulator-cpu=arm64` flag to the iOS build if you are going to test on the simulator. +- Build host artifacts (this will take a while): `ninja -C out/host_debug` +- Build iOS artifacts (this will take a while): `ninja -C out/ios_debug` + - If targeting the simulator: `ninja -C out/ios_debug_sim_arm64` +- Clone the demo repository: `git clone https://github.com/bdero/flutter-scene-example.git` and move into the directory. +- Plug in your device or open `Simulator.app`, then run `flutter devices` to note the device identifier. +- Run the demo application: `flutter run -d [device_id] --local-engine ios_debug --local-engine-host host_debug` (or `ios_debug_sim_arm64` if you are running on the Simulator). + - On Silicon Macs, prefer `--local-engine-host host_debug_arm64` (adjusting your `ninja` command above accordingly) + +We hope to continue evolving the API and have it available on the stable channel soon! diff --git a/docs/impeller/README.md b/docs/impeller/README.md new file mode 100644 index 0000000000000..0f04e700ad935 --- /dev/null +++ b/docs/impeller/README.md @@ -0,0 +1,7 @@ +_For user information about Flutter's new rendering backend, check out [Impeller preview](https://docs.flutter.dev/perf/impeller)._ + +Team-facing documentation specific to Impeller: + +- [Setting up MoltenVK on macOS for Impeller](Setting-up-MoltenVK-on-macOS-for-Impeller.md) +- [Flutter GPU](Flutter-GPU.md) +- [Instructions for playing with the Impeller Scene demo|Impeller Scene](Impeller-Scene.md) diff --git a/docs/impeller/Setting-up-MoltenVK-on-macOS-for-Impeller.md b/docs/impeller/Setting-up-MoltenVK-on-macOS-for-Impeller.md new file mode 100644 index 0000000000000..2c0231e137c78 --- /dev/null +++ b/docs/impeller/Setting-up-MoltenVK-on-macOS-for-Impeller.md @@ -0,0 +1,7 @@ +- Get the MoltenVK SDK from https://vulkan.lunarg.com/sdk/home#mac. +- Make sure to check off `System Global Installation`: +image + +- When running `flutter/tools/gn`, add the `--impeller-enable-vulkan` flag, e.g. `./flutter/tools/gn --impeller-enable-vulkan --unopt --mac-cpu arm64` + +You should now be able to build and run the Vulkan host tests, e.g. `out/host_debug_unopt_arm64/impeller_unittests --gtest_filter="*Vulkan*"`. diff --git a/docs/release/Code-signing-metadata.md b/docs/release/Code-signing-metadata.md new file mode 100644 index 0000000000000..4efd26a13ad49 --- /dev/null +++ b/docs/release/Code-signing-metadata.md @@ -0,0 +1,196 @@ +# Code signing + +This covers the process of how to add / update code signing metadata of flutter +engine binaries. + +## Overview + +Flutter engine binaries are built with GN and ninja, referencing pre-defined +configurations such as ci/builders +[JSON files](https://github.com/flutter/engine/blob/main/ci/builders/mac_host_engine.json). +During flutter releases, engineers need to code sign mac engine binaries to +assure users that they come from a known source, have not been tampered with, +and should not be quarantined by Gatekeepers. + +Each of the Flutter engine binaries are either code signed with entitlements, or +code signed without entitlements. (An entitlement, along with information from +the developer account, grant particular permissions to binaries, such as +capability to access the user's home automation network.) For example, impellerc +is code signed with flutter entitlements, whereas .dylib files are usually code +signed without entitlements. + +## Add / Update code signing metadata + +### Glossary + +1. BUILD.gn files: files that include build rules of GN targets. An example is + the + [BUILD.gn file of flutter engine](https://github.com/flutter/engine/blob/main/BUILD.gn). +2. leaf node of an engine binary: the minimal gn target that could produce such + an engine binary. That is, this target does not have any dependencies on + other gn targets that could build this engine binary. +3. dependencies: Every gn target could have dependencies on other gn targets. + The dependency of a gn target is defined in the `deps` field of the target's + build rule. + +### ways to generate engine binary + +Generally, there are two ways to generate an engine binary: + +1. Through build rules defined in BUILD.gn files. + +2. Through global generator scripts. (these scripts are normally .py files) + +To distinguish between the two, an engine binary is built through global +generator if it is listed in the `archives` -> `destination` field of the +builder JSON +([mac_ios_engine.json](https://github.com/flutter/engine/blob/main/ci/builders/mac_ios_engine.json) +or +[mac_host_engine.json](https://github.com/flutter/engine/blob/main/ci/builders/mac_host_engine.json)). +For example, `darwin-x64/FlutterEmbedder.framework.zip`. Whereas binaries built +with BUILD.gn files are listed among the `builds` field of the JSON file. For +example, `darwin-x64/artifacts.zip`. We will provide examples for both +scenarios. + +### To add / update code signing metadata in BUILD.gn files: + +1. Find the leaf node where the target engine binary is built. To do so, + Recursively trace the `deps` field of the engine artifact. The paths in + `deps` field of the GN target correspond to the paths of other GN targets + that are dependencies of the current GN target. + +2. Add / Update the `metadata` field of the leaf node. For a new engine binary: + + 2.1 if it should be code signed with entitlements, add [the name of the + engine binary] to the `entitlement_file_path` field in `metadata` . + + 2.2 if the binary shouldn't be code signed with entitlements, add [the name + of the engine binary] to the `without_entitlement_file_path` field in + `metadata` . + +3. If a `entitlement_file_path` or a `without_entitlement_file_path` field does + not exist: + + **note**: this step is only needed if the target includes solely binaries + that have never been code signed before. This step also requires some + background on flutter engine and gn build rules. + + Add a `metadata` field in the gn target of the leaf node, and put the name + of the binary in this field. e.g. + + ``` + metadata = { + entitlement_file_path = [ "libtessellator.dylib" ] + } + ``` + + In the same file that produces the engine artifact(zip file), add a build + rule to collect the data keys. e.g. + + ``` + generated_file("artifacts_entitlement_config") { + outputs = [ "$target_gen_dir/entitlements.txt" ] + + data_keys = [ "entitlement_file_path" ] + + deps = [ "//flutter/lib/snapshot:generate_snapshot_bin" ] + if (flutter_runtime_mode == "debug") { + deps += [ + "//flutter/impeller/compiler:impellerc", + "//flutter/impeller/tessellator:tessellator_shared", + "//flutter/shell/testing:testing", + "//flutter/tools/path_ops:path_ops", + ] + } + } + ``` + + Finally, embed the file with collected data keys in the zip artifact. e.g. + + ``` + if (host_os == "mac") { + deps += [ ":artifacts_entitlement_config" ] + files += [ + { + source = "$target_gen_dir/entitlements.txt" + destination = "entitlements.txt" + }, + ] + } + ``` + +#### Example + +Suppose impellerc is a binary that exist in a zip bundle called artifacts.zip. +Then impellerc is the name of the binary, and artifacts.zip is the flutter +engine artifact. + +1. Following step 1, the `deps` field of the GN target of artifacts.zip + includes the path of impeller dependency: + `//flutter/impeller/compiler:impellerc`. Following this path, we locate the + GN file at `flutter/impeller/compiler/BUILD.gn`, and find the leaf node that + builds impellerc: `impeller_component("impellerc")`. + +2. Following step 2, since `impellerc` should be code signed with entitlements, + we go to the `metadata` field of the impellerc target, and add the name + `impellerc` to the `entitlement_file_path` array inside the `metadata` + field. + +You can reference the +[BUILD.gn file of impellerc](https://github.com/flutter/engine/blob/main/impeller/compiler/BUILD.gn). + +### To add / update code signing metadata in global generator files: + +1. Find the generator script path listed under `generators` -> `tasks` -> + `script` of the ci/builder JSON files + ([mac_ios_engine.json](https://github.com/flutter/engine/blob/main/ci/builders/mac_ios_engine.json) + or + [mac_host_engine.json](https://github.com/flutter/engine/blob/main/ci/builders/mac_host_engine.json)). + + The generator script related to iOS is located at + `sky/tools/create_full_ios_framework.py`, and generator script related to + macOS is located at `sky/tools/create_macos_framework.py`. + +2. Add / Update the variables ending with `with_entitlements` / + `without_entitlements` suffix from the generator script you found in step + one. + + As an example, you can find variables `ios_file_without_entitlements` and + `ios_file_with_entitlements` in sky/tools/create_full_ios_framework.py; and + find variables `filepath_without_entitlements` and + `filepath_with_entitlements` in sky/tools/create_macos_framework.py + + 2.1 if the binary should be code signed with entitlements, add [the name of + the binary] to the variable name with the `with_entitlements` suffix. + (`ios_file_with_entitlements` or `filepath_with_entitlements` depending on + which script) + + 2.2 if the binary shouldn't be code signed with entitlements, add [the name + of the binary] to the variable name with the `without_entitlements` suffix. + +#### Example + +Suppose `Flutter.xcframework/ios-arm64/Flutter.framework/Flutter` is a binary +that exist in a zip bundle called `ios/artifacts.zip`. + +1. Following step 1, in + [mac_ios_engine.json](https://github.com/flutter/engine/blob/main/ci/builders/mac_ios_engine.json), + it builds the artifact with the + `flutter/sky/tools/create_full_ios_framework.py` script. + +2. Following step 2, since + `Flutter.xcframework/ios-arm64/Flutter.framework/Flutter` shouldn't be code + signed with entitlements, we add the binary name + `Flutter.xcframework/ios-arm64/Flutter.framework/Flutter` to the + `ios_file_without_entitlements` variable. + +You can reference the generator script +[create_full_ios_framework.py](https://github.com/flutter/engine/blob/main/sky/tools/create_full_ios_framework.py). + +## Code signing artifacts other than flutter engine binaries + +The code signing functionality is implemented as [a recipe module in flutter recipes](https://cs.opensource.google/flutter/recipes/+/master:recipe_modules/signing/api.py). Therefore it can also be used to +code sign arbitrary flutter artifacts built through recipe, for example, flutter iOS usb dependencies. + +To code sign, after the artifacts are built, pass the file paths into +the code signing recipe module and invoke the function. An example is [how engine V2 invokes the code signing recipe module](https://cs.opensource.google/flutter/recipes/+/master:recipes/engine_v2/engine_v2.py;l=197-212). \ No newline at end of file diff --git a/docs/testing/Testing-the-engine.md b/docs/testing/Testing-the-engine.md new file mode 100644 index 0000000000000..4426f805736d5 --- /dev/null +++ b/docs/testing/Testing-the-engine.md @@ -0,0 +1,350 @@ +## Testing the engine + +Pull requests submitted to the [engine repository](https://github.com/flutter/engine) +should be tested to prevent functional regressions. + +This guide describes how to write and run various types of tests in the engine. + +## C++ - core engine + +If you edit `.cc` files in https://github.com/flutter/engine/tree/main, +you're working on the core, portable Flutter engine. + +### Unit tests + +C++ unit tests are co-located with their header and source files. For instance, +`fml/file.h` and `fml/file.cc` have a `fml/file_unittest.cc` in the same +directory. When editing C++ files, look for its `_unittest.cc` sibling or create +one if there isn't one present. + +The engine repo has a unified build system to build C, C++, Objective-C, +Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and +[Ninja](https://ninja-build.org/). Individual `_unittest.cc` files are +referenced by the `BUILD.gn` build rule located in the folder or in an ancestor +folder. + +You can run the C++ unit tests with: + +``` +testing/run_tests.py --type=engine +``` + +from the `flutter` directory, after building the engine variant to test +(by default `host_debug_unopt`). To use a different variant (e.g. if you use +an Apple Silicon Mac), run: + +``` +testing/run_tests.py --type=engine --variant=host_debug_unopt_arm64 +``` + +Behind the scenes, those tests in the same directory are built together as a +testonly executable when you build the engine variant. The `run_tests.py` script +executes them one by one. + +C++ unit tests are executed during pre-submit on our CI system when submitting +PRs to the `flutter/engine` repository. + +#### Google Tests + +C++ unit tests in the core engine uses the [Google Test](https://google.github.io/googletest/primer.html) +C++ testing framework to facilitate C++ test discovery, assertions, etc. + +Since the engine is portable, these unit tests are compiled and run directly +on and for your host machine architecture. + +It's best practice to test only one real production class per test and create +mocks for all other dependencies. + +## Java - Android embedding + +If you edit `.java` files in the https://github.com/flutter/engine/tree/main/shell/platform/android +directory, you're working on the Android embedding which connects the core C++ +engine to the Android SDK APIs and runtime. + +### Robolectric JUnit tests + +For testing logic within a class at a unit level, create or add to a JUnit test. + +Existing Java unit tests are located at https://github.com/flutter/engine/tree/main/shell/platform/android/test +and follow the Java package directory structure. Files in the `shell/platform/android/io/flutter/` +package tree can have a parallel file in the `shell/platform/android/test/io/flutter/` +package tree. Files in matching directories are considered [package visible](https://docs.oracle.com/javase/tutorial/java/javaOO/accesscontrol.html) +as is the case in standard Java. + +When editing production files in `shell/platform/android/io/flutter/`, +the easiest step to add tests is to look for a matching `...Test.java` file in +`shell/platform/android/test/io/flutter/`. + +See the [Java unit test README](https://github.com/flutter/engine/blob/main/shell/platform/android/test/README.md) +for details. + +The engine repo has a unified build system to build C, C++, Objective-C, +Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and +[Ninja](https://ninja-build.org/). Because it doesn't use the more common +Gradle build system (which can't build C++ for instance), the tests and its +dependencies can't be directly built and run inside Android Studio like a +standard Android project. + +Instead, the engine provides the script: + +``` +testing/run_tests.py --type=java +``` + +to easily build and run the JUnit tests. + +This script only has a limited amount of smartness. If you've never built the engine before, it'll build the test and classes under test with a reasonable default configuration. If you've built the engine before, it'll re-build the engine with the same GN flags. You may want to double check your [GN flags](../contributing/Compiling-the-engine.md#compiling-for-android-from-macos-or-linux) if you haven't built the engine for a while. + +Behind the scenes, it invokes GN and Ninja to build a single .jar file +containing the test runner and dependencies. Then it uses the system `java` +runtime to execute the .jar. JDK v8 must be set as your `$JAVA_HOME` to run +the Robolectric tests. + +See [Setting-up-the-Engine-development-environment#using-vscode-as-an-ide-for-the-android-embedding-java](../dev/Setting-up-the-Engine-development-environment.md) +for tips on setting up Java code completion and syntax highlighting in Visual +Studio when working on the engine and tests. + +JUnit tests are executed during pre-submit on our CI system when submitting +PRs to the `flutter/engine` repository. + +#### Robolectric + +[Robolectric](http://robolectric.org/) is a standard Android testing library to +mock the Android runtime. It allows tests to be executed on a lightweight Java +JVM without booting a heavy Android runtime in an emulator. This allows for +rapid test iterations and allows our tests to run better on CI systems. + +All engine JUnit tests are Robolectric tests. This means all `android.*` imports +are mocked by Robolectric. If you need to modify how Android components (such as +an [android.view.View](https://developer.android.com/reference/android/view/View.html) +or an [android.app.Activity](https://developer.android.com/reference/android/app/Activity.html)) +behave in the test, see other tests for examples or see docs at +http://robolectric.org/ on how to interact with shadows. + +#### Mockito + +[Mockito](https://site.mockito.org/) is also a standard Android testing library +used to mock non-Android dependencies needed to construct and test interactions +with your your under-test production class. + +It's best practice to test only one real production class per test and +mock all other dependencies with mockito. + +The Mockito library is an available test dependency when writing Robolectric +tests. + +### Component integration tests + +Component tests test the interaction of multiple embedding Java classes together +but they don't test all production classes end-to-end. In the Android embedding +case, we test groups of Java classes by their function in `...ComponentTest.java` +files that are also in the `shell/platform/android/test/io/flutter/` +directory. C++ engine parts via JNI are not tested here. + +Component tests are also Robolectric JUnit tests and are invoked together with +unit tests when running: + +``` +testing/run_tests.py --type=java +``` + +JUnit component tests are executed during pre-submit on our CI system when +submitting PRs to the `flutter/engine` repository. + +### End-to-end tests + +End-to-end tests exercise the entire Android embedding with the C++ engine on +a real Android runtime in an emulator. It's an integration test ensuring that +the engine as a whole on Android is functioning correctly. + +The project containing the Android end-to-end engine test is at +https://github.com/flutter/engine/tree/main/testing/scenario_app/android. + +This test project is build similarly to a normal Flutter app. The Dart code is +compiled into AOT and the Android part is compiled via Gradle with a dependency +on the prebuilt local engine. The built app then installed and executed on an +emulator. + +Unlike a normal Flutter app, the Flutter framework on the Dart side is a +lightweight fake at https://github.com/flutter/engine/tree/main/testing/scenario_app/lib +that implements some of the basic functionalities of `dart:ui` Window rather +than using the real Flutter framework at `flutter/flutter`. + +The end-to-end test can be executed by running: + +``` +testing/scenario_app/run_android_tests.sh +``` + +Additional end-to-end instrumented tests can be added to https://github.com/flutter/engine/tree/main/testing/scenario_app/android/app/src/androidTest/java/dev/flutter/scenarios. + +If supporting logic is needed for the test case, it can be added to the +Android app under-test in https://github.com/flutter/engine/tree/main/testing/scenario_app/android/app/src/main/java/dev/flutter/scenarios +or to the fake Flutter framework under-test in https://github.com/flutter/engine/tree/main/testing/scenario_app/lib. + +As best practice, favor adding unit tests if possible since instrumented tests +are, by nature, non-hermetic, slow and flaky. + +End-to-end tests on Android are run on presubmit for flutter/engine PRs. + +## Objective-C - iOS embedding + +If you edit `.h` or `.mm` files in the https://github.com/flutter/engine/tree/main/shell/platform/darwin/ios +directory, you're working on the iOS embedding which connects the core C++ +engine to the iOS SDK APIs and runtime. + +### XCTest unit tests + +For testing logic within a class in isolation, create or add to a XCTestCase. + +The iOS unit testing infrastructure is split in 2 different locations. The +`...Test.mm` files in https://github.com/flutter/engine/tree/main/shell/platform/darwin/ios +contain the unit tests themselves. The +https://github.com/flutter/engine/tree/main/testing/ios/IosUnitTests directory +contains an Xcode container project to execute the test. + +See the [iOS unit test README](https://github.com/flutter/engine/blob/main/testing/ios/IosUnitTests/README.md) +for details on adding new test files. + +The engine repo has a unified build system to build C, C++, Objective-C, +Objective-C++, and Java files using [GN](https://gn.googlesource.com/gn/) and +[Ninja](https://ninja-build.org/). Since GN and Ninja has to build the C++ +dependencies that the Objective-C classes reference, the tests aren't built by +the Xcode project in https://github.com/flutter/engine/tree/main/testing/ios/IosUnitTests. + +Instead, the engine provides the script: + +``` +testing/run_tests.py --type=objc +``` + +to easily build and run the XCTests. + +- Add the `--ios-variant ios_debug_sim_unopt_arm64` argument when using an arm64 Mac simulator (built with `--simulator-cpu=arm64`). + +This script only has a limited amount of smartness. If you've never built the engine before, it'll build the test and classes under test with a reasonable default configuration. If you've built the engine before, it'll re-build the engine with the same GN flags. You may want to double check your GN flags ([See compiling for ios from macos](../contributing/Compiling-the-engine.md#compiling-for-ios-from-macos)) if you haven't built the engine for a while. + +Behind the scenes, it invokes GN and Ninja to build the tests and dependencies +into a single `.dylib`. Then it uses Xcode and the Xcode project at +`testing/ios/IosUnitTests` to import and execute the XCTests in the `.dylib`. + +If you get an `AssertionError: libios_test_flutter.dylib doesn't exist` error, you may need to manually run the ninja command that is printed to the terminal. e.g. `ninja -C $FLUTTER_ENGINE/out/ios_debug_sim_unopt_arm64 ios_test_flutter` + +See [Setting-up-the-Engine-development-environment#editor-autocomplete-support](../dev/Setting-up-the-Engine-development-environment.md) +for tips on setting up C/C++/Objective-C code completion and syntax highlighting +when working on the engine and tests. + +To debug the XCTests, you can open the Xcode project at `testing/ios/IosUnitTests/IosUnitTests.xcodeproj` +and run the tests (such as via ⌘U). Note you cannot modify the test source and +build the tests in Xcode for reasons mentioned above. If you modify the test, +you need to run `testing/run_tests.py` again. + +XCTests are executed during pre-submit on our CI system when submitting PRs to +the `flutter/engine` repository. + +#### XCTest + +[XCTest](https://developer.apple.com/documentation/xctest) is the standard way +of creating unit tests in Xcode projects. Since iOS has x86 simulators and +since we can build x86 engines, we can execute the XCTests directly on macOS +in a headless simulator using the real iOS SDK. + +#### OCMock + +[OCMock](https://ocmock.org) is a standard iOS testing library used to mock +dependencies needed to construct and test interactions with your under-test +production class. + +It's best practice to test only one real production class per test and +mock all other dependencies with OCMock. + +The OCMock library is available as a test dependency when writing XCTests for +the engine. + +### End-to-end tests + +End-to-end tests exercise the entire iOS embedding with the C++ engine on +a headless iOS simulator. It's an integration test ensuring that +the engine as a whole on iOS is functioning correctly. + +The project containing the iOS end-to-end engine test is at +https://github.com/flutter/engine/tree/main/testing/scenario_app/ios. + +This test project is build similarly to a normal debug Flutter app. The Dart +code is bundled in JIT mode and is brought into Xcode with a `.framework` +dependency on the prebuilt local engine. It's then installed and executed on a +simulator via Xcode. + +Unlike a normal Flutter app, the Flutter framework on the Dart side is a +lightweight fake at https://github.com/flutter/engine/tree/main/testing/scenario_app/lib +that implements some of the basic functionalities of `dart:ui` Window rather +than using the real Flutter framework at `flutter/flutter`. + +The end-to-end test can be executed by running: + +``` +testing/scenario_app/run_ios_tests.sh +``` + +Additional end-to-end instrumented tests can be added to https://github.com/flutter/engine/tree/main/testing/scenario_app/ios/Scenarios/ScenariosTests. + +If supporting logic is needed for the test case, it can be added to the +Android app under-test in https://github.com/flutter/engine/tree/main/testing/scenario_app/ios/Scenarios/Scenarios +or to the fake Flutter framework under-test in https://github.com/flutter/engine/tree/main/testing/scenario_app/lib. + +As best practice, favor adding unit tests if possible since end-to-end tests +are, by nature, non-hermetic, slow and flaky. + +End-to-end tests on iOS are executed during pre-submit on our CI system when +submitting PRs to the `flutter/engine` repository. + +## Dart - dart:ui + +If you edit `.dart` files in https://github.com/flutter/engine/tree/main/lib/ui, +you're working on the 'dart:ui' package which is the interface between +the C++ engine and the Dart Flutter framework. + +### Unit tests + +Dart classes in https://github.com/flutter/engine/tree/main/lib/ui have matching +unit tests at https://github.com/flutter/engine/tree/main/testing/dart. + +When editing production files in the 'dart:ui' package, add to or create a +test file in `testing/dart`. + +To run the Dart unit tests, use the script: + +``` +testing/run_tests.py --type=dart +``` + +Behind the scenes, it invokes the engine repo's unified [GN](https://gn.googlesource.com/gn/) +and [Ninja](https://ninja-build.org/) build systems to use a version of the Dart +SDK specified in the `DEPS` file to create a `sky_engine` Dart package. Then it +compiles and runs each `_test.dart` file under `testing/dart`. + +To debug the test, open `src/out/ios_debug_sim_unopt/scenario_app/Scenarios.xcodeproj` in +Xcode and hit CMD+U. + +Dart unit tests are executed during pre-submit on our CI system when submitting +PRs to the `flutter/engine` repository. + +_See also: [Flutter Test Fonts](../../contributing/testing/Flutter-Test-Fonts.md)_ + +### Framework tests + +Dart tests in the `flutter/flutter` framework repo are also executed on top of +the `dart:ui` package and underlying engine. + +These tests are executed during pre-submit on our CI system when +submitting PRs to the `flutter/engine` repository. + +Assuming your `flutter` and `engine` working directories are siblings, you can run the framework tests locally using the following command from the root of your `flutter` repository: + +```bash +(cd packages/flutter; ../../bin/flutter test --local-engine=host_debug_unopt --local-engine-host=host_debug_unopt) +``` + +## Web engine + +Web tests are run via the `felt` command. More details can be found in [lib/web_ui/README.md](https://github.com/flutter/engine/blob/main/lib/web_ui/README.md#hacking-on-the-web-engine).