Add headless renderer

[Shatur]

What problem does this solve or what need does it fill?

When creating multiplayer, often the logic of the game is executed on the headless server. But registration of renderer types such as Asset<Meshes> is now tied to the renderer. Therefore, to use exactly the same logic on the server, I have to add compile-time checks:

    pub fn spawn(
        commands: &mut Commands,
        transform: Transform,
        #[cfg(not(feature = "headless"))] meshes: &mut Assets<Mesh>,
        #[cfg(not(feature = "headless"))] materials: &mut Assets<StandardMaterial>,
    ) -> Self {
    // ...
    }

This looks ugly and very inconvenient to use.

What solution would you like?

It would be great to have a headless renderer.

[cart]

The new renderer does have a clear separation between "render code" and "app code", but the types are still exported from the new bevy_render crate (Mesh) and the new bevy_pbr crate (StandardMaterial). I think it makes sense logically to keep it that way.

I do agree that we should support headless mode, but I think it would probably look more like this:

• Add a "null / headless" render backend to wgpu (to ensure that people writing render code don't need to think too hard about adding "headless" support to their apps)
• Add the ability to disable render logic entirely (this is straightforward as this is already fully decoupled)

[Shatur]

Sounds like a great idea! The same approach is used in Godot.

I understand that the 0.6 release is already huge, but it there any chance to see headless render mode in 0.6?

[cart]

I think wgpu already supports "null" mode in that it still compiles / runs if you dont have any supported devices + backends available. So the biggest missing piece is just ensuring the right things are disabled / nothing panics in headless mode. This seems like a pretty small chunk of work. I can't promise that it will happen in time, but we might be able to make it work!

[alice-i-cecile]

Related to #2896 and #1057.

[cart]

A relevant message I just left on discord in response to @Shatur:

I would like to start working on #3155 Is it a good time to do it?

Yeah now seems like as good a time as any! I think we should start by using "runtime configuration", such as alternative plugins (or maybe configuration of RenderPlugin). I don't have a strong opinion on the UX yet, so feel free to experiment and present your thoughts.

How can I enable null mode in wgpu?

I know that wgpu will fail to initialize a device when there are no backends available:

bevy/pipelined/bevy_render2/src/renderer/mod.rs

Line 70 in bab4ee9

let adapter = instance

So step one will either be:

A) ensuring we don't panic when there is no device available (and headless mode is enabled)
B) just not trying to create the device at all when headless mode is enabled
C) add a "null backend" to wgpu

Using (A) would probably require compiling wgpu with backends disabled (which may or may not be possible right now). (B) would avoid the failure in the first place and feels cleaner to me than (A). Both (A) and (B) would require gracefully handling the fact that the RenderDevice and RenderQueue resources dont exist at all. This would mainly mean that we don't run the render app schedule when headless mode is enabled.

However we do currently support the ability for normal app code to access the RenderDevice as a resource (although we don't currently use it for anything). If we go with (A) or (B), that type of code would panic when we don't insert the RenderDevice. Theres also a future where asset loaders directly produce gpu assets in a separate thread instead of going through the Render App schedule. The only way to support these scenarios properly is with a "null wgpu backend" that does nothing, but still results in successful creation of a RenderDevice. I don't think wgpu has an actual "null backend" yet, but I'm hoping they're open to it.

Imo we should start with (B) as it is easy and should generally work.
I think its also important to distinguish between "headless without any renderering or windows at all" and "headless without windows but with rendering"
as people have expressed interest in using bevy to render to textures and write them to files (or send them somewhere else via RPC)
I know you're mainly interested in the former scenario, but we should figure out the best way to present those two modes

[Shatur]

Here is the summary of messages that I wrote in the discord.

I decided to try researching the suggested step B. I managed to get the 3d_scene_pipelined example to work without the device by commenting out just a couple of lines:

• RenderApp creation with wgpu device
• All accesses to RenderApp (13 references).

After that I had a window with no render. Then I commented out window creation - it worked.
Now we need to decide how it will look.

Renderer

Maybe make it a compile option because wgpu is not needed on servers? E.g. just allow bevy_render2 compilation without wgpu. Here is some options on how this can be implemented:

1. Fake RenderApp. In this case, the other plugin does not have to worry about the renderer not existing and it's great. But is it possible to implement it? What if other plugins add systems to the RenderApp that require a RenderDevice?
2. Return Option from sub_app and force plugin authors to handle RenderApp absence:
   

   bevy/crates/bevy_app/src/app.rs

   Lines 857 to 862 in 2def902

   	pub fn sub_app(&mut self, label: impl AppLabel) -> &mut App {
   	match self.get_sub_app(label) {
   	Ok(app) => app,
   	Err(label) => panic!("Sub-App with label '{:?}' does not exist", label),
   	}
   	}

3. Force plugin users to support headless mode explicitly (via compile option). This is the least obvious way, but it may allow plugin authors to exclude some dependencies for headless mode.

Window

With a window, things are more complicated. If we just disable the bevy_winit, the game loses its game loop and exists immediately. So, here is some options on how this can be implemented:

1. Decouple game loop from bevy_winit. Is it possible? If yes, I think it would be the best solution as it allows to simply remove bevy_winit from the server.
2. Compile time option that disables window creation and winit dependency for bevy_winit.
3. We can add a runtime option not to create a window.

@cart Which of the following options would you like more? Do you have any alternative implementation suggestions?

[cart]

Glad you got the initial "headless mode" implemented!

Renderer

I do think that we need to separate the tasks of "logical headless mode that doesn't set up the renderer when enabled" and "completely removing wgpu from the tree". The first two solutions you suggest for the renderer don't solve the "remove wgpu from the tree" problem because users still code to wgpu symbols in those scenarios. The problem here is that wgpu is a part of our public API. If we want to disable the dependency entirely at compile time, we need user code to either:

1. Never touch wgpu directly (and we would instead build an abstraction layer on top of wgpu that users interact with)
2. Accommodate the wgpu feature at compile time. I can think of two options here. Either by convention wgpu render logic and other logic are completely separate crates, with wgpu crates being optional, or engine developers and users would agree to only use wgpu logic behind a "wgpu" cargo feature.

Neither (1) or (2) are actually options in my opinion. In (1) we'd just be re-inventing the wgpu api. We've already agreed that wgpu is our user-facing gpu api and I'm not reverting that decision. (2) involves foisting too much responsibility on Bevy users to manually draw hard compile-time lines between render logic and app logic. Unity / Godot / Unreal / etc don't force users to do this. We can't either. So what can we do? Did we design ourselves into a hole?

Nope! We can actually accomplish (1), we just need to shift our perspective a bit. The goal of removing wgpu from the tree is to remove cruft that isn't needed by servers. With the constraints I have established above, I am basically asserting that headless apps need to be compiled with a gpu api if they are coding to something that uses bevy_render (things like Meshes, Images, Sprites, etc). That is not cruft, its a part of the public Bevy api. Note that bevy_render is already an optional dependency. If you aren't writing code that spawns "rendered things", you can disable it. If you are spawning "rendered things" wgpu is our public gpu api and that needs to be usable whenever someone pulls in bevy_render as a dependency. What we don't need are the gpu backends like Vulkan, Metal, OpenGL, etc. We only need the actual users facing api. Internally wgpu can either fail when called or use a "null" backend (we might want to support both). We just need a way to tell wgpu to compile without these backends. Fortunately wgpu-hal backends are already expressed as cargo features, so making this configurable shouldn't be too hard.

To solve the other "logical headless mode that doesn't set up the renderer when enabled" problem, we can either:

A. Keep user / engine code as it is and implement a "null backend" in wgpu that doesn't fail when used like a gpu api. This would likely still incur some overhead as we'd still be running the renderer, it just wouldn't do anything. We probably don't want that in most headless scenarios.
B. Check a run-time "headless mode" flag that ensures we dont run the renderer or try to init the render app.

I think we should scope this issue to solving (B). Your idea to return Option when users call sub_app is reasonable (we actually already have a get_sub_app api that does this, so the solution would be to remove the panic-ing api to force users to contend with the Option). But it will still fail in headless scenarios the second users try to unwrap that Option. Maybe thats acceptable (and it does seem like a reasonable short term solution), but we should also discuss ways to make this line stricter / harder for users to mess up. Maybe we should have separate "RenderPlugins" that we can just avoid initializing at all. Combine that with deferred plugin init (ex: bevyengine/rfcs#33) and we could choose to just not initialize render app plugins when headless mode is enabled. This will require design work and I can't just give you an easy answer about what that should look like.

Window

We can't really decouple the game loop from bevy_winit because winit needs to own the loop. We could shim that out, but we'd effectively be re-inventing the winit api. I'm heavily considering ripping out the existing "bevy window layer" in favor of treating winit as our public window api (much like the bevy_render2 + wgpu changes). In that case we would want something like a "null window backend" if the goal is to remove "backend cruft".

Fortunately we've already abstracted out "app runners". You can already just disable the optional bevy_winit dependency and define a custom app runner: https://github.com/bevyengine/bevy/blob/main/examples/app/custom_loop.rs. Custom runners can just choose to ignore window creation events, so this should satisfy "headless" needs already.

[Shatur]

Thank you a lot for your input! The suggested design is so elegant, I like it. Will start working on it.

[Shatur]

What we don't need are the gpu backends like Vulkan, Metal, OpenGL, etc. We only need the actual users facing api. Internally wgpu can either fail when called or use a "null" backend (we might want to support both). We just need a way to tell wgpu to compile without these backends. Fortunately wgpu-hal backends are already expressed as cargo features, so making this configurable shouldn't be too hard.

@cart There is an issue with removing dependencies. Bevy depends on wgpu-hal indirectly. It uses wgpu and all features of wgpu-hal is controlled by wgpu-core. Here is an issue about it: gfx-rs/wgpu#1221. So for now we can't remove unnecessary cruft from server builds. Should we ask the developers about this feature?
But "logical headless mode" is doable right now. Since it's an unrelated change, can I try to implement it and send a PR?

[Shatur]

Fortunately we've already abstracted out "app runners". You can already just disable the optional bevy_winit dependency and define a custom app runner: https://github.com/bevyengine/bevy/blob/main/examples/app/custom_loop.rs. Custom runners can just choose to ignore window creation events, so this should satisfy "headless" needs already.

@cart It looks like here we have the same issue as with renderer.
If I disable bevy_winit then the application crashes because the window is expected on some systems. Examples (e.g. it crashes in these places):

bevy/pipelined/bevy_pbr2/src/light.rs

Line 314 in 172f4d6

let window = windows.get(camera.window).unwrap();

bevy/pipelined/bevy_core_pipeline/src/main_pass_3d.rs

Lines 50 to 53 in 172f4d6

	let (opaque_phase, alpha_mask_phase, transparent_phase, target, depth) = self
	.query
	.get_manual(world, view_entity)
	.expect("view entity should exist");

Should we apply the same approach as for wgpu? E.g. force plugins to handle unwraps?

[cart]

So for now we can't remove unnecessary cruft from server builds. Should we ask the developers about this feature?

Yeah I think its worth asking about. But for now I wouldn't consider it a priority, so I wouldn't push too hard 😄

But "logical headless mode" is doable right now. Since it's an unrelated change, can I try to implement it and send a PR?

Yup sounds good to me!

Should we apply the same approach as for wgpu? E.g. force plugins to handle unwraps?

Hmm yeah this is interesting. I think it probably makes sense to handle this "inline" for now by not assuming cameras correlate to windows. Ultimately this feels like it will intersect with the "RenderTargets" work I have planned, which will treat windows as "just another render target type" and further decouple cameras from windows. Theres a lot to cover there that I don't have the time to unpack at the moment, so for now I think getting rid of the unwraps makes the most sense.

[Shatur]

@cart thanks for the input!
But I have one more question about design 😄 How users could activate "headless mode"? I see two approaches:

1. Resource. The creation of a renderer device is decided depending on its existence (or absence?).
2. Compile-time feature in cargo.

Which one would you think is better? 1 is a common pattern in Bevy. But is there a use case to disable the renderer at run-time? We discussed this in chat with @alice-i-cecile and we do not have a clear opinion on this...
If we pick 1, should we create a separate HeadlessPlugins by the analogy with MinimalPlugins?

[Shatur]

so for now I think getting rid of the unwraps makes the most sense.

Good news, it was just two unwraps :) I'll send a PR once you merge #3312.

[Shatur]

Good news, it was just two unwraps :) I'll send a PR once you merge #3312.

Opened #3321.

Will try to implement dummy backend for wgpu: gfx-rs/wgpu#2291 and then use this backend in Bevy if headless feature is enabled. Will take a look at logical device creation after it.

[Shatur]

Will try to implement dummy backend for wgpu: gfx-rs/wgpu#2291 and then use this backend in Bevy if headless feature is enabled. Will take a look at logical device creation after it.

I've discussed this in more detail with the wgpu developers in Matrix and there doesn't seem to be a nice solution for it.
This comment describes the issue pretty well: gfx-rs/wgpu#2291 (comment)

So maybe we should implement only "logical headless" mode to run Bevy on servers without GPU and close the issue.
We could add a resource that disables device creation and remove sub_app and sub_app_mut (panicing API) to force users to check (as cart suggested). Should be a very small change. I will send a PR if you agree.

@cart so what do you think? Maybe you have a better suggestion?
Sorry if I bothering you with questions too much :)

[alice-i-cecile]

So maybe we should implement only "logical headless" mode to run Bevy on servers
We could add a resource that disables device creation and return Option from sub_app and sub_app_mut (currently we just panics).

This sounds like a nice path forward. We can close this issue once that PR is merged, and reopen this issue if the solution isn't enough or if things get easier on wgpu's side IMO.

[Shatur]

@alice-i-cecile, @cart just got another idea about the design. Instead of enable "headless mode" via resource we could split RenderPlugin:

bevy/crates/bevy_render/src/lib.rs

Line 107 in 22c665f

impl Plugin for RenderPlugin {

Into RenderPrimitivesPlugin (that creates all rendering primitives) and RenderDevicePlugin (that creates a render device and sub app). Then add RenderDevicePlugin to DefaultPlugins group and user will be able to opt-out it via group.disable. What do you think?

[alice-i-cecile]

Hmm. The plugin splitting is more elegant in some ways. However, it may be a bit harder to discover, and I don't believe we can (currently) enable / disable plugins at runtime 🤔

That said, I think this is a natural split, and I suspect that this will effectively always be a choice made at app start.

The ability to add/remove systems at runtime is a feature we eventually need, and with a properly structured approach to plugin management I think we should be able to get the same effect for plugins.

Overall, mild preference towards the plugin splitting approach.

[cart]

Into RenderPrimitivesPlugin (that creates all rendering primitives) and RenderDevicePlugin (that creates a render device and sub app). Then add RenderDevicePlugin to DefaultPlugins group and user will be able to opt-out it via group.disable. What do you think?

This is a reasonable design, but I think I prefer just adding a configuration field to the existing WgpuOptions resource. That way all wgpu configuration is in one place / easily discoverable.

[LucCADORET]

Was this addressed ? I'm trying to run the headless_defaults on some ubuntu server, and I end up with thread 'main' panicked at 'Failed to initialize any backend! Wayland status: "backend disabled" X11 status: XOpenDisplayFailed'
Isn't the headless_defaults supposed to work on a machine without display ?

[sQu1rr]

It does not seem to be, as a window appears to be required for the "headless" server, that uses RenderPlugin. And RenderPlugin is required for Mesh, Shader, etc. So the original issue is not addressed in my opinion. Currently, I am working around it by manually injecting the MeshPlugin/etc (and getting some missing labels warnings), but it would be nice to have a genuinely headless server that supports some of the functionality provided by the RenderPlugin

[Shatur]

You can run without a window, there is an example in this repo.

[sQu1rr]

I must be completely missing the correct way of doing it. There are 3 examples that I found:

• headless : kind of what I need, but I also need RenderPlugin (or parts of it) to be included.
• no_renderer : this allows me to include the renderplugin into headless and disable the rendering, but the window stays
• no_winit : this shows how to disable the window, but it only runs once. i tried setting SchduleRunnerSettings : run_loop with no success.

What I need is basically headless server with some of the default plugins (or rather minimal plugins + manually added plugins) required for physics (e.g. mesh, gltf) and no window. Could you point me in the right direction please?

[Shatur]

What I need is basically headless server with some of the default plugins (or rather minimal plugins + manually added plugins) required for physics (e.g. mesh, gltf) and no window. Could you point me in the right direction please?

It looks like no_winit is what you need. But runner is tied to winit, this is why it runs only once :(
You just need to write your own. It shouldn't be hard.

[sQu1rr]

Would be nice to have it working out of the box though. I guess would also be useful to not require WindowPlugin in RenderPlugin, especially when there are no backends to render to. This way it should be fine to use MinimalPlugins + RenderPlugin instead of working around it or writing a custom runner.

[Shatur]

Sure, there was a talk about this decoupling. I believe it will happen in the future. Maybe after #5589

[richardanaya]

Is this now possible?

[2-3-5-41]

To those that may come across this issue:

You don't need to make your own schedule runner, bevy already has one build in that is used in the MinimalPlugins. However, if you are making a bevy server that need physics; you'll first want to disable winit as done so in the without_winit example, then you can insert the ScheduleRunnerSettings resource from the headless example, and lastly, you can add the ScheduleRunnerPlugin with .add_plugin(ScheduleRunnerPlugin) to create a better headless bevy server for your game servers, or physics simulation servers.

---

TL;DR

Just add this to your app builder to get a true headless bevy app that supports physics:

...

App:new()
    .insert_resource(ScheduleRunnerSettings::run_loop(Duration::from_secs_f64(1.0/60.0)))
    .add_plugins(DefaultPlugins.build().disable::<WinitPlugin>())
    .add_plugin(ScheduleRunnerPlugin)

...

I'm currently using this in Bevy version 0.10.0

[cs50victor]

any update on this? @2-3-5-41 have a repo / a more detailed example?

[cs50victor]

Here's a working example ( bevy v0.12.1 ) for anyone else who needs a headless renderer - https://github.com/mosure/bevy_gaussian_splatting/blob/main/examples/headless.rs

[richardanaya]

@cs50victor

Thanks for your awesome example, I tried to create a really minimalistic version of it, unfortunately I had some issues on OSX.

use bevy::{
    app::ScheduleRunnerPlugin, core_pipeline::tonemapping::Tonemapping, prelude::*,
    render::renderer::RenderDevice,
};

/// Derived from: https://github.com/bevyengine/bevy/pull/5550
mod frame_capture {
    pub mod image_copy {
        use std::sync::Arc;

        use bevy::prelude::*;
        use bevy::render::render_asset::RenderAssets;
        use bevy::render::render_graph::{self, NodeRunError, RenderGraph, RenderGraphContext};
        use bevy::render::renderer::{RenderContext, RenderDevice, RenderQueue};
        use bevy::render::{Extract, RenderApp};

        use bevy::render::render_resource::{
            Buffer, BufferDescriptor, BufferUsages, CommandEncoderDescriptor, Extent3d,
            ImageCopyBuffer, ImageDataLayout, MapMode,
        };
        use pollster::FutureExt;
        use wgpu::Maintain;

        use std::sync::atomic::{AtomicBool, Ordering};

        pub fn receive_images(
            image_copiers: Query<&ImageCopier>,
            mut images: ResMut<Assets<Image>>,
            render_device: Res<RenderDevice>,
        ) {
            for image_copier in image_copiers.iter() {
                if !image_copier.enabled() {
                    continue;
                }
                // Derived from: https://sotrh.github.io/learn-wgpu/showcase/windowless/#a-triangle-without-a-window
                // We need to scope the mapping variables so that we can
                // unmap the buffer
                async {
                    let buffer_slice = image_copier.buffer.slice(..);

                    // NOTE: We have to create the mapping THEN device.poll() before await
                    // the future. Otherwise the application will freeze.
                    let (tx, rx) = futures_intrusive::channel::shared::oneshot_channel();
                    buffer_slice.map_async(MapMode::Read, move |result| {
                        tx.send(result).unwrap();
                    });
                    render_device.poll(Maintain::Wait);
                    rx.receive().await.unwrap().unwrap();
                    if let Some(image) = images.get_mut(&image_copier.dst_image) {
                        image.data = buffer_slice.get_mapped_range().to_vec();
                    }

                    image_copier.buffer.unmap();
                }
                .block_on();
            }
        }

        pub const IMAGE_COPY: &str = "image_copy";

        pub struct ImageCopyPlugin;
        impl Plugin for ImageCopyPlugin {
            fn build(&self, app: &mut App) {
                let render_app = app
                    .add_systems(Update, receive_images)
                    .sub_app_mut(RenderApp);

                render_app.add_systems(ExtractSchedule, image_copy_extract);

                let mut graph = render_app.world.get_resource_mut::<RenderGraph>().unwrap();

                graph.add_node(IMAGE_COPY, ImageCopyDriver);

                graph.add_node_edge(IMAGE_COPY, bevy::render::main_graph::node::CAMERA_DRIVER);
            }
        }

        #[derive(Clone, Default, Resource, Deref, DerefMut)]
        pub struct ImageCopiers(pub Vec<ImageCopier>);

        #[derive(Clone, Component)]
        pub struct ImageCopier {
            buffer: Buffer,
            enabled: Arc<AtomicBool>,
            src_image: Handle<Image>,
            dst_image: Handle<Image>,
        }

        impl ImageCopier {
            pub fn new(
                src_image: Handle<Image>,
                dst_image: Handle<Image>,
                size: Extent3d,
                render_device: &RenderDevice,
            ) -> ImageCopier {
                let padded_bytes_per_row =
                    RenderDevice::align_copy_bytes_per_row((size.width) as usize) * 4;

                let cpu_buffer = render_device.create_buffer(&BufferDescriptor {
                    label: None,
                    size: padded_bytes_per_row as u64 * size.height as u64,
                    usage: BufferUsages::MAP_READ | BufferUsages::COPY_DST,
                    mapped_at_creation: false,
                });

                ImageCopier {
                    buffer: cpu_buffer,
                    src_image,
                    dst_image,
                    enabled: Arc::new(AtomicBool::new(true)),
                }
            }

            pub fn enabled(&self) -> bool {
                self.enabled.load(Ordering::Relaxed)
            }
        }

        pub fn image_copy_extract(
            mut commands: Commands,
            image_copiers: Extract<Query<&ImageCopier>>,
        ) {
            commands.insert_resource(ImageCopiers(
                image_copiers.iter().cloned().collect::<Vec<ImageCopier>>(),
            ));
        }

        #[derive(Default)]
        pub struct ImageCopyDriver;

        impl render_graph::Node for ImageCopyDriver {
            fn run(
                &self,
                _graph: &mut RenderGraphContext,
                render_context: &mut RenderContext,
                world: &World,
            ) -> Result<(), NodeRunError> {
                let image_copiers = world.get_resource::<ImageCopiers>().unwrap();
                let gpu_images = world.get_resource::<RenderAssets<Image>>().unwrap();

                for image_copier in image_copiers.iter() {
                    if !image_copier.enabled() {
                        continue;
                    }

                    let src_image = gpu_images.get(&image_copier.src_image).unwrap();

                    let mut encoder = render_context
                        .render_device()
                        .create_command_encoder(&CommandEncoderDescriptor::default());

                    let block_dimensions = src_image.texture_format.block_dimensions();
                    let block_size = src_image.texture_format.block_size(None).unwrap();

                    let padded_bytes_per_row = RenderDevice::align_copy_bytes_per_row(
                        (src_image.size.x as usize / block_dimensions.0 as usize)
                            * block_size as usize,
                    );

                    let texture_extent = Extent3d {
                        width: src_image.size.x as u32,
                        height: src_image.size.y as u32,
                        depth_or_array_layers: 1,
                    };

                    encoder.copy_texture_to_buffer(
                        src_image.texture.as_image_copy(),
                        ImageCopyBuffer {
                            buffer: &image_copier.buffer,
                            layout: ImageDataLayout {
                                offset: 0,
                                bytes_per_row: Some(
                                    std::num::NonZeroU32::new(padded_bytes_per_row as u32)
                                        .unwrap()
                                        .into(),
                                ),
                                rows_per_image: None,
                            },
                        },
                        texture_extent,
                    );

                    let render_queue = world.get_resource::<RenderQueue>().unwrap();
                    render_queue.submit(std::iter::once(encoder.finish()));
                }

                Ok(())
            }
        }
    }
    pub mod scene {
        use std::path::PathBuf;

        use bevy::{
            app::AppExit,
            prelude::*,
            render::{camera::RenderTarget, renderer::RenderDevice},
        };
        use wgpu::{Extent3d, TextureDescriptor, TextureDimension, TextureFormat, TextureUsages};

        use super::image_copy::ImageCopier;

        #[derive(Component, Default)]
        pub struct CaptureCamera;

        #[derive(Component, Deref, DerefMut)]
        struct ImageToSave(Handle<Image>);

        pub struct CaptureFramePlugin;
        impl Plugin for CaptureFramePlugin {
            fn build(&self, app: &mut App) {
                println!("Adding CaptureFramePlugin");
                app.add_systems(PostUpdate, update);
            }
        }

        #[derive(Debug, Default, Resource, Event)]
        pub struct SceneController {
            state: SceneState,
            name: String,
            width: u32,
            height: u32,
            single_image: bool,
        }

        impl SceneController {
            pub fn new(width: u32, height: u32, single_image: bool) -> SceneController {
                SceneController {
                    state: SceneState::BuildScene,
                    name: String::from(""),
                    width,
                    height,
                    single_image,
                }
            }
        }

        #[derive(Debug, Default)]
        pub enum SceneState {
            #[default]
            BuildScene,
            Render(u32),
        }

        pub fn setup_render_target(
            commands: &mut Commands,
            images: &mut ResMut<Assets<Image>>,
            render_device: &Res<RenderDevice>,
            scene_controller: &mut ResMut<SceneController>,
            pre_roll_frames: u32,
            scene_name: String,
        ) -> RenderTarget {
            let size = Extent3d {
                width: scene_controller.width,
                height: scene_controller.height,
                ..Default::default()
            };

            // This is the texture that will be rendered to.
            let mut render_target_image = Image {
                texture_descriptor: TextureDescriptor {
                    label: None,
                    size,
                    dimension: TextureDimension::D2,
                    format: TextureFormat::Rgba8UnormSrgb,
                    mip_level_count: 1,
                    sample_count: 1,
                    usage: TextureUsages::COPY_SRC
                        | TextureUsages::COPY_DST
                        | TextureUsages::TEXTURE_BINDING
                        | TextureUsages::RENDER_ATTACHMENT,
                    view_formats: &[],
                },
                ..Default::default()
            };
            render_target_image.resize(size);
            let render_target_image_handle = images.add(render_target_image);

            // This is the texture that will be copied to.
            let mut cpu_image = Image {
                texture_descriptor: TextureDescriptor {
                    label: None,
                    size,
                    dimension: TextureDimension::D2,
                    format: TextureFormat::Rgba8UnormSrgb,
                    mip_level_count: 1,
                    sample_count: 1,
                    usage: TextureUsages::COPY_DST | TextureUsages::TEXTURE_BINDING,
                    view_formats: &[],
                },
                ..Default::default()
            };
            cpu_image.resize(size);
            let cpu_image_handle = images.add(cpu_image);

            commands.spawn(ImageCopier::new(
                render_target_image_handle.clone(),
                cpu_image_handle.clone(),
                size,
                render_device,
            ));

            commands.spawn(ImageToSave(cpu_image_handle));

            scene_controller.state = SceneState::Render(pre_roll_frames);
            scene_controller.name = scene_name;
            RenderTarget::Image(render_target_image_handle)
        }

        fn update(
            images_to_save: Query<&ImageToSave>,
            mut images: ResMut<Assets<Image>>,
            mut scene_controller: ResMut<SceneController>,
            mut app_exit_writer: EventWriter<AppExit>,
        ) {
            if let SceneState::Render(n) = scene_controller.state {
                if n < 1 {
                    for image in images_to_save.iter() {
                        let img_bytes = images.get_mut(image.id()).unwrap();

                        let img = match img_bytes.clone().try_into_dynamic() {
                            Ok(img) => img.to_rgba8(),
                            Err(e) => panic!("Failed to create image buffer {e:?}"),
                        };

                        let images_dir =
                            PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("test_images");
                        print!("Saving image to: {:?}\n", images_dir);
                        std::fs::create_dir_all(&images_dir).unwrap();

                        let uuid = bevy::utils::Uuid::new_v4();
                        let image_path = images_dir.join(format!("{uuid}.png"));
                        if let Err(e) = img.save(image_path) {
                            panic!("Failed to save image: {}", e);
                        };
                    }
                    if scene_controller.single_image {
                        app_exit_writer.send(AppExit);
                    }
                } else {
                    scene_controller.state = SceneState::Render(n - 1);
                }
            }
        }
    }
}

pub struct AppConfig {
    width: u32,
    height: u32,
    single_image: bool,
}

fn headless_app() {
    let mut app = App::new();

    let config = AppConfig {
        width: 1920,
        height: 1080,
        single_image: true,
    };

    // setup frame capture
    app.insert_resource(frame_capture::scene::SceneController::new(
        config.width,
        config.height,
        config.single_image,
    ));
    app.insert_resource(ClearColor(Color::rgb_u8(0, 0, 0)));

    app.add_plugins(
        DefaultPlugins
            .set(ImagePlugin::default_nearest())
            .set(WindowPlugin {
                primary_window: None,
                exit_condition: bevy::window::ExitCondition::DontExit,
                close_when_requested: false,
            }),
    );

    app.add_plugins(frame_capture::image_copy::ImageCopyPlugin);

    // headless frame capture
    app.add_plugins(frame_capture::scene::CaptureFramePlugin);

    app.add_plugins(ScheduleRunnerPlugin::run_loop(
        std::time::Duration::from_secs_f64(1.0 / 60.0),
    ));

    app.init_resource::<frame_capture::scene::SceneController>();
    app.add_event::<frame_capture::scene::SceneController>();

    app.add_systems(Startup, setup);

    app.run();
}

pub fn main() {
    headless_app();
}

fn setup(
    mut commands: Commands,
    mut images: ResMut<Assets<Image>>,
    mut scene_controller: ResMut<frame_capture::scene::SceneController>,
    render_device: Res<RenderDevice>,
) {
    let render_target = frame_capture::scene::setup_render_target(
        &mut commands,
        &mut images,
        &render_device,
        &mut scene_controller,
        15,
        String::from("main_scene"),
    );

    commands.spawn(Camera3dBundle {
        transform: Transform::from_xyz(0.0, 6., 12.0).looking_at(Vec3::new(0., 1., 0.), Vec3::Y),
        tonemapping: Tonemapping::None,
        camera: Camera {
            target: render_target,
            ..default()
        },
        ..default()
    });
}

gave me this unexpected error

headless % cargo run
   Compiling headless v0.1.0 (/Users/richardanaya/headless)
    Finished dev [optimized + debuginfo] target(s) in 3.53s
     Running `target/debug/headless`
2024-01-21T17:25:57.039614Z  INFO bevy_render::renderer: AdapterInfo { name: "Apple M1", vendor: 0, device: 0, device_type: IntegratedGpu, driver: "", driver_info: "", backend: Metal }
Adding CaptureFramePlugin
2024-01-21T17:25:57.213653Z  INFO bevy_diagnostic::system_information_diagnostics_plugin::internal: SystemInfo { os: "MacOS 13.2.1 ", kernel: "22.3.0", cpu: "Apple M1", core_count: "8", memory: "8.0 GiB" }
thread '<unnamed>' panicked at /Users/richardanaya/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wgpu-0.17.2/src/backend/direct.rs:2289:30:
Error in Queue::submit: Validation Error

Caused by:
    Buffer (0, 1, Metal) is still mapped

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in exclusive system `bevy_render::renderer::render_system`!
thread 'Compute Task Pool (2)' panicked at /Users/richardanaya/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bevy_render-0.12.1/src/pipelined_rendering.rs:145:45:
called `Result::unwrap()` on an `Err` value: RecvError

my dependencies of my Cargo are

[package]
name = "headless"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
bevy = "0.12.1"
pollster = "0.3.0"
wgpu = "0.17.1"
futures-intrusive = { version = "0.5.0" }

[profile.dev]
opt-level = 1

# Enable high optimizations for dependencies (incl. Bevy), but not for our code:
[profile.dev.package."*"]
opt-level = 3

[cs50victor]

@richardanaya hey, i also use macOS. Think I've seen a similar error. Did you run both release and dev?

[richardanaya]

@richardanaya hey, i also use macOS. Think I've seen a similar error. Did you run both release and dev?

yep, seems like it happens in release. interesting you use macos :/ not sure what the difference is.

[cs50victor]

i can try running it locally if you put up a simple repo based on the code you posted above.

[cs50victor]

@richardanaya the error is coming from wgpu version 0.17.2, the supported version is 0.17.1

[richardanaya]

@cs50victor sorry about the late reply, here is a repo with my just minimal code :)

https://github.com/richardanaya/headless

What's strange is I appear to be using 0.17.1, i'm not sure why it's not working.

[cs50victor]

@richardanaya what's odd is that your repo works but I'm getting a similar error running the headless frame capture in a different project i'm working on. Investigating this issue. It might not be the wgpu crate.

[richardanaya]

@cs50victor interesting :)

We both seem to be using a metal adapter, but i'm on Apple M1

2024-01-21T17:25:57.039614Z INFO bevy_render::renderer: AdapterInfo { name: "Apple M1", vendor: 0, device: 0, device_type: IntegratedGpu, driver: "", driver_info: "", backend: Metal }

maybe something in wgpu is really m1/m2 specific?

[cs50victor]

yh. maybe. came across this issue - denoland/deno#10098

but a fix already got merged in v14

[richardanaya]

I wonder if there is some sneaky intersystem dependency that's not being respected, or rather, maybe it succeeds in certain scheduling of the systems?

[cs50victor]

@richardanaya think I just found the culprit. PR : richardanaya/headless#1

Hey @DGriffin91 know if your frame capture tool is compatible with Bevy's multithreaded feature ?

• The frame capture tool should work when the multithreaded feature is disabled on macOS !

Divided different parts of my previous post into crates here - https://github.com/cs50victor/new_media

[richardanaya]

@cs50victor that worked for me! Thank you. Updated my repo.

[cs50victor]

multithreaded support for anyone looking for something similar in future - cs50victor/newmedia#10