Audio: Write to device's buffer directly without temporary buffers. #90013

kus04e4ek · 2024-03-29T10:14:19Z

Closes Add support for more output types in AudioDriver godot-proposals#8797
Closes Bring the compatibility methods from AudioDriverWASAPI into AudioDriver godot-proposals#8815
Fixes Godot does not listen to microphones with over 2 channels #82823 (will take the first 2 channels) (didn't test, but the code should account for that)

Also, a little bit of a clean-up.

TODO:

Implement for the input buffer;
Test on MacOS/iOS. (I don't have them, so would appreciate some help)

kus04e4ek · 2024-03-29T11:01:43Z

drivers/xaudio2/audio_driver_xaudio2.h

@@ -36,7 +36,6 @@
 #include "core/templates/safe_refcount.h"
 #include "servers/audio_server.h"

-#include <mmsystem.h>


Maybe it's needed for MSVC?

kus04e4ek · 2024-03-29T11:02:01Z

platform/android/audio_driver_opensl.cpp

@@ -30,11 +30,6 @@

 #include "audio_driver_opensl.h"

-#include <string.h>


Probably not needed? Android builds always use clang

kus04e4ek · 2024-03-29T11:03:07Z

platform/web/audio_driver_web.h

@@ -54,7 +54,8 @@ class AudioDriverWeb : public AudioDriver {

 	int buffer_length = 0;
 	int mix_rate = 0;
-	int channel_count = 0;
+	int channels = 0;


Changed the name to match other audio drivers.

kus04e4ek · 2024-03-29T11:05:52Z

servers/audio_server.h

-	virtual SpeakerMode get_speaker_mode() const = 0;
+	virtual int get_total_channels() const = 0;


It could be better to implement new SpeakerMode values instead of get_total_channels?

kus04e4ek · 2024-03-29T11:11:43Z

servers/audio_server.cpp

+
+					if (k == cs - 1 && stride == (uintptr_t)cs * 2 - 1) {
+						// Downmix.
+						_write_sample(p_buffer, idx, (int32_t)(((int64_t)vl2 + (int64_t)vr2) / 2));


Maybe it would be better to pass _write_sample as an argument or make it a virtual method in AudioDriver? It looks like all existing drivers don't need it though

drivers/pulseaudio/audio_driver_pulseaudio.cpp

drivers/wasapi/audio_driver_wasapi.cpp

drivers/coreaudio/audio_driver_coreaudio.cpp

fire · 2024-03-29T14:05:25Z

How would you suggest testing this?

kus04e4ek · 2024-03-29T16:06:01Z

How would you suggest testing this?

Just using AudioStreamPlayer and switching driver in the settings.
On Linux, ALSA and PulseAudio always use 16-bit integers.
On Android, OpenSL always uses 16-bit integers.
On MacOS and iOS, CoreAudio always uses ~~floats~~ 16-bit integers.
On Web, Worklet always uses floats.
On Windows, XAudio2 always uses 16-bit integers, WASAPI uses the most fitting format, format can be obtained only through debugging, there might be different formats for different output devices.
MovieWriter always uses 32-bit integers. Also setting channels can be useful for testing too.
AudioDriverDummy uses no buffer, but still processes audio.

I checked downsampling by modifying PulseAudio code like this:

Patch

diff --git a/drivers/pulseaudio/audio_driver_pulseaudio.cpp b/drivers/pulseaudio/audio_driver_pulseaudio.cpp
index 2e7a3155da1..166ee9472a8 100644
--- a/drivers/pulseaudio/audio_driver_pulseaudio.cpp
+++ b/drivers/pulseaudio/audio_driver_pulseaudio.cpp
@@ -200,6 +200,8 @@ Error AudioDriverPulseAudio::init_output_device() {
 		return err;
 	}
 
+	pa_channel_map_init_mono(&pa_map);
+
 	if (pa_map.channels <= 8) {
 		channels = pa_map.channels;
 	} else {

kus04e4ek · 2024-04-19T12:42:10Z

Web audio is broken on master too. (#90906)
MovieWriter is not recording any video (#90932) and corrupts audio when using any SpeakerMode other than Stereo (can't reproduce it anymore?) on master too.
Apart from that (and the fact that I didn't test on MacOS and iOS), this PR seems fully functional.

kus04e4ek · 2024-04-19T13:09:49Z

servers/audio_server.cpp

+		case AudioDriver::BUFFER_FORMAT_INTEGER_32:
+			// (1u << (32 - 1)) - 1 can't be stored in float correctly.
+			((int32_t *)p_buffer)[p_idx] = (double)p_sample * ((1u << (32 - 1)) - 1);


It is different from the implementation that was before:

godot/servers/audio_server.cpp

Lines 288 to 291 in 9bc49a6

float l = CLAMP(buf[from + j].left, -1.0, 1.0);

int32_t vl = l * ((1 << 20) - 1);

int32_t vl2 = (vl < 0 ? -1 : 1) * (ABS(vl) << 11);

*dest = vl2;

But I'm not sure why it was like this, all I can find on the Internet is how to convert from float to 16-bit integer and it's just to multiply a float to match the range of the number (eg. -1.0 is the minimum value that can fit in 16-bit integer and 1.0 is the maximum), so I expect it to be the same with 32-bit integers?
Also, used everywhere ((1u << (*bits* - 1)) - 1), though it would be better to use (1u << (*bits* - 1)) for negative numbers and ((1u << (*bits* - 1)) - 1) for positive ones. But I think it's really hard to hear the difference so kept it like this for a small optimization (which would be good as audio callbacks really should run as fast as possible).

kus04e4ek · 2024-04-19T13:11:32Z

servers/audio_server.cpp

+		case AudioDriver::BUFFER_FORMAT_INTEGER_24: {
+			int32_t sample = int32_t(((int8_t *)p_buffer)[p_idx * 3 + 2]) << 16;
+			sample |= int32_t(((int8_t *)p_buffer)[p_idx * 3 + 1]) << 8;
+			sample |= int32_t(((int8_t *)p_buffer)[p_idx * 3 + 0]);
+
+			return (float)sample / (1u << (24 - 1));


A sign will probably be wrong, but unsure how to fix that.

kus04e4ek · 2024-04-19T13:11:55Z

servers/audio_server.cpp

+		case AudioDriver::BUFFER_FORMAT_INTEGER_24: {
+			int32_t sample = p_sample * ((1u << (24 - 1)) - 1);
+			int32_t sign = p_sample < 0 ? -1 : 1;
+
+			((int8_t *)p_buffer)[p_idx * 3 + 2] = sign * (ABS(sample) >> 16);
+			((int8_t *)p_buffer)[p_idx * 3 + 1] = sign * (ABS(sample) >> 8);
+			((int8_t *)p_buffer)[p_idx * 3 + 0] = sample;


Same as the above.

kus04e4ek · 2024-04-19T13:14:21Z

platform/web/js/libs/library_godot_audio.js

 				const inb = GodotRuntime.heapSub(HEAPF32, p_in_buf, p_in_size);
 				const input = event.inputBuffer;
+				const input_channels = input.numberOfChannels;
 				if (GodotAudio.input) {
-					const inlen = input.getChannelData(0).length;
-					for (let ch = 0; ch < 2; ch++) {
+					for (let ch = 0; ch < input_channels; ch++) {
 						const data = input.getChannelData(ch);
-						for (let s = 0; s < inlen; s++) {
-							inb[s * 2 + ch] = data[s];
+						for (let sample = 0; sample < data.length; sample++) {
+							inb[sample * input_channels + ch] = data[sample];


Might not be a good idea to pass all of the channels of the input buffer, but thought that downmixing code should be kept in the same place for all AudioDrivers.

Faless · 2024-04-19T13:53:39Z

Web audio is broken on master too.

Can you expand on this? If you use threaded export audio should work fine.

kus04e4ek · 2024-04-19T16:09:14Z

Web audio is broken on master too.

Can you expand on this? If you use threaded export audio should work fine.

Created an Issue: Web audio is distorted when using headphones #90906.

kus04e4ek · 2024-04-29T12:20:55Z

drivers/SCsub

+if env["platform"] == "windows" and not env.msvc:
+    SConscript("backtrace/SCsub")


Moved out of Sounds drivers section

kus04e4ek · 2024-04-29T12:23:39Z

drivers/SCsub

+if env["platform"] in ["ios", "macos"]:
+    SConscript("coreaudio/SCsub")
+
+if env["platform"] == "linuxbsd":
+    if env["alsa"]:
+        SConscript("alsa/SCsub")
+    if env["pulseaudio"]:
+        SConscript("pulseaudio/SCsub")
+
 if env["platform"] == "windows":
+    if env["xaudio2"]:
+        SConscript("xaudio2/SCsub")
    SConscript("wasapi/SCsub")


I think it's better this way, as no empty files will be compiled. Maybe it would be better to add checks like this to other drivers? Also should I delete checks like #ifdef ALSA_ENABLED in files?

kus04e4ek · 2024-04-29T12:25:14Z

drivers/xaudio2/audio_driver_xaudio2.cpp


 	int latency = Engine::get_singleton()->get_audio_output_latency();
-	buffer_size = closest_power_of_2(latency * mix_rate / 1000);


I don't think power of 2 is required, can't find anything about it online

kus04e4ek · 2024-04-29T12:26:47Z

drivers/xaudio2/audio_driver_xaudio2.h

-	WAVEFORMATEX wave_format = { 0 };
+	WAVEFORMATEX wave_format;


Fixes g++ warning

kus04e4ek · 2024-04-29T12:36:23Z

platform/web/audio_driver_web.cpp

+	if (unlikely(audio_context.input_channels == 0)) {
+		audio_context.input_channels = godot_audio_get_input_channels();
+	}


Don't really know if there is a better way to initialize input_channels

kus04e4ek · 2024-04-29T12:47:52Z

servers/audio_server.h

 	unsigned int input_position = 0;
 	unsigned int input_size = 0;

-	void audio_server_process(int p_frames, int32_t *p_buffer, bool p_update_mix_time = true);
+	void audio_server_process(int p_frames, void *p_buffer, bool p_active = true, bool p_update_mix_time = true);


Not really sure about p_active, it tells whether or not to process audio or just fill buffer with 0. I think it should be an argument, because AudioDriverOpenSL and AudioDriverCoreAudio pass false if callback is unable to lock mutex, it doesn't feel right to have a function like is_active because of that. input_process has something simmilar to this, if you pass nullptr as a buffer (it's needed for AudioDriverWASAPI), internall buffer will be filled with 0, we can't pass nullptr in audio_server_process, as we still need to fill a buffer. It's just a simplification to not call memset and to not reapeat the same code

kus04e4ek · 2024-07-06T16:29:38Z

Marked as a draft, cause I want to separate some changes into different PRs/commits

kus04e4ek requested review from a team as code owners March 29, 2024 10:14

AThousandShips added enhancement topic:audio labels Mar 29, 2024

AThousandShips added this to the 4.x milestone Mar 29, 2024

kus04e4ek force-pushed the audio-fix branch from 6fbf194 to e6f62c3 Compare March 29, 2024 11:16

kus04e4ek commented Mar 29, 2024

View reviewed changes

kus04e4ek force-pushed the audio-fix branch from e6f62c3 to f35f1e4 Compare March 29, 2024 11:24

kus04e4ek force-pushed the audio-fix branch from f35f1e4 to 01a3669 Compare April 19, 2024 12:41

kus04e4ek force-pushed the audio-fix branch from 01a3669 to bcf65b5 Compare April 19, 2024 12:48

kus04e4ek commented Apr 19, 2024

View reviewed changes

kus04e4ek force-pushed the audio-fix branch 2 times, most recently from e86d4df to b1d4f24 Compare April 19, 2024 13:29

kus04e4ek mentioned this pull request Apr 20, 2024

Godot does not listen to microphones with over 2 channels #82823

Open

kus04e4ek force-pushed the audio-fix branch from b1d4f24 to 453bc79 Compare April 20, 2024 11:25

kus04e4ek closed this Apr 20, 2024

kus04e4ek force-pushed the audio-fix branch from 453bc79 to 9bc49a6 Compare April 20, 2024 11:43

kus04e4ek reopened this Apr 20, 2024

kus04e4ek force-pushed the audio-fix branch 3 times, most recently from 4320fea to 620e598 Compare April 29, 2024 12:17

kus04e4ek requested a review from a team as a code owner April 29, 2024 12:17

kus04e4ek commented Apr 29, 2024

View reviewed changes

Audio: Write to device's buffer directly without temporary buffers.

f39ef50

kus04e4ek force-pushed the audio-fix branch from 620e598 to f39ef50 Compare May 3, 2024 14:24

kus04e4ek mentioned this pull request Jun 6, 2024

Replace the current audio driver with Oboe on Android #86776

Draft

13 tasks

kus04e4ek marked this pull request as draft July 6, 2024 16:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Audio: Write to device's buffer directly without temporary buffers. #90013

Audio: Write to device's buffer directly without temporary buffers. #90013

kus04e4ek commented Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

fire commented Mar 29, 2024

kus04e4ek commented Mar 29, 2024 •

edited

Loading

kus04e4ek commented Apr 19, 2024 •

edited

Loading

kus04e4ek Apr 19, 2024

kus04e4ek Apr 19, 2024

kus04e4ek Apr 19, 2024

kus04e4ek Apr 19, 2024 •

edited

Loading

Faless commented Apr 19, 2024

kus04e4ek commented Apr 19, 2024

kus04e4ek Apr 29, 2024

kus04e4ek Apr 29, 2024 •

edited

Loading

kus04e4ek Apr 29, 2024

kus04e4ek Apr 29, 2024

kus04e4ek Apr 29, 2024

kus04e4ek Apr 29, 2024

kus04e4ek commented Jul 6, 2024

		@@ -30,11 +30,6 @@

		#include "audio_driver_opensl.h"

		#include <string.h>

		virtual SpeakerMode get_speaker_mode() const = 0;
		virtual int get_total_channels() const = 0;

	float l = CLAMP(buf[from + j].left, -1.0, 1.0);
	int32_t vl = l * ((1 << 20) - 1);
	int32_t vl2 = (vl < 0 ? -1 : 1) * (ABS(vl) << 11);
	*dest = vl2;

		if env["platform"] == "windows" and not env.msvc:
		SConscript("backtrace/SCsub")


		int latency = Engine::get_singleton()->get_audio_output_latency();
		buffer_size = closest_power_of_2(latency * mix_rate / 1000);

Audio: Write to device's buffer directly without temporary buffers. #90013

Are you sure you want to change the base?

Audio: Write to device's buffer directly without temporary buffers. #90013

Conversation

kus04e4ek commented Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

kus04e4ek Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

kus04e4ek Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

kus04e4ek Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

kus04e4ek Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

fire commented Mar 29, 2024

kus04e4ek commented Mar 29, 2024 • edited Loading

kus04e4ek commented Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kus04e4ek Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

Faless commented Apr 19, 2024

kus04e4ek commented Apr 19, 2024

Choose a reason for hiding this comment

kus04e4ek Apr 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kus04e4ek commented Jul 6, 2024

kus04e4ek commented Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek Mar 29, 2024 •

edited

Loading

kus04e4ek commented Mar 29, 2024 •

edited

Loading

kus04e4ek commented Apr 19, 2024 •

edited

Loading

kus04e4ek Apr 19, 2024 •

edited

Loading

kus04e4ek Apr 29, 2024 •

edited

Loading