Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add first draft of mixer library #1

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add first draft of mixer library #1

wants to merge 5 commits into from

Conversation

glopesdev
Copy link
Member

@glopesdev glopesdev commented Nov 8, 2024

This PR sketches the high-level architecture for the audio mixer library. The basic entity-relationship diagram for the draft mixer library is below:

erDiagram
    MIXER_STREAM ||--o{ MIXER_BUFFER : queues
    MIXER_STREAM {
        int deviceIndex
        double sampleRate
    }
    MIXER_BUFFER {
        float[] samples
        float[] channelScale
    }
Loading

The overall idea is that a MixerStream connects to a specified device and configures, opens and starts a new playback stream. The samples in the stream are served by a single callback, which pulls samples from a collection of MixerBuffer objects. Each MixerBuffer contains a 1D buffer of samples and a vector of weights, specifying a scale factor for mixing each sample in the output buffer serviced by the stream callback. When all samples from each buffer have been mixed, the buffer is removed from the list.

The approach is naive but functional, and will be used as the basis for further optimization. StartMixer and StopMixer are used as tentative control nodes to allow pre-queueing multiple buffers for playback.

Example workflow:

image
Workflow
<?xml version="1.0" encoding="utf-8"?>
<WorkflowBuilder Version="2.8.5"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns:rx="clr-namespace:Bonsai.Reactive;assembly=Bonsai.Core"
                 xmlns:dsp="clr-namespace:Bonsai.Dsp;assembly=Bonsai.Dsp"
                 xmlns:scr="clr-namespace:Bonsai.Scripting.Expressions;assembly=Bonsai.Scripting.Expressions"
                 xmlns:p1="clr-namespace:Bonsai.Mixer;assembly=Bonsai.Mixer"
                 xmlns="https://bonsai-rx.org/2018/workflow">
  <Workflow>
    <Nodes>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="rx:Range">
          <rx:Start>0</rx:Start>
          <rx:Count>3</rx:Count>
        </Combinator>
      </Expression>
      <Expression xsi:type="rx:SelectMany">
        <Workflow>
          <Nodes>
            <Expression xsi:type="WorkflowInput">
              <Name>Source1</Name>
            </Expression>
            <Expression xsi:type="rx:AsyncSubject">
              <Name>Index</Name>
            </Expression>
            <Expression xsi:type="SubscribeSubject">
              <Name>Index</Name>
            </Expression>
            <Expression xsi:type="Add">
              <Operand xsi:type="IntProperty">
                <Value>1</Value>
              </Operand>
            </Expression>
            <Expression xsi:type="Multiply">
              <Operand xsi:type="FloatProperty">
                <Value>261.626</Value>
              </Operand>
            </Expression>
            <Expression xsi:type="PropertyMapping">
              <PropertyMappings>
                <Property Name="Frequency" />
              </PropertyMappings>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="dsp:FunctionGenerator">
                <dsp:BufferLength>44100</dsp:BufferLength>
                <dsp:Frequency>784.8780517578125</dsp:Frequency>
                <dsp:Waveform>Sine</dsp:Waveform>
                <dsp:SampleRate>44100</dsp:SampleRate>
                <dsp:Depth>F32</dsp:Depth>
                <dsp:Amplitude>1</dsp:Amplitude>
                <dsp:Offset>0</dsp:Offset>
                <dsp:Phase>0</dsp:Phase>
              </Combinator>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="rx:Take">
                <rx:Count>1</rx:Count>
              </Combinator>
            </Expression>
            <Expression xsi:type="dsp:ConvertToArray">
              <dsp:Depth>F32</dsp:Depth>
            </Expression>
            <Expression xsi:type="SubscribeSubject">
              <Name>Index</Name>
            </Expression>
            <Expression xsi:type="scr:ExpressionTransform">
              <scr:Expression>TimeSpan.FromSeconds(0.5 * it)</scr:Expression>
            </Expression>
            <Expression xsi:type="PropertyMapping">
              <PropertyMappings>
                <Property Name="DueTime" />
              </PropertyMappings>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="rx:DelaySubscription">
                <rx:DueTime>PT1S</rx:DueTime>
              </Combinator>
            </Expression>
            <Expression xsi:type="WorkflowOutput" />
          </Nodes>
          <Edges>
            <Edge From="0" To="1" Label="Source1" />
            <Edge From="2" To="3" Label="Source1" />
            <Edge From="3" To="4" Label="Source1" />
            <Edge From="4" To="5" Label="Source1" />
            <Edge From="5" To="6" Label="Source1" />
            <Edge From="6" To="7" Label="Source1" />
            <Edge From="7" To="8" Label="Source1" />
            <Edge From="8" To="12" Label="Source1" />
            <Edge From="9" To="10" Label="Source1" />
            <Edge From="10" To="11" Label="Source1" />
            <Edge From="11" To="12" Label="Source2" />
            <Edge From="12" To="13" Label="Source1" />
          </Edges>
        </Workflow>
      </Expression>
      <Expression xsi:type="rx:BehaviorSubject">
        <Name>Buffer</Name>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="p1:CreateMixerContext">
          <p1:HostApi>Windows WASAPI</p1:HostApi>
          <p1:DeviceName>Speakers (Realtek(R) Audio)</p1:DeviceName>
          <p1:SampleRate>44100</p1:SampleRate>
        </Combinator>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="p1:StartMixer" />
      </Expression>
      <Expression xsi:type="rx:BehaviorSubject">
        <Name>Mixer</Name>
      </Expression>
      <Expression xsi:type="SubscribeSubject">
        <Name>Buffer</Name>
      </Expression>
      <Expression xsi:type="SubscribeSubject">
        <Name>Mixer</Name>
      </Expression>
      <Expression xsi:type="PropertyMapping">
        <PropertyMappings>
          <Property Name="Mixer" />
        </PropertyMappings>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="p1:QueueBuffer">
          <p1:ChannelScale>
            <p1:float>0.5</p1:float>
            <p1:float>0.5</p1:float>
          </p1:ChannelScale>
        </Combinator>
      </Expression>
    </Nodes>
    <Edges>
      <Edge From="0" To="1" Label="Source1" />
      <Edge From="1" To="2" Label="Source1" />
      <Edge From="3" To="4" Label="Source1" />
      <Edge From="4" To="5" Label="Source1" />
      <Edge From="6" To="9" Label="Source1" />
      <Edge From="7" To="8" Label="Source1" />
      <Edge From="8" To="9" Label="Source2" />
    </Edges>
  </Workflow>
</WorkflowBuilder>

@glopesdev glopesdev added the feature New planned feature label Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New planned feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant