-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AudioMixer with effects #8974
Comments
I wanted to add effects on synth channels and saw this issue also about adding audio effects. After some discussion on discord and looking through the code and some other audio libraries I see all the audio code uses get My idea is to create a new module At this point I have done a proof-of-concept to see if a basic echo was possible on an RP2350. It worked fine and the memory allowed for a long buffer delay. The POC was done without setting up a module just an in-place test. If this sounds like a decent idea I'd next start on a more complete test with a separate module and maybe hooked into two+ audio components. Anyone have any thoughts or comments? |
Putting some ideas down both for others to critique and to organize my own thoughts: This is basic code to play a note: audio = audiobusio.I2SOut(...)
mixer = audiomixer.Mixer(voice_count=1, channel_count=1,...)
audio.play(mixer)
synth = synthio.Synthesizer(channel_count=1, sample_rate=44100)
mixer.voice[0].play(synth)
note = synthio.Note(261)
synth.play(note) There are two potential ways to add effects. The first as tannewt suggested: # ... similar to above
echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
echo.play(synth)
mixer.voice[0].play(echo) In theory effects could be chained together e.g. Another way is to add effects to audio objects. # ... similar to above
echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
mixer.voice[0].play(synth)
synth.addeffect(echo) This method also allowed effects to be chained, call addeffect again e.g. Anyone have any thoughts? Are dynamically changing effects a good idea? Would we want effects on a per channel basis in the synth? In the meantime I'm going to keep trying things out to see what may work best as time permits. |
First, this is very cool! So that would I guess look like: echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
mixer.voice[0].play(synth)
mixer.voice[1].play(wavfile)
mixer.addeffect(echo) # add effect to mixed output
# and then also
echo2 = audioeffects.EffectEcho(delay=150, decay=0.7) # define another echo effect
mixer.voice[0].addeffect(echo2) # add effect only to voice 0 |
I think it's important to think about how you'd remove an effect as well. I do think we want to change them dynamically. I don't think we need them on a synth channel basis. Instead, you can have two synths. |
So no audio effects for |
That's not what I meant. I was responding to "Would we want effects on a per channel basis in the synth?". I still think the best way is through separate intermediate objects that get played. echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
echo.play(synth)
# with echo
mixer.voice[0].play(echo)
time.sleep(1)
# without
mixer.voice[0].play(synth) |
After playing around with the code all weekend the first way @tannewt suggested I think makes the most sense. Looking at how the Teensy audio library works it seems similar. # give the synth and echo effect
echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
echo.play(synth)
# give a wave file a chorus effect
wavesound = audiocore.WaveFile("wave.wav")
chorus = audioeffects.Chorus(voices=5)
chorus.play(wavesound)
# combine them in a mixer
mixer.voice[0].play(echo)
mixer.voice[1].play(chorus)
# add a reverb to the overall mixed sound
reverb = audioeffects.EffectReverb()
reverb.play(mixer) I have to look closer but in some instances playing objects into others may reset buffers / reset the sound but that can be tweaked easily enough. I am getting closer to having some code I can push as a draft PR to give something more concrete to look at. |
Normally Would representing the signal chain in code, rather than as a data structure, create glitches when the chain is changed? Let me think through this with an example. e.g. let's set up a synth playing through chorus & reverb, then remove the chorus: # standard audio setup
audio = audiobusio.I2SOut(...)
mixer = audiomixer.Mixer(..., buffer_size=2048)
audio.play(mixer)
synth = synthio.Synthesizer()
# mixer.voice[0].play(synth) is what we'd normally do here, instead...
# wire up effects: synth -> chorus -> reverb -> mixer -> i2s
chorus = audioeffects.Chorus(voices=5)
chorus.play(synth)
reverb = audioeffects.Reverb()
reverb.play(chorus)
mixer.voice[0].play(reverb)
# time passes, remove chorus from signal chain: synth -> reverb -> mixer -> i2s
reverb.play(synth) # would this cause a glitch? I forget if doing the equivalent does in the TAL. I'll see if I can find my Teensy audio boards and try it. If all audio effects have a "wet/dry" mix knob and zero-effort pass-through case when dry=100%, then we can get glitchless removal of an effect without altering the signal chain. |
This was more an example of "you could do this", then any practical or good idea.
I'm still not sure about this case. I do want it to work, or to have a way to switch effects in/out at runtime. I'm just not sure the optimal way to do that yet. I did get a "do nothing"/pass-thru effect working tonight. Next step to make it do something. |
+1 on @tannewt 's suggestion of running audio buffer sources through the effect and then to the final mixer object before the output. In a way, I think it is reminiscent of guitar pedals as to how you'd chain them together. Though I know the naming scheme we're playing around with isn't final, I think Some of the effects that I'd potentially like to see:
Something I remember seeing this repo a while back that was able to achieve some decent results with the rp2040: https://github.com/StoneRose35/cortexguitarfx. |
I like the direction of this. I want to point out we probably want more specific module names instead of |
I actually had realized the same thing and in my proof-of-concept code changed it already.
For now I'm trying to get a base framework up, and willing to look at other effect but I'm not an expert so probably need some guide on what/how they work. But I like the ideas! |
Would more modules be preferrable over having flags to turn on/off individual effects within one module? I would think we would still want some broad categories and not one module/one effect? New modules aren't hard so no real preference from me. Something that doesn't have to be decided this moment still at least. |
Once we have the framework set up, I'd love to contribute where needed.
Personally, I'd like to see it all compiled into one module and then disabled on an individual effect basis. I feel that would provide more cohesion in the implementation, but I'd really like to see what you might have in mind, @tannewt . |
I prefer separate modules because import errors normally happen early on startup. If you have optional portions of the module then you'll find its missing later. It'd be ok if related effects are in the same module, especially if they share code under the hood. |
So at this point I have three questions:
Probably cannot be at the CircuitPython meeting tomorrow as these may be good for in the weeds. But I'm about here/discord if anyone wants to discuss anything. |
If the grouping desire is based on memory usage, then perhaps grouping names that imply "no buffer" vs. "small buffer" vs. "big buffer". e.g. "Compressor" would go in the "no buffer" group, "Chorus" in "small buffer" and "Reverb" in the "big buffer" group. Otherwise, maybe organize by user-facing effect type. Groups like:
This would mostly match the memory usage-based grouping, so I think I like it more.
I'd like to see a "mix" parameter being part of the standard API, to adjust how much of the effect to apply: 0.0 = no effect / 100% "dry" to 1.0 = only effect / 100% "wet", defaults to 0.5. And it would be nice if "mix=0.0" would be a "true bypass" that would be an early return path to minimize processing. I'd be willing to try out any PR you put out and try making a few simple effects too. This is very neat! |
Yup! A PR is a perfect place to discuss this. No need to decide beforehand. My main driver for separate modules is code size. Small amounts of code can fit before the larger ones.
Yup! Doesn't need to be a draft either.
I think mixer is a good spot for this code. We can assume we have mixer when we have these effects. |
Those make sense to me so we could have:
Not that it isn't hard to add more categories later.
That should not be that hard to do as a final step, take the output buffer * mix and add the original sample * 1-mix. Also easy to check if mix=0.0 just bypass it all early. |
It would be nice if effects could be added to the voices on the audio mixer. For example: pitch (playback rate and/or time-stretch), reverb and EQ maybe?
This can't be done near-realtime in Python, so it is better suited to a compiled library. Most effects would require buffering but should be achievable on something like a Pico.
The text was updated successfully, but these errors were encountered: