Sound driver ideas

[dlech]

Since I was working on sound drivers in Pybricks MicroPython recently, just wanted to write down a few ideas before I forget since I'm not going to get around to them any time soon.

• Drivers should have separate tone mode and PCM playback mode. In the current implementation there is only PCM playback. To make tones, it currently requires a bit rate of 128 times the frequency of the tone which results in crazy high bit rates for higher frequencies. Actual max bit rate for PCM playback should be 22050 Hz. Handling of tone mode can be platform-specific depending on the available hardware.
• PCM playback should accept 8-bit unsigned data (and possibly/optionally signed 16-bit data). Currently it is implemented using 16-bit unsigned. For resource-constrained systems like NXT, using 8-bit cuts the sound file size in half. And if we do allow 16-bit, it should be signed instead of unsigned since that is what audio editing programs generally deal with.
• If we want to support large sound files (bigger than will fit in RAM all at once) on platforms with external flash memory (probably just SPIKE at this time since EV3 has huge RAM) we would need a Python API to load the next chunk of sound while the current chunk is still playing and a boolean flag to test when the buffer becomes available again and we can write the next chunk.
• Volume control should be moved to the driver level. Currently it is implemented by modifying the PCM data as it is generated, but in the future, PCM data may be in ROM or some other source that should not be modified. Also some drivers have to modify each sample anyway, so it makes sense to do it in the driver all at once instead of iterating the PCM data twice. Lejos has a nice method of generating a volume table for 8-bit data that we could use. Calling set_volume would generate this table, then it would be used until the volume is changed.
• Instead of adding support for reading sound files like .wav or LEGO .rsf files in the firmware (and associated changes to pybricksdev/Pybricks Code to support downloading such files) we could add a feature to Pybricks Code to convert a sound file to Python code. It might work like this:
  • drag and drop or click import button and select a sound file or use microphone to record a new sound
  • user selects name for the new Python file (default name based on sound file name)
  • possibly allow user to select sample size (8-bit/16-bit) and bit rate (8kHz to 22050Hz) to balance size vs. sound quality - UI should show final size in KB so the user can make an informed decision
  • convert the sound file to the selected format
  • generate a Python file that has the sample data as a bytes literal (split into chunks if needed) and a function to play it - it can also have a comment stating the path of the original file that was imported and the size of the data in bytes
  • then users just import the play function from the new Python file to use it in other programs

[redmerl]

It would be awesome if this could be implemented. Gives robot creations so much more personality (i'm working on a Wall-E with lots of functions). I would love to help with some testing if that speeds things up.

[bhintzma]

I will add my vote for support to play .wav or .mp3 files on Spike Prime. @dlech - The kids on our FLL team have been devoted users of ev3dev for several years. We've just moved to Spike Prime and are only now realizing that Pybricks is where you're spending most of your time now. We're totally sold, huge fans! The kids want to add playing sound files (Rick Roll) when the bot completes each mission. Fingers crossed that you have time for this soon. Like @redmerl, I am also a willing beta-tester if needed.

[DanieleBenedettelli]

@dlech any updates on this feature?
Is there a way to access the file system to upload the files on SPIKE, if the file system even exists?

Thanks!

[dlech]

No, we've been focusing on other areas so haven't made any progress with sound or file systems.

[dsplaisted]

It's outside my programming area of expertise, but I'd like to try to add support for playing sound files for my FLL team this year. Can you provide guidance or links to documentation, samples, or similar code that would help me do this?

Looking at the current code, I think I more or less understand how the speaker type creates the square wave form and passes it to pbdrv_sound_start. Would it work to do the same thing but instead of using a 128 word waveform, to use a much longer array that includes the entire sound file data?

For our purposes it's probably easiest to hard-code the sound file data in the firmware code. Wav files are pretty big though, what is the size limit for the firmware for Spike Prime?

Thanks!

[dlech]

This was my attempt to make 8-bit PCM sound work, but I didn't actually get it working: pybricks/pybricks-micropython@master...dlech:pybricks-micropython:pcm

The on-chip flash is 1MB and we are only using a bit over 1/4 of that, so there should be a decent amount of room for a few sounds, even if they end up needing to be 16-bit.

[dsplaisted]

I'd like to play a song that lasts for a 2.5 minute robot run...

How much program memory does the Spike Prime Hub have?

[dlech]

Not enough for 2 minutes for sure.

But you can use play_notes() to play a song already. It isn't as nice of quality as a wav file, but it will be louder, which is good when the motors are running. And no custom firmware required!