Loop Supreme, part 11: Exporting stems and changing inputs

This is part 11 in a series about building a browser-based audio live looper

• Part 12: v1.0 release, and project retro
• Part 11: Exporting stems and changing inputs
• Part 10: Keyboard bindings
• Part 9: Visualizing the waveform
• Part 8: Building and hosting
• Part 7: Latency and adding Track functionality
• Part 6: Workers and AudioWorklets
• Part 5: Record and loop a track
• Part 4: Adding a Scene
• Part 3: Metronome click
• Part 2: Adding a Metronome
• Part 1: New project: building a web-based audio looper!

Goal

• Add the ability to export stems
• Add the ability to change inputs per-track

Implementation

Lots has happened since the last update! I found myself going down rabbit hole after rabbit hole. As I got the two main “goals” finished, I realized that the most important aspect of the app - recording audio - was still working very poorly. There was a bizarre attenuation effect on the audio recorded from my audio interface that made the app effectively unusable. Trying to fix this led me down several refactors that were ultimately unnecessary. Bizarrely, the fix was extremely simple and unrelated to any code I actually wrote. Rather, it was an implementation detail of the Web Audio API.

Another bug I noticed was that, even with the timing correction I made in #23, loops were experiencing noticeable drift over the course of several loops. I decided that the most pragmatic solution was to stop using the loop parameter on the AudioBufferSourceNode and simply restart the buffer on each loop start event. This seems to introduce a bit more clipping on loop start/end, but for the moment it’s a worthwhile tradeoff for accurate timing.

In addition to fixing those two major bugs, I did add the ability to export stems, and also change inputs on a track. Both of these were important. Changing inputs is important because the “default device” is not always what a user might expect in a browser. And exporting audio is important because it’s fun to have artifacts of what you record! I used another Worker for the exporter, and copied some code from a Google project to write the WAV file blob, so it was pretty easy to implement. The hardest part was learning that an AudioBuffer couldn’t be serialized directly in a Worker message, so I had to extract the Float32Arrays from the channel data directly to export it. I’m not convinced this was worth it, because writing the WAV file is pretty fast and could probably happen on the UI thread, but it doesn’t hurt anything and now it’s done 🤷🏻. Adding the option to change inputs was also pretty straight forward. The Web Audio API provides an option to enumerate devices, so it was just a matter of setting the MediaStream when the device changed.

There were lots of minor bug fixes and improvements, but those are the big updates to cover! I tried to keep notes in the PRs for each specific fix that I made.

Learnings

• Audio constraints are important! As noted above, I was getting a really horrible attenuation modulation effect when recording tracks. I thought it was due to some weird React architecture but it turns out that specifying constraints on the user media request totally fixed it! Very strange, but I’m glad it was an easy solution
• Worker messages cannot handle all data types. AudioBuffers cannot be passed in a postMessage function, which surprised me. I imagine that most types can be serialized just fine, but it’s good to know that certain native classes cannot be. Luckily, it’s fairly easy to extract the channel data as raw Float32Arrays, which work just fine as a Worker message.
• Keyboard bindings deserve to be defined at the component level, rather than in a global context. Defining them globally introduces all sorts of weird edge cases around duplicate key handlers and weird behavior when mounting/unmounting components rapidly. Using a hook that can be called at the component level made this design much more straightforward.
• Be careful of extra renders! One of the rabbit holes I went down when trying to debug the recording quality issue was fixing extra renders in the Scene and Track components. Originally, I nested the Scene under the Metronome component. This is a fine choice in itself, but unfortunately, I was attaching the currentTick property to an object that was passed to every child. This meant that every child was re-rendered on every beat. This is no good! Lot’s of extra work being done. I realized there was only 1 piece of UI that actually needed to update on each beat: the BeatCounter component. I moved the relevant event listener and state into that component, and it fixed the issue. I also wanted to de-couple the components of my app a bit more, so I placed Scene as a peer component of Metronome and just initialized the Clock worker as a parent. This was a nice simplification and I think made the data flow a lot easier to follow.
• Looping in time is hard! I have no doubt that using scheduled events on the AudioContext clock would fix this issue, but I still think that would make my life very challenging to deal with changes to the metronome settings. For my purposes, it seems that starting a new AudioBufferSourceNode on each loop start is the best way to achieve consistent timing with the metronome.

State of the app

• Merged PRs
  • #25 UX improvements
  • #26 Export to WAV
  • #27 Loop start correction
  • #28 Refactor props tree
  • #29 Key bindings hook
  • #30 Fix recording settings (define audio constraints)
• Most major features are working!
  • Exporting stems
  • Changing input per track
  • Keyboard bindings
  • Looping playback with reasonable accuracy

Next steps

• Potentially improve garbage collection by disconnecting unused components
• A few minor UI improvements
• See if there is an easy way to record a performance