Live Acoustic Neural Harmonizer

View GitHub Repo When creating musical ideas, it is often hard to concepualize the full scope of harmonies and rhythms when working alone. Having new background ideas availble in real-time greatly expands the freedom and flexibility musicians have when demoing and/or tinkering with ideas, even with just a single instrument or voice. This project aims to create a real-time system that can take a live audio signal from any instrument and generate real-time harmonies through MIDI.

Demo

System Pipeline

---
config:
  theme: dark
---
flowchart TB
	A[Live Audio]
	B[torchcrepe Pitch Detection]
	C{Confidence Gate}
	D[Notochord Harmonization]
	E[DAW / Software Synth]
	F[Harmonized Output]

    subgraph detect[" "]
    direction TB
	    A --> B --> C
    end

    subgraph harmonize[" "]
    direction LR
	    D --> E --> F
    end
  
    detect -- pass - send over OSC --> harmonize

    style A fill:none,stroke:none
    style F fill:none,stroke:none

---
title: Detailed Detection Pipeline
config:
  theme: dark
  themeVariables:
    secondaryColor: '#2E3440'
---
flowchart LR
	A[Live Audio]
    B[sounddevice Capture @10Hz]
	C[torchcrepe Pitch Detection]
	D{Confidence Gate}
    E[Convert to MIDI]
	F[Send over OSC]

	A --> B --> C --> D -- pass --> E --> F
    D -- fail --> B

    style A fill:none,stroke:none
    style F fill:none,stroke:none

Notochord is a real-time neural network model for MIDI performances developed by the Intelligent Instruments Lab. It was designed utilizing autoregression and probabilistic formulations to allow for extreme low-latency performance usage. The MIDI data recieved from torchcrepe is run through this predictive model to determine a harmonized note. The model outputs the MIDI representation of its predicted note, which is then routed through to a DAW/software synthesize to be played aloud.

Latency?

Testing Harmonization

Musician’s POV

Future Improvements