Spleeter Model

What is Spleeter?

Spleeter is an open-source music source separation model developed by Deezer. It separates audio into 4 stems: vocals, drums, bass, and other.

Key features:

• Based on U-Net architecture
• Works on spectrograms
• Trained on MUSDB18 dataset (150 songs with stems)
• Open source (MIT license)

U-Net Architecture

U-Net is a convolutional neural network with an encoder-decoder structure and skip connections.

// U-Net structure
Encoder:
    Input: [batch, channels, time, freq]
    ↓ Conv + ReLU + BatchNorm
    ↓ MaxPool (downsample)
    ↓ Conv + ReLU + BatchNorm
    ↓ MaxPool
    ↓ ... (more layers)
    
Bottleneck:
    ↓ Conv + ReLU + BatchNorm
    
Decoder:
    ↑ ConvTranspose (upsample)
    ↑ Concatenate skip connection
    ↑ Conv + ReLU + BatchNorm
    ↑ ConvTranspose
    ↑ Concatenate skip connection
    ↑ ... (more layers)
    
Output: [batch, 4, time, freq]
// 4 channels: vocals, drums, bass, other

The skip connections help preserve fine details that would be lost in downsampling.

Spectrogram Processing

// Convert audio to spectrogram
function audioToSpectrogram(audioBuffer, fftSize = 4096, hopSize = 1024) {
    const samples = audioBuffer.getChannelData(0); // Mono
    const numFrames = Math.floor((samples.length - fftSize) / hopSize) + 1;
    const numBins = fftSize / 2;
    
    const spectrogram = new Float32Array(numFrames * numBins);
    
    for (let frame = 0; frame < numFrames; frame++) {
        const start = frame * hopSize;
        const segment = samples.slice(start, start + fftSize);
        
        // Apply window function
        const windowed = applyHannWindow(segment);
        
        // FFT (simplified)
        const complex = fft(windowed);
        
        // Magnitude spectrum
        for (let bin = 0; bin < numBins; bin++) {
            const magnitude = Math.sqrt(
                complex[bin].real * complex[bin].real + 
                complex[bin].imag * complex[bin].imag
            );
            spectrogram[frame * numBins + bin] = magnitude;
        }
    }
    
    return spectrogram;
}

// Convert spectrogram back to audio
function spectrogramToAudio(spectrogram, fftSize = 4096, hopSize = 1024) {
    // Inverse STFT (Short-Time Fourier Transform)
    // 1. Convert magnitude to complex (add phase)
    // 2. Inverse FFT each frame
    // 3. Overlap-add frames
    // ... complex implementation
}

// Spleeter works on magnitude spectrograms
// Phase is estimated or reconstructed

Training Data

Spleeter was trained on MUSDB18, a dataset of 150 full-length songs with separated stems:

• 100 songs for training
• 50 songs for testing
• Each song has: mix, vocals, drums, bass, other
• Genres: rock, pop, electronic, hip-hop, etc.

The model learns to map mixed spectrograms to separated spectrograms by minimizing the difference between predicted and actual stems.

// MixMind's Spleeter implementation
class SpleeterModel {
    constructor() {
        this.session = null;
    }
    
    async load() {
        // Load ONNX model
        this.session = await ort.InferenceSession.create('./models/spleeter.onnx');
    }
    
    async separate(audioBuffer) {
        // 1. Preprocess
        const spectrogram = this.audioToSpectrogram(audioBuffer);
        const normalized = this.normalize(spectrogram);
        const inputTensor = this.createTensor(normalized);
        
        // 2. Run model
        const outputs = await this.session.run({ 'input_1': inputTensor });
        
        // 3. Postprocess
        const stems = {
            vocals: this.spectrogramToAudio(outputs['vocals']),
            drums: this.spectrogramToAudio(outputs['drums']),
            bass: this.spectrogramToAudio(outputs['bass']),
            other: this.spectrogramToAudio(outputs['other'])
        };
        
        // 4. Post-processing cleanup
        const cleaned = this.cleanupStems(stems);
        
        return cleaned;
    }
    
    // Post-processing: remove artifacts
    cleanupStems(stems) {
        // Apply high-pass filter to vocals (remove low rumble)
        stems.vocals = this.applyFilter(stems.vocals, 'highpass', 80);
        
        // Smooth instrumental (reduce harshness)
        stems.other = this.applyFilter(stems.other, 'peaking', 3000, -3);
        
        // Normalize levels
        stems = this.normalizeLevels(stems);
        
        return stems;
    }
}

// Model details:
// - Input: [1, 1, time, 1024] (mono spectrogram)
// - Output: [1, 4, time, 1024] (4 stem spectrograms)
// - Parameters: ~30 million
// - Size: ~120 MB (compressed)
// - Inference time: ~5-10 seconds per 5-second chunk