Can a Machine Hear How You Feel?
Speech Emotion Recognition with CNNs

Hugo Durieux
École polytechnique · RO11 course project · September 2026
Code Live demo Dataset Results

Abstract

We classify seven emotions (anger, boredom, disgust, fear, happiness, sadness and neutral) from short utterances, using log-mel spectrograms as images. We compare two convolutional networks. The first is a small CNN trained from scratch (99 k parameters). The second is a ResNet-18 pretrained on ImageNet, kept frozen, with a new linear head. Both are evaluated on the Berlin Emotional Speech Database (EmoDB) with leave-two-speakers-out cross-validation, so every score is measured on voices the model never heard.

The CNN trained from scratch reaches 75.4 % unweighted average recall, against 61.6 % for the transfer model (chance level: 14.3 %). Most errors happen between emotions with similar arousal, for example happiness and anger. The full pipeline, from waveform to spectrogram to both networks, is exported to ONNX and runs entirely in the browser below.

Live demo

Record a sentence spoken with a clear emotion, or upload a WAV file. Inference runs locally in your browser, and no audio is sent anywhere.

Loading the models (45 MB)…
Live spectrogram

Updated while recording (0–8 kHz).

Network input

Waveform: 3 s window at 16 kHz, silence trimmed and centred.

Log-mel spectrogram, 64 mel bands × 301 frames (25 ms windows, 10 ms hop).

Model 1: CNN from scratch
–
Model 2: ResNet-18 transfer
–

Both models were trained on acted German speech from 10 speakers. Expect lower accuracy on other languages, microphones and spontaneous speech.

Method

Data

EmoDB contains 535 utterances in German from 10 actors (5 female, 5 male), labelled with 7 emotions. The classes are imbalanced (46 disgust clips against 127 anger clips), so we use class-weighted losses and report unweighted average recall (UAR), the mean of per-class recalls.

Features

Each clip is resampled to 16 kHz mono, stripped of leading and trailing silence, and centred in a 3-second window. We compute a log-mel spectrogram with 64 bands, 25 ms frames and a 10 ms hop, in dB, standardised per clip. This gives a 64 × 301 image.

Models

  • CNN from scratch. Four blocks of 3×3 convolution, batch normalisation, ReLU and 2×2 max-pooling (16, 32, 64 and 128 channels), followed by global average pooling, dropout and a linear layer. It has 99 k parameters. It is trained for 40 epochs with AdamW and a one-cycle schedule. We augment the data with random time shifts and SpecAugment (one time mask and one frequency mask).
  • Transfer learning. A ResNet-18 pretrained on ImageNet, frozen. The spectrogram is scaled to [0, 1], replicated on three channels and normalised with ImageNet statistics. Only a dropout and linear head (3.6 k parameters) is trained, on the 512-dimensional embeddings.

Speaker-independent evaluation

If clips are split at random, the same speaker appears in both training and test data. The network can then recognise the voice rather than the emotion. We use five folds, and each fold holds out one female and one male speaker, so no speaker ever appears in both splits. The deployed models are then retrained on all ten speakers.

Results

ModelParameters trainedUAR (mean ± std, 5 folds)Accuracy
Chance–14.3 %–
CNN from scratch99 k75.4 ± 6.1 %78.2 %
ResNet-18, frozen + linear head3.6 k61.6 ± 8.5 %64.5 %
Confusion matrix of the CNN trained from scratch

Confusion matrix, CNN from scratch (row-normalised, %).

Confusion matrix of the ResNet-18 transfer model

Confusion matrix, ResNet-18 transfer (row-normalised, %).

  • The small CNN trained from scratch outperforms the frozen ImageNet backbone by about 14 points. Features learned on natural images transfer only partly to spectrograms. A frozen backbone cannot adapt, and precomputed embeddings cannot benefit from data augmentation.
  • Errors follow arousal. 34 % of happiness clips are predicted as anger (both high-energy), and boredom and neutral are confused with each other (both low-energy). Sadness is the easiest class (95 % recall).
  • Per-fold UAR ranges from 64 % to 82 %. With only ten speakers, the choice of held-out speakers has a large effect.

References