task 06 / nlp
28-way multi-label emotions, running in your browser.
type a message and see which emotions it carries, out of 27 plus neutral, and which words gave them away. everything runs in your browser, nothing you type is uploaded.
1write something, or pick an example
2pick what to show
reads the text as one message and shows every emotion it picks up, plus which words gave it away.
starting the model (only slow the first time)
score on 5,427 held-out reddit comments, averaged over all 28 emotions so rare ones count as much as common ones.
the small model on this page scores about the same as the big one it learned from, while being 17x smaller and about 45x faster than a single big model, and both clearly beat the original paper.
GoEmotions test set, 28 labels. every model gets its own temperature and per-label thresholds tuned on the validation set. ci = 95% bootstrap interval over test comments. ood = zero-shot accuracy on the dair-ai/emotion twitter test set after mapping both label sets to ekman groups. ece = calibration error before / after temperature scaling.
| model | macro-f1 | ci | micro-f1 | ekman | ood acc | ece | mb |
|---|---|---|---|---|---|---|---|
| tf-idf + logreg | 0.470 | 0.45-0.49 | 0.549 | 0.592 | 0.453 | 0.074 / 0.051 | |
| bigru + glove | 0.482 | 0.46-0.50 | 0.572 | 0.612 | 0.593 | 0.009 / 0.006 | |
| distilroberta (hard labels) | 0.520 | 0.50-0.53 | 0.592 | 0.627 | 0.628 | 0.006 / 0.006 | 313 |
| distilroberta + kd | 0.534 | 0.51-0.55 | 0.611 | 0.646 | 0.632 | 0.030 / 0.010 | 313 |
| distilroberta + kd, int8 onnx | 0.543 | 0.52-0.56 | 0.612 | 0.646 | 0.631 | 0.029 / 0.010 | 79 |
| roberta-large (bce) | 0.542 | 0.52-0.56 | 0.616 | 0.653 | 0.621 | 0.004 / 0.004 | 1356 |
| roberta-large (asl) | 0.547 | 0.52-0.57 | 0.610 | 0.647 | 0.627 | 0.214 / 0.062 | 1356 |
| roberta-large ens. | 0.547 | 0.52-0.56 | 0.616 | 0.659 | 0.624 | 0.028 / 0.026 |
GoEmotions, a set of 58k reddit comments released by google in 2020. each comment was labelled by three or more people with any of 27 emotions, or neutral. a comment can have several at once (about 1 in 6 do). some emotions are very rare: grief has only 77 training examples, neutral has 14,000.
first a big model (roberta-large, 355M weights) was trained on the comments, twice with two different losses. then a small model (distilroberta, 82M weights) was trained to copy the big models' softened answers as well as the human labels. this is called knowledge distillation, and it gave the small model about +0.014 macro-f1 over training it on the labels alone.
the small model was exported to ONNX and stored in 8 bits per weight (79 mb instead of 313 mb), which changes its score by about 0.001. here it runs with transformers.js in a background thread.
a model is rarely confident about rare emotions, so a 50% cutoff would almost never predict grief or relief. the cutoff for each emotion was picked on separate validation comments to get the best f1, after smoothing, because with only 13 grief examples the best value jumps around a lot.
sarcasm and irony often fool it (“great start” reads as positive). rare or subtle emotions like realization, relief and grief are still hard, with f1 between 0.2 and 0.45. it learned from reddit, so very different text (formal writing, other languages) will be read less reliably. on tweets it got the main emotion right 63% of the time.
the full write-up with all experiments is in the report, the code is on github and the model on the hugging face hub.