Emote Detector

Computer vision, in the browser

Give your camera a thumbs-up, flex beside your head, or yawn, and the matching Clash Royale emote pops up with its sound. MediaPipe's face, hand and pose landmarkers run on your device; no video ever leaves it.

Press “Start camera” (about 40 MB of models load once) or “Play demo”.

  • Thumbs Up
  • Goblin Muscle
  • Princess Yawn

Your camera shows here. Nothing is recorded or uploaded.

The three gestures

Hold a thumbs-up or a flex for a moment (0.15 s) and a yawn a little longer (0.4 s). When a gesture is almost there, the line under its meter says what to change. The same emote waits two seconds before playing again; a different one can follow after 0.7 s.

How it works

  1. Each video frame goes through three MediaPipe models on your device: 478 face points, 21 points per hand, 33 body points.
  2. Three hand-written rules turn those points into cues, each scored 0 to 1, and a gesture's score is its weakest cue. Flex: elbow bend, fist above the shoulder, fist beside (not in front of) the head, elbow at shoulder height, fist clear of the face. Thumbs-up: four fingers folded, thumb well above the wrist, thumb pointing up, thumb above the fist and away from the face. Yawn: mouth tall relative to its width, eyes closed, brows relaxed (a scream knits them).

Threshold

A score above 0.5 is a candidate. Everything is timed in milliseconds rather than in frames, so a phone at 8 frames per second behaves like a laptop at 25.

Hold

A score over the threshold has to hold for 0.15 s — 0.2 s for a thumbs-up, 0.4 s for a yawn — to switch the gesture on. It switches off only after the score has been low for half a second longer than it was high, so a gesture held for ten seconds fires once and a jittery frame changes nothing.

Arbitration

Scores are averaged over about 0.2 s before the gestures are compared, so the body model's frame-to-frame jitter cannot flip a thumbs-up beside the head into a flex. A clear flex keeps its fist even with the thumb up — its score must be at least 0.9 × the thumbs-up's — otherwise the thumbs-up wins. And because the body model takes a few hundred milliseconds to see where an arm went while the hand model is instant, a thumbs-up waits, at most 0.5 s, while the body model is still catching up with the hands.

Cooldown

The strongest active gesture fires its emote. The same emote again waits two seconds, a different one 0.7 s, and a gesture that arrives during that wait plays as soon as it may, if it is still held.

Threshold
score > 0.5
Hold
0.15 s · thumbs-up 0.2 s · yawn 0.4 s
Release
low for 0.5 s longer than it was high
Smoothing
averaged over ~0.2 s
Flex vs thumbs-up
flex wins at ≥ 0.9 × · thumbs-up waits ≤ 0.5 s
Repeat
same emote 2 s · a different one 0.7 s

The rules started as a line-for-line port of the original Python app and were rewritten against a corpus of 95 real photos and 70 ground-truth clips (source, numbers and details). There is no trained classifier in this page: the original's six-class MobileNetV2 reached 68 % validation accuracy, so it stayed out.