Smile Rain Fireworks · Independent work · Browser AR real-time interaction
Real-time expression recognition and interaction
Expression recognition turned into responsive, expression-driven interactions: a smile makes it rain, a laugh sets off fireworks, and sparks bounce off your head. Opens in a phone browser.
- Independent work · Browser AR real-time interaction
- Expression-driven · Runs in the browser
- Open source · Live demo
01
Goal
Most interactions ask you to do something first: tap, swipe, hold. This piece tries something lighter: once the experience starts, your expression drives the picture. Smile and it rains. Laugh and fireworks rise. When the sparks fall on your head, they really bounce off. The question was how to turn an expression into feedback you can see, and that makes sense.
02
My role
Interaction rules and experience definition, the look and feel of the feedback, edge-state design, and round after round of review and tuning until it opens straight from a phone browser.
03
Key decisions
A number is not a smile. The model only reports how far the mouth corners lift and how wide the mouth opens; what counts as a smile is a design decision. Expressions became three states, idle, smiling and laughing, with entry thresholds higher than exit thresholds and a short hold before a change counts. Continuous judgement and separate entry and exit thresholds reduce the false triggers and flicker caused by talking or borderline expressions. Two progress bars show how close you are. Feedback has to make sense. An early version fired fireworks out of the mouth, which reads backwards. Now a shell rises from the bottom of the frame, bursts at the top and falls back onto the person; the ascent is the ceremony. Rain does not collide, so there is no hard-edged hole in front of the face. A head is more than a face. The model's landmarks stop at the forehead, so sparks would pass through hair. A stable ellipse fitted to the full outline and extended upward, with smoothing that stays steady at rest and keeps up when you shake your head, makes the collision believable.
04
Real conditions and visible result
No camera: a manual mode. Permission denied: per-browser instructions. Model loading: visible progress. Nobody there for a while: it stops on its own. The piece is open source and can be tried online. Relevant commissions: on-site interaction for exhibitions and events, camera interaction on brand pages, and proofs of concept that need to run before they can be judged.
Want something similar?
Send the goal, the expected deliverable and references, and I will reply with a scope note.
