mirrons ＊

your body is the brush. a webcam feed is run through pyramidal lucas-kanade optical flow to work out the real velocity field of your movement, not merely that something changed but which way it went, and that field drives a particle system. the pointer feeds the same field at the same time, so mouse and camera both work at once. every stroke piles onto a surface that never clears, which makes this a long exposure of your gestures rather than a live view of them.

two surfaces accumulate at once, one colour and one height. carve reads the height buffer, turns it into surface normals, and lights it, so the same gesture is pigment or relief depending on a checkbox.

why optical flow

frame differencing only tells you a pixel changed. every stroke here needs a direction, so the brush has to know your hand went left rather than merely that your hand was there. per window, assuming brightness constancy and locally constant motion, the spatial and temporal image gradients give a 2x2 system whose solution is the velocity. that matrix is the structure tensor, and it only inverts where the image has gradient in two directions, so a flat wall or a bare edge gives no unique answer. that is the aperture problem, and the smaller eigenvalue is used as a confidence weight so unreliable cells get thrown out instead of believed.

music

the piano is generated in the browser with no samples and no hardcoded scale. it holds one seventh chord at a time as a set of intervals and moves to the next root by an interval drawn mostly from "down a fifth", with the odd tritone substitution. the arpeggio links across the change by picking whichever tone of the new chord sits closest in pitch to the note just played. tempo follows how much the field is moving.

ghosts

record a gesture and it is stored as normalised motion samples, then replayed as its own painter. ghosts save to a file anyone else can load, so a painting can be built by several people who were never in the room at the same time.

built with

• no dependencies. one html file. canvas 2d, getusermedia, web audio and mediarecorder are all browser native. no build step, no framework, no api keys, no server.
• lucas-kanade optical flow (lucas and kanade, 1981) with pyramidal refinement after bouguet.
• eigenvalue confidence test after shi and tomasi, good features to track, 1994.
• blinn-phong shading (blinn, 1977) for the relief lighting.
• 2d simplex noise, perlin 2001, written compactly following gustavson's public domain reference.
• no third party images, fonts, audio or libraries.

privacy

the camera stream is drawn into an offscreen canvas, reduced to 160x120 greyscale, and thrown away every frame. only the single previous frame is kept, because the temporal gradient needs it. nothing is recorded, uploaded or transmitted, and the page makes no network requests at all after it loads. ghost files contain motion vectors only, no imagery.

keys