Phone as 3D Controller

What happens when your phone stops being a screen and becomes a controller? That’s the question driving this experiment. Scan a QR code on the desktop display, and your phone connects peer-to-peer — from that point on, tilting and rotating your phone moves a layered 3D object in real time on screen. No app install, no account, just a QR code and a WebRTC handshake.

How it works

The desktop view shows a large typographic mark — the “Instrument” logo — built entirely from SVG. Rather than converting the SVG into a true 3D mesh, the depth effect is simulated by stacking 10 copies of the same SVG, offset slightly, then driving shadow and light positions based on the phone’s live orientation data. The result reads as a convincingly sculptural object that responds to physical movement. Connection is handled via PeerJS and WebRTC, with short-lived Cloudflare TURN credentials generated server-side through a Netlify Function — the API token never touches the browser.

The build

The trickiest part wasn’t the physics — it was the connection infrastructure. TURN servers are required for WebRTC to work reliably across different networks, but TURN credentials need to stay secret. A lightweight serverless function handles credential generation at connection time, issuing two-hour tokens that expire automatically. The phone UI itself is minimal: an enable button, a replay option, and then pure gesture input from the Device Orientation API.

Why it matters

This is part of an ongoing investigation into what inputs are actually available to us on the web — and how far we can push them without native apps or custom hardware. The phone-as-controller pattern has real legs beyond experiments like this. Interactive retail — point at a product and rotate it in 3D. Spatial presentations — a presenter moves through slides by tilting. Multiplayer experiences — multiple phones each controlling a different axis. The web already has the primitives. Most products just haven’t used them yet.