Wind-Controlled Foliage with a Hand-Worn IMU

I wanted to control something physical from the computer — or really, something on the computer from something physical. No Unreal Engine experience, no soldering experience, nothing. So I handed the whole problem to Claude and said treat me like a beginner. It asked questions, settled on the idea — wind moving real foliage in a 3D scene, driven by my own hand — and came back with a shopping list: an ESP32, a GY-521 breakout (the MPU-6050 IMU), some jumper cable, a USB cable. It walked me through soldering the board, flashing it, and wiring it up, and when something didn’t read right I’d send a screenshot and it would diagnose from there.

The pipeline

The ESP32 streams pitch/roll/yaw off the MPU-6050 over USB serial at 115200 baud, 50Hz, as CSV. A small Python script (serial_to_osc.py) reads that stream and re-broadcasts it as OSC to 127.0.0.1:9000. On the Unreal Engine 5 side, an actor blueprint (BP_PlantController) runs its own OSC server — the one snag that took a few screenshots to debug was that it has to bind 0.0.0.0, not 127.0.0.1, or it never receives anything. That OSC data feeds a Material Parameter Collection (MPC_Wind, exposing WindStrength/WindSpeed) consumed by a custom material — M_WindFoliage — that displaces vertices with a sine-based World Position Offset. The scene runs on Nanite foliage and Lumen GI, using Megascans’ Megaplant library for the bush itself, so the wind reads as physically real rather than a shader trick layered on top.

A web build with no hardware required

I also rebuilt the whole thing from scratch as a single self-contained HTML file using Three.js r128 — no imported 3D model, no asset pipeline. Geometry, bark and needle textures, even grass, are all generated procedurally at runtime, alongside hand-rolled SSAO, TAA, god rays, and depth-of-field. It supports two input modes: WebSerial, which reads the exact same ESP32/MPU-6050 CSV stream directly in Chrome or Edge, and a planned DeviceMotion fallback that uses a phone’s tilt sensors instead — so eventually no hardware is required at all, just a browser tab.

The actual point

None of these were tools I knew three weeks ago — not Unreal, not soldering, not OSC. The plan isn’t really the deliverable; the bigger toolbox is. This same pipeline doesn’t have to end at a bush in Unreal — it’s a generic “physical input → real-time parameter” bridge that could just as easily drive a Three.js scene on the web, an in-person installation, or anything else that takes a continuous signal. Building this took three evenings spread over two weeks, only one of which was actually soldering and wiring. The point was never the bush — it was proving that not knowing a tool yet isn’t a real constraint anymore.