How Claude Creates and Synchronizes 3D Objects with its own Composed Music

Shelby Thomas — Mon, 28 Sep 2026 00:00:00 GMT

[RubatoGlow — an interactive figure, on the page]

[RubatoFilm — an interactive figure, on the page]

"Can You?"

I had Opus 5.5 one-shot a prompt "make a piece of music that feels like a machine learning to play music and video to go along with it".

I provided a few tools. LilyPond, which turns plain text into sheet music and MIDI, Blender 5.2, and Runpod with a $20 GPU budget which it used to render the 3D piano. It additionally used FluidSynth to play the notes with a piano sound, and Python code to draw the 2D video without any guidance.

This post covers what Claude did to keep all three in sync. Hint: code, which continues to prove itself as a substrate for solving problems across more domains than people once imagined.

Keeping Three Mediums in Sync

The most interesting part for me was syncing music, video, and render and I was particularly impressed with keeping the 3D piano exactly in sync with the real notes. Blender renders images one frame at a time, with no sound to follow, so every key has to go down on exactly the right frame from the notes alone.

Here's an example of the main theme, with a piano below it to show that the video matches the actual notes. Press play and the keyboard lights up with the clip, then press your turn and play it yourself and prove to yourself it is playing the actual notes.

[RubatoTheme — an interactive figure, on the page]

JSON is King

The approach Claude took was to first create the score, then turn the score into one list of notes and built the 2D video and 3d render from it. Lets take a look below, here's the theme's first note, a B, from the score to the final video:

[RubatoFlow — an interactive figure, on the page]

A Note Is a Run of Frames

Every note goes through the same steps. The 3D animation shows 24 still pictures every second, and each picture is a frame. A note covers every frame while its key is down, so the first B covers 24 frames.

Here are all 136 frames of the close-up shot, stacked in 3D. Drag to turn the stack, and move the slider or press Play to go through the frames:

[RubatoFrames — an interactive figure, on the page]

The Loop That Presses the Keys

This loop moves the 3D piano's keys in Blender. For every frame, it goes through the list and presses the keys whose notes are sounding. The drawn video runs the same loop at 30 frames a second, and the synthesizer plays the same list, so all three line up. Step through it, and the stack above follows along:

[RubatoStepper — an interactive figure, on the page]

The Final Product

The final product is the film at the top of this post. Here's the code building its 3D piano, from a wireframe to the finished look:

[Video — an interactive figure, on the page]

Findings

A shared source of truth did the syncing. Everything goes through one list of notes, saved as JSON for Blender and the video and as MIDI for the synthesizer. Nothing lines the three up after the fact.
The model used existing tools exceeding well. It connected LilyPond, FluidSynth and Blender with code instead of building anything new.
The visuals took the most iteration. The music only needed small tweaks at the beginning and the end, but the video took several rounds of notes on camera angles, what to show, and what to cut.

Full score here.