Blueprinter: pulling an object apart from one photo

A quick note before anything else. I'm going to talk about what this does and the reasoning behind a few of the design choices, but not how to build it. The build details are the least interesting part anyway, and this is the kind of capability I'd sooner write about than hand over fully assembled. What I care about is the capability itself and where it's heading.

What it does

You give it one photo of an object. It gives you back the same object taken apart, with the pieces floating separately so you can drag them around, and each piece labelled with a guess at what it is, what it's probably made of, what it does, and how sure it is. It also tells you what it couldn't figure out, which turns out to matter a lot more than I expected when I started.

There's no CAD file involved. Nobody takes a screwdriver to anything. It's a photo going in and an annotated teardown coming out.

I've been calling it Blueprinter. The longer I used it the more it stopped feeling like a demo and started feeling like an early look at something that's going to be genuinely useful in a few years, in hardware work, in intelligence, in science, in teaching. And the reason I think that has almost nothing to do with how clever the code is, which is the part I want to get to.

Why I'm putting this on a security blog

Working out what a device is and how it's put together from the outside is slow and expensive. It's usually a bench, some tools, a person who knows what they're looking at, and a few hours. Counterfeit checks start there. So does supply-chain verification, and teardown work, and figuring out what a piece of captured hardware actually is. All of it begins with taking the thing apart in your head and naming the pieces.

The bet Blueprinter makes is that a decent chunk of that first pass can happen from a photo, in a few seconds, before anyone touches the object. It's not a replacement for the bench. You can't read a tolerance or a die marking off a JPEG and you shouldn't pretend you can. But getting to the bench already holding a rough hypothesis is worth a lot, and that kind of triage-first-then-verify workflow is something security people already do everywhere else.

There's also one property here that most AI demos don't have, which is that the hard part keeps improving without me doing anything. More on that at the end.

How it works, roughly

The system goes back and forth between two very different kinds of work. Some steps need a model to imagine something that isn't in the original photo. Other steps need no intelligence at all and are just arithmetic on a clean input. The whole design is about ordering those steps so each one leaves an easier problem for the next.

Four moves, more or less:

First, it takes the one photo and imagines the object with its parts spread out in space, the way a repair manual draws an exploded view, except it's inventing that view rather than copying one.

Second, it takes that spread-out picture and cuts it into separate layers, one per part. This step is deliberately stupid. No model, just geometry. It only works because the previous step was told to produce something a stupid algorithm could handle, and I think that's the quiet lesson worth taking away: sometimes the smartest move for a generative model is to hand off to a boring algorithm that can't hallucinate.

Third, now that the parts are separated, something finally tries to name them. Each isolated part gets looked at on its own, and gets a role, a likely material, a confidence number, and a note about what can't be told from a photograph.

Fourth, it reconciles everything. Duplicates get merged, contradictions across the whole object get resolved, and it produces one report instead of a pile of per-part guesses.

The decision I keep coming back to is the wall in the middle of that.

The model that draws is not allowed to label

The obvious thing is to let one model both draw the exploded view and tell you what the parts are. I'd avoid that. The model making pixels is being graded on whether the result looks convincing. The model making claims has to be graded on whether the claim is true. Those are different jobs and they fail in different ways, and if you let one model do both, a good-looking render can hand you a confident false fact and you'll have no way to tell.

So there's a hard line. The image side produces pixels and stops there. A separate language side reads those pixels and gets leaned on hard to stay honest. Part of staying honest is that it sees each isolated part next to the whole object, so a lonely "metal cylinder" gets read correctly as a can once it's in context. The other part is that it has to write down what it doesn't know, rather than filling the gap with a plausible-sounding alloy spec it made up.

A picture that invents a part is a bug you'll notice. A caption that invents one is a bug that ships in your report.

The mistake that taught me the most

Early on I let the labelling side do some research first, look the object up, and then use what it learned to guide the decomposition. Seems obviously good. Tell the system what it's looking at and it should do better work.

It got worse, and the way it got worse is the useful bit.

Research comes back with a name. Names are compact and often singular. So it'd come back with "a six-pack of cans," and when I wired that in as a target, telling the system to output exactly those parts, it would dutifully squash seven clearly separate cans down into one part, because the noun it had been handed was singular. The outside knowledge, pointed the wrong way, overrode what the model could see perfectly well with its own eyes.

The fix wasn't to drop the research. It was to flip which direction it pushed. Instead of a target it became a floor: expect at least these things, and when you're unsure, split more rather than less. Same information, opposite wiring. As a ceiling it destroys detail. As a floor it can only ever add.

I think that goes well beyond this project. Grounding a model in facts is genuinely useful and genuinely risky at the same time, and the risk isn't the facts, it's whether you've aimed them at a minimum or a maximum. Knowing what something is should never limit how closely you're allowed to look at it.

The part I actually care about

Here's the argument that made me want to write any of this down.

Almost none of the hard capability in Blueprinter is mine. The imagination, the ability to correctly picture an object pulled apart with the parts shaped right and complete and sitting in the right relationships, lives in the underlying image models. What I built is the scaffolding around them: the separation trick, the pressure to stay honest, the floor-not-ceiling grounding. But the raw ability to picture a teardown is borrowed, and it's borrowed from a technology that an enormous number of well-funded people are racing to improve for reasons that have nothing to do with me.

So how good this whole thing is comes down, mostly, to how good image models are, and that number is climbing fast. Every new generation of image models raises the ceiling here for free. I don't have to touch the code. The exploded views get more accurate, the parts get more complete, and the fine internal structure that today's models smear or skip starts to actually resolve.

And decomposition has thresholds. Each level of fidelity opens up a heavier kind of use than the last.

Right now we're in the range where it's great for education and for "what's inside this thing" explainers, the kind of use that forgives the occasional wrong guess. A bit better and it's good enough to triage hardware and draft a rough parts list before you pick up a tool. Better still and you can start using it where being wrong actually costs something, forensic and scientific work, which really depends on the "I don't know" being trustworthy. At the top of the range you're estimating how a device is built, and maybe where it came from, from imagery alone. That's the band with the sharpest dual-use edge and the one I think about most.

We're somewhere in the lower-middle of that today. The strange thing about sitting on someone else's fast-improving technology is that you don't have to climb. You just get carried up.