The Gallery Had Exactly One Person in It. It Recognized My Flatmate as Me Anyway.
I was the only person enrolled. The system looked at my flatmate and said it was me — with decent confidence. Digging back: three individually fine decisions stacked up. Blurry enrollment photos, max-score matching, and a sticky identity that locked in a single bad frame.
The Gallery Had Exactly One Person in It. It Recognized My Flatmate as Me Anyway.
I’ve been adding face recognition to a desktop assistant. Everything runs locally, nothing touches the cloud, and the gallery only holds photos from people who actually agreed to be enrolled.
The night I got the pipeline working end to end, I was sitting in front of the webcam tweaking things when my flatmate came home and walked past behind me.
His face got a box around it. The label said my name.
The gallery had one person in it. The entire set of humans this system knows is: me. It looked at my flatmate and said, that’s me.
With decent confidence, too.
How It Works
The chain itself isn’t complicated.
MediaPipe watches the webcam and shouts when there’s a face. This layer is deliberately forgiving — blinks and off-angle poses don’t bother it, because its only job is “someone is here,” not “who.” The face then gets cropped and handed to OpenCV: YuNet aligns it, SFace pulls out a 128-dimensional embedding, and that gets compared against the gallery with cosine similarity.
On top of that I added something of my own: sticky identity. Confidence naturally dips the moment you blink or turn your head, and if the system re-decides who you are on every frame, the name flickers. So once it knows you, it keeps you — as long as any face stays in frame, you’re still you. Only after you’ve been gone about five seconds does it forget. Glancing down at your phone doesn’t reset it. Actually leaving does.
At the time I thought this was a rather elegant design. Remember it. It’s about to take most of the blame.
Digging Backwards
Honestly, my first thought was that maybe we just look alike.
We don’t.
After digging a while, my best guess was three things landing at once.
One: I did zero quality checking at enrollment. A few of the photos were taken while I was moving. Blurry. And what does the embedding of a blurry photo look like? The average of a smudged face — all the distinguishing detail wiped out. A vector like that is a little bit similar to everyone.
Two: the matcher takes the best score across all samples. The current frame gets compared against every photo in the gallery, and whoever scores highest wins. Which means one bad sample quietly ruins the whole gallery — my flatmate scored terribly against my sharp photos, but that didn’t matter. He only had to line up with the blurry one.
Three: the sticky identity I was so pleased with. The first two things combined only produce a bad call on some individual frame. The next frame would probably have corrected it. But the sticky logic has no idea a frame is wrong. All it knows is “recognized,” and then it does its job faithfully: face still in frame, keep the identity.
One frame’s mistake, locked in as a permanent one.
The Fix
Two changes.
Enrollment now rejects blurry photos outright. Cut off at the source — those match-anyone vectors never get created in the first place.
And identity is no longer decided by a single frame. The system has to recognize the same person over several consecutive frames before the sticky logic commits. A one-frame fluke doesn’t get promoted anymore.
I did briefly consider popping up a “please hold still” prompt during enrollment. Then decided just rejecting the blurry ones was better — the user doesn’t have to cooperate with anything. The system gets stricter on its own.
My Supervisor’s Follow-Up
I told my supervisor about all this. He thought the fix was right — better than a prompt, since nobody has to hold still — and then casually pushed the problem one level deeper:
there’s a lot to think about in the baseline reference image you compare against in the first place. And going a step further, there should be a minimum acceptable similarity threshold — not hand-picked, but calibrated from the normal variation between people, as the gallery actually grows.
That one got me. I’d been treating the threshold as a tuning knob — turn until it feels right. But it’s a statistics question: the threshold’s whole job is to separate “how different two people are” from “how much one person varies,” so it should come out of the data, not out of my wrist.
And when the gallery holds exactly one person, the data for “how different two people are” doesn’t exist. The system had never seen a second human. It didn’t know what not similar even was. So this bug picking the moment when only I was enrolled to blow up — that wasn’t a coincidence.
Afterword
If there’s a lesson, it’s roughly this: every layer being fine on its own doesn’t mean the stack is fine. A forgiving detector is correct. Max-score matching sounds harmless. Sticky identity genuinely makes the experience smoother. Three “sounds fine”s, stacked into one remarkably confident wrong answer.
And one more: a test environment with a single user can’t test anything about telling people apart — which is the entire job of a recognition system. That ability had simply never been exercised until a second person walked into frame.
The threshold auto-calibration is still in progress. I’ll write that one up when it’s done.