From Max to Centroid, From Gut Feel to Data: The Last Piece of the Flatmate Bug

The post about my flatmate being recognized as me ended with a hole: my supervisor said the similarity threshold shouldn’t be hand-picked, it should be computed from data. The sharpness post filled the small hole — blurry samples can’t get in the door anymore. This one fills the big one: how the matching logic itself changed, and how the threshold now calibrates itself.

Only with this post is the bug actually, fully fixed.

Step 0: The Foundation — the Geometry of Embedding Space

What a model like SFace does is turn any aligned face image into a 128-dimensional vector, trained so that different photos of the same person land in one tight cluster and different people land far apart. After normalization, every vector lives on a 128-dimensional unit sphere, and “similarity” is just the angle between two vectors — cosine similarity, the dot product of unit vectors.

Everything that follows rests on this. My own measurements confirm the geometry directly: my samples score 0.84–0.91 against each other (same cluster, small angles), while my test subject and I score roughly 0 — nearly orthogonal. In high dimensions, “orthogonal” is “unrelated”: two random high-dimensional vectors are almost inevitably close to perpendicular, a phenomenon called concentration of measure. The model’s way of saying “these two faces have nothing to do with each other” is to make their vectors perpendicular.

Step 1: Deriving the Centroid from the Incident

The original bug’s mechanism was “one blurry enrollment sample + best score per sample.” The key move is translating that into the language of statistics:

Taking the max over samples means any single sample can trigger a match on its own. Max is the statistic most sensitive to outliers — nine good samples out of ten cannot save you from the one bad one.

Which raises the natural question: is there a statistic where the good samples can outvote the bad one? The answer almost says itself: the mean. Average ten samples and a single bad one’s influence gets diluted to a tenth.

But “the mean” has a genuinely elegant justification on the sphere, beyond intuition. Ask: which point on the sphere has the highest average cosine similarity to all of my enrollment samples? The dot product is linear over sums, so “maximize the average similarity” is equivalent to “find the unit vector with the smallest angle to the sample mean” — and the answer is the mean’s own direction, i.e. the normalized mean. In code it’s two lines:

centroid = feats.mean(axis=0)
centroid /= np.linalg.norm(centroid)

So the centroid isn’t an engineering shortcut. It is the exact mathematical solution to “the point that looks most like all of your enrollment samples taken together.”

Two implementation details now derive themselves:

Why normalize each sample before averaging: without it, samples with larger norms carry more weight in the sum. I want one vote per sample, not “the loudest voice counts double.”
Why normalize again after averaging: the mean of unit vectors always lands inside the sphere (norm below 1). Pulling it back onto the sphere is what keeps later dot products being genuine cosine similarities.

This whole construction has long had names in the literature: the Nearest Class Mean classifier in classic machine learning, Prototypical Networks in few-shot learning (Snell et al., 2017), template averaging in face recognition evaluation. My “thinking of it” was really just recognizing that my bug and these solved problems are the same problem.

Step 2: From “What Should the Threshold Be” to Calibration

The fixed threshold of 0.363 is OpenCV’s default, tuned on some public dataset — what makes it right for the two particular people in my living room?

This brings in the classic framework of biometrics: any verification system only ever performs two kinds of comparisons — genuine (a person against their own gallery) and impostor (someone else against your gallery) — and each produces a score distribution. The threshold is a line you draw between the two: move it left and impostors slip through more easily (FAR rises); move it right and the real person gets rejected more often (FRR rises). A threshold has no “correct value,” only a trade-off you’re willing to accept.

What makes my scenario special: a misrecognition (calling my flatmate by my name) is far more embarrassing than a failure to recognize. So I’d rather eat a higher FRR to keep FAR low — the threshold belongs beyond the right tail of the impostor distribution.

No large-scale impostor dataset? The enrollment gallery is its own small impostor dataset: scoring each of A’s samples against B’s centroid is a genuine impostor comparison. Take the highest score across all cross-person comparisons (the empirical impostor ceiling), add a 0.05 margin, and that is the entirety of _calibrate_threshold. Calibration only ever raises the threshold, never lowers it — a ceiling estimated from a two-person gallery may well be too low, so the default acts as a floor.

Measured in practice: impostor ceiling -0.017, genuine floor 0.78, a gap of 0.8 between them. The default 0.363 stands safely inside that gap, so calibration kept it. But its meaning changed: it used to be safe by luck; now it’s safe by measurement.

This move — draw both distributions, look at the gap — is exactly the standard posture in NIST face recognition evaluations and the ISO/IEC 19795 standard. I just ran the same ritual at the scale of two people and a living room.

Step 3: “Deriving It” Is Really Three Learnable Skills

Replaying the whole chain, no step ran on inspiration. Every one of them is trainable:

Translate the bug into statistics. “My flatmate got misrecognized” becomes “max is outlier-sensitive.” This step is worth the most and is the most practicable — once translated, the solution mostly surfaces by itself.
Recognize isomorphic problems. “Many samples representing one class” = prototype; “picking a threshold” = the genuine/impostor trade-off. The point of reading the literature isn’t memorizing solutions — it’s building the dictionary of problems, so the next incident can be looked up.
Validate on your own data. The first two steps only produce candidates. The similarity table — the 0.84–0.91 cluster, the near-zero orthogonality, the 0.8 gap — is the evidence that any of it holds in my setting.

Three Questions I’m Leaving Myself

Just as I finished writing, three follow-up questions arrived. Each came with a hint, but I haven’t thought them through to where I could write them up. Recording them here:

One: why the mean and not the median? Isn’t the median more robust to outliers? Hint: think about whether the vector you get by taking the median per dimension in 128-D still points at the cluster center. My current intuition: the median can only be taken coordinate by coordinate, and a vector assembled from 128 independently chosen coordinates doesn’t necessarily point anywhere near the cluster center — it has no optimality proof on the sphere the way the mean does, and the answer changes if you rotate the coordinate system. Besides, the outlier-handling job already belongs to the sharpness bouncer at the door. But this is intuition I haven’t actually computed.

Two: if enrollment goes from 10 samples to 100, does the centroid get better? When does it get worse? Hint: averaging suppresses random noise, not systematic bias. A hundred photos all lit from the same side — the average removes the jitter but not the lighting itself, which gets baked into the centroid untouched. If that’s right, what samples need is diversity, not quantity.

Three: when does the threshold calibration fail? Hint: with only two people in the gallery, whose ceiling is the “impostor ceiling” actually estimating? Only ours. A third person who happens to look like me could score far above it. Which is presumably the whole reason “only raise, never lower, default as floor” exists — and it means calibration has to re-run every time a new person enrolls.

Whichever one I figure out, I’ll come back and fill it in.

Afterword

The flatmate-bug series ends here: a quality gate at the door, a centroid for matching, a threshold grown from data.

One misrecognition, and what got fixed in the end wasn’t really three pieces of code — it was three exchanges of intuition for something that can be tested.