Built an A/B analysis layer; 4 product teams now read results without a data scientist in the loop.
See the full storyProof of work
Search results were mediocre and clickthrough had been flat for a year, with nobody owning the number end to end. I took relevance from the ranking model through the evaluation set to the online test, and made every change defensible against a holdout instead of a hunch.
Built an A/B analysis layer; 4 product teams now read results without a data scientist in the loop.
Everyone wanted 'AI in the product' and nobody had said what good would look like, so we were one demo away from shipping something confidently wrong. I set the evaluation first — the metric, the holdout, the bar to clear — so we could tell real progress from a nice-looking demo.
Quantized and batched serving; cost per 1k requests fell 65% at the same quality bar.
The feature pipeline and the training code computed the same features two subtly different ways, and the gap quietly cost us accuracy nobody could explain. I unified them behind one definition, so what the model learned and what it saw in production were finally the same thing.
Scheduled retraining with drift detection and a promotion gate; the model stopped silently going stale between releases.
Cut complexity until what's left is obvious — then ship it.
Off the clock
Practical bits
Unlock with a company account to message directly and reveal:
Jump back to the board with this background already filtered in.
