Moved training to preemptible hardware with safe checkpointing; compute cost fell 60% at the same wall-clock.
See the full storyProof of work
The models were good in a notebook and useless in production, where nothing was reproducible and every handoff lost something. I built the path from experiment to serving that actually held up — versioned data, a real evaluation gate, and deploys that could be rolled back without ceremony.
Moved training to preemptible hardware with safe checkpointing; compute cost fell 60% at the same wall-clock.
Search results were mediocre and clickthrough had been flat for a year, with nobody owning the number end to end. I took relevance from the ranking model through the evaluation set to the online test, and made every change defensible against a holdout instead of a hunch.
Scheduled retraining with drift detection and a promotion gate; the model stopped silently going stale between releases.
The feature pipeline and the training code computed the same features two subtly different ways, and the gap quietly cost us accuracy nobody could explain. I unified them behind one definition, so what the model learned and what it saw in production were finally the same thing.
Added a learned reranker behind a flag; clickthrough lifted from 40% to 71% in the holdout.
We were flying blind on what the model did in production, with no idea it was wrong until a user complained loudly enough. I added monitoring on the predictions themselves, so drift and degradation showed up on a dashboard instead of in the support queue.
Added dbt tests and freshness SLAs; data-quality incidents fell 65% in a quarter.
Bring craft and measurement to work that usually gets neither.
Off the clock
In public
Practical bits
Unlock with a company account to message directly and reveal:
Jump back to the board with this background already filtered in.
