Built an A/B analysis layer; 11 product teams now read results without a data scientist in the loop.
See the full storyProof of work
The team wanted to ship an LLM feature without lighting money on fire or releasing something that confidently made things up. I designed the retrieval and the evaluation together, so we could tell whether a change actually helped before it ever reached a user.
Built an A/B analysis layer; 11 product teams now read results without a data scientist in the loop.
The models were good in a notebook and useless in production, where nothing was reproducible and every handoff lost something. I built the path from experiment to serving that actually held up — versioned data, a real evaluation gate, and deploys that could be rolled back without ceremony.
Moved training to preemptible hardware with safe checkpointing; compute cost fell 45% at the same wall-clock.
We were flying blind on what the model did in production, with no idea it was wrong until a user complained loudly enough. I added monitoring on the predictions themselves, so drift and degradation showed up on a dashboard instead of in the support queue.
Scheduled retraining with drift detection and a promotion gate; the model stopped silently going stale between releases.
Help teams move fast without setting tomorrow on fire.
In public
Practical bits
Unlock with a company account to message directly and reveal:
Jump back to the board with this background already filtered in.
