Search results were mediocre and clickthrough had been flat for a year, with nobody owning the number end to end. I took relevance from the ranking model through the evaluation set to the online test, and made every change defensible against a holdout instead of a hunch.

Instrumented live predictions for drift and degradation on 8 models; problems surfaced on a dashboard, not in support.

We were flying blind on what the model did in production, with no idea it was wrong until a user complained loudly enough. I added monitoring on the predictions themselves, so drift and degradation showed up on a dashboard instead of in the support queue.

Added dbt tests and freshness SLAs; data-quality incidents fell 60% in a quarter.

Labeling was slow, inconsistent, and the bottleneck on every model we wanted to ship. I built the tooling and the guidelines that made it fast and repeatable, and put the annotator disagreements to work as a signal instead of noise.

Moved training to preemptible hardware with safe checkpointing; compute cost fell 15% at the same wall-clock.

Everyone wanted 'AI in the product' and nobody had said what good would look like, so we were one demo away from shipping something confidently wrong. I set the evaluation first — the metric, the holdout, the bar to clear — so we could tell real progress from a nice-looking demo.

Built an A/B analysis layer; 12 product teams now read results without a data scientist in the loop.