We were flying blind on what the model did in production, with no idea it was wrong until a user complained loudly enough. I added monitoring on the predictions themselves, so drift and degradation showed up on a dashboard instead of in the support queue.

Quantized and batched serving; cost per 1k requests fell 30% at the same quality bar.

Retraining was a manual ritual only one person knew, so the model quietly went stale between the times they had a free afternoon for it. I automated the loop — scheduled retraining, drift detection, and a gate that only promoted a model that actually beat the one already live.

Added dbt tests and freshness SLAs; data-quality incidents fell 25% in a quarter.

The feature pipeline and the training code computed the same features two subtly different ways, and the gap quietly cost us accuracy nobody could explain. I unified them behind one definition, so what the model learned and what it saw in production were finally the same thing.

Reworked features and validation; offline accuracy rose from 13% to 33% and held online.

Labeling was slow, inconsistent, and the bottleneck on every model we wanted to ship. I built the tooling and the guidelines that made it fast and repeatable, and put the annotator disagreements to work as a signal instead of noise.

Unified training and serving features behind one definition; the accuracy gap nobody could explain simply disappeared.