How often should I retrain my model?

Leland Burns & Jim McGuire
Jan 12
4 min read

It’s a question we get from nearly every client with a credit model in production. Once you’ve launched a model, how often should you revisit it? Is there a fixed schedule you should follow? Or is it only necessary when something breaks?

As with many modeling questions, the answer is: it depends. There are some clear principles we use to guide retraining cadence — and some concrete signs that it’s time to act.

What we mean by retraining

Let’s start by clarifying what we mean. Retraining is different from a full model rebuild or version change.

A rebuild starts from scratch. It might include rethinking your features, running a fresh round of business diagnostics, and testing model lifts across a variety of new designs. It’s typically months of work — the equivalent of a V0 to V1 change.
A retraining, by contrast, is usually a light-touch refit. We take your existing model structure and simply refit it on fresh data that includes newly originated loans and their observed performance. It’s a targeted update, not a full redesign.

Both are useful tools. But retraining is easier to automate, and if you set things up right, it can be an ongoing part of your monitoring process — not a disruptive rebuild.

How often should you retrain?

That depends on a few key factors:

1. Maturity of your business

If you’re a new lender or launching a new product, retraining can pay dividends quickly. Why? Because your population is evolving — fast. You might double your dataset in just a few months. Each new row brings fresh signal, so a retrain could sharpen the model’s performance significantly.

By contrast, if you’re an established lender with a stable dataset, adding a few thousand more records might only represent a 5–10% increase. That might not move the needle — and in those cases, you can afford a slower cadence.

2. Lag in your target variable

Unlike real-time prediction problems, credit models deal with delayed outcomes. You can’t know at origination whether a borrower will repay. So we model outcomes like 60+ day delinquency at 6 or 12 months on books (MOB).

That means any retraining cadence is gated by how long it takes to observe outcomes. You might launch a model today, but you can’t even begin measuring its actual performance for 6–12 months. And even then, you need time to collect enough observations.

This delays how often new data can meaningfully improve a retrain.

3. Ongoing model performance

Once you’re monitoring a live model, you’ll want to pay attention to two key signals:

Rank ordering, often captured by AUC
Score calibration, which measures how predicted risk compares to observed delinquency rates

AUC is a great high-level metric. But it doesn’t tell the whole story. A model can maintain good AUC even as calibration drifts. If score buckets are shifting, or if conversion and loss rates are creeping out of alignment, that’s a red flag.

We also look at Population Stability Index (PSI). Are score distributions moving significantly month-to-month? That could signal drift in your applicant base or the features driving the model. And that might warrant retraining — or even a broader rebuild.

Benchmarking against a shadow model

One of the most powerful tools we’ve deployed for clients is a shadow retrain — a benchmark model that’s updated regularly in the background.

We use this technique to ask: If we retrained today using the latest data, how much better would that model perform? Not just in terms of AUC, but also in:

Stability of feature importance (e.g., SHAP values)
Calibration across key segments
Performance in marginal credit bands

This gives us a point of comparison to the production model. Most months, the answer is “not much changes.” But when performance does diverge — or when the drivers of prediction shift meaningfully — we know it’s time to explore a retrain or possibly a full rebuild.

Make it repeatable

For this kind of monitoring to work, you need technical infrastructure that supports it. We’ve helped clients set up simple pipelines with:

Automated data refreshes into a modeling table (e.g., Snowflake)
Reusable model code that can rerun on new data with minimal tweaks
Dashboards or summary outputs to track changes in performance over time

The goal isn’t to fully automate deployment — just to streamline the retrain so your analysts can monitor it efficiently and decide when a change is warranted.

Think of it as building a retraining engine, not just a one-off script.

Final thoughts

There’s no one-size-fits-all answer to how often you should retrain. But here’s the framework we follow:

New products? Retrain frequently. Your data is evolving, and each retrain is a chance to learn and improve.
Established products? Let the data and monitoring guide you. Retrain when performance shifts or a benchmark model shows meaningful lift.
Not sure where to start? Build a shadow retrain into your workflow. Monitor model performance and retraining potential side by side. Let evidence, not guesswork, drive your decisions.

If you're running a credit model and aren't sure when or how to revisit it, we can help. At Ensemblex, we’ve designed retraining playbooks for fintech startups, credit-card issuers, and legacy lenders alike.

The key isn’t retraining more — it’s retraining smarter.