What Is a Good AUC for a Credit Model?

If you’ve built credit underwriting models, you're familiar with the modeling metric called “AUC.” AUC is the primary metric used to measure the power and effectiveness of models.

We’re often asked questions like: “What’s a good AUC for our model?” or “What AUCs can your models achieve?” And the answer is... it depends. What makes a "good" AUC is nuanced and context is key.

The Basics: What is AUC?

AUC, short for "Area Under the Receiver Operator Curve," measures how well a model separates positive from negative outcomes across all possible decision thresholds. In credit models, that's the probability that the model will rank a randomly chosen bad loan as riskier than a randomly chosen good one.

For example: Say I have a credit model that predicts whether a loan will default and has a 0.75 AUC. To test my model, I take all the loans in my dataset and sort them into two buckets, one for “bad” loans that defaulted and one for “good” loans that didn’t. Then I pick one random good loan and one random bad loan and check how my model scored them. AUC is the probability that my model will have sorted those two loans into the right buckets. Since my model has a 0.75 AUC, there is a 75% likelihood that my model correctly assigned a higher probability of default to the bad loan.

AUC values range between 0.5 and 1, with 0.5 being completely random and 1 being a “perfect” model.

What Factors Determine a “Good” AUC?

1. Volume and quality of data

Models are only as good as the data you train them on. Higher quality data and more of it will yield better models. All of the following impact how good your model can be:

- How many total rows do I have to train on?

- How many “bad” loans are available to train on?

- What duration of loan performance do I have available to train on?

- How many columns do I have available to use as independent variables?

- Are those columns from a range of sources, with reliable data?

If you are building a model for a young business in an emerging market with limited performance data and predictive data sources, a model with an AUC of 0.63 might still be a huge value add! Conversely, a mature lender with years of performance data, hundreds of thousands of records, and access to lots of reliable data sources will shoot for an AUC well above 0.7.

2. Variance in the model population

Credit models rank-order the likelihood of default across a population. The further that population stretches across the credit spectrum, the easier ranking becomes. Imagine a model ranking risk across a broad group of consumers ranging from subprime to super-prime. Two randomly chosen people from this population likely have quite different credit performance. In this case, even an average model might have an AUC above 0.8.

On the other hand, if I’m predicting risk for only subprime consumers in a small group of states applying to a specific network of auto dealers, it is harder to distinguish these applicants. A model with an AUC of 0.65 could be quite useful here.

3. The model's purpose

Say your priority is weeding out toxic risk in the worst 10% of the population. How well your model distinguishes between the best and second-best borrower isn't as much of a concern. AUC describes the model’s performance across the full population, while your focus is on a specific segment.  In this case, AUC isn’t wrong, and AUC improvement may still be helpful. But how “good” your AUC is depends on how you want to use the model.

What’s the Right Way To Use AUC?

1. Remember it's all relative

While there is no “good” AUC threshold in absolute terms, comparing a new model's AUC to an incumbent or industry standard generic model is a valuable assessment. Make sure you compare apples to apples: use a consistent data sample!

2. Use it to iterate

AUC is especially valuable for honing your model—weighing what data to include, how much to reduce the feature space, or how the model interacts with different segments.

3. Tie it to business impact

Ultimately, you're not seeking the highest AUC. You're trying to reduce losses, increase approval rates, and improve conversion—so use projections of those metrics alongside AUC when choosing your model.

Final Thoughts

Never measure a model solely by its AUC. Ensemblex's expert AI credit modelers can help contextualize your model's AUC for a more impactful, accurate assessment of its performance.