What Should I Actually Expect from a Model Build?

We've written a lot about specific aspects of credit modeling — how to choose features, what AUC actually measures, when to retrain. But we haven't spent much time on the bigger picture: if you've never done a custom model build before and you're considering one, what does the process actually look like? How long does it take? And where does it tend to go sideways?

This post is for companies that have relied on off-the-shelf scores or externally developed policies and are now thinking about building something custom — whether on their own or with a partner like Ensemblex. Here's what to expect.

The Major Phases

A model build typically moves through three broad phases, though in practice they often blur together and loop back on each other.

Preparation. This is where most timelines are won or lost. The core work here is gathering data — both the predictive features available at the point of origination and the outcome data (the target variable) needed to train the model. For lenders with mature internal data environments, this is relatively straightforward. For those expanding into a new product or geography, it often means working with a credit bureau or other data source for a retrospective study — pulling a representative dataset of trade lines that match the target population. Retros are powerful tools, but they require lengthy conversations with bureaus, internal approvals, and meaningful cost. They routinely take longer than clients expect.

Beyond the data itself, preparation includes business alignment: understanding your goals, how the data maps to your current product structure, and what the model actually needs to do. If you're moving upmarket, launching a new product, or entering a new customer segment, the data you have may not cleanly match the problem you're trying to solve. Working through that misalignment early is far better than discovering it mid-build.

Build. Once the dataset is in hand and the problem is well-scoped, the actual modeling work is, in many ways, the most straightforward part of the process. Exploratory data analysis, feature selection, preliminary modeling, iteration, refinement, adverse action development — this is well-understood work, and with modern tools and good data, experienced teams can move through it efficiently.

That said, it's rarely linear. Early modeling results have a way of surfacing new questions: a feature behaves unexpectedly, a particular segment warrants a closer look, an early target variable turns out to be less predictive than anticipated. That iteration is normal and healthy — it's part of how a model gets built right — but it's worth setting the expectation upfront that the build phase involves ongoing dialogue, not just a handoff and a wait.

Implementation. Getting a model into production is often where the timeline stretches, especially at larger organizations. This phase includes model documentation, governance review, validation (internal, external, or both), technical integration into origination systems, and QA. Some of these steps — governance sign-off, legal review, compliance checks — are entirely outside a modeling team's control once they've submitted their work.

The complexity here scales with the organization and the market. A nimble fintech can sometimes move from a finished model to production in a matter of weeks. A large bank operating in a highly regulated environment may require sign-off from compliance, legal, external validators, and risk committees — a process that can take months on its own.

One nuance worth flagging: a lot of what's required in the implementation phase isn't unique to custom model builds. Adverse action notices, feature logging, decisioning infrastructure — these are things you need regardless of whether you're running a custom model or an off-the-shelf score. In many cases, a custom model slots directly into the place of whatever you're using today, and the implementation process is less disruptive than it might initially appear.

How Long Does It Take?

The honest answer is: it varies considerably. But if we're setting a baseline expectation for a US lender doing this for the first time, six months is a reasonable starting point — from scoping the project to having a model decisioning in production.

That estimate can compress significantly under the right conditions. If the data is available and well-organized, the modeling problem is clearly defined, the team can move quickly, and governance requirements are manageable, we've run projects that came together in a matter of weeks. On the other end, when data procurement is difficult, the organization moves slowly, and the regulatory environment requires extensive external review, a year or more is not unusual.

Three factors tend to drive a project toward the longer end:

• Data readiness. Is the data in hand, or does it need to be procured? Is it clean and consistently structured, or does it require significant work before it's usable?

• Organizational speed. How quickly can the client team make decisions, clear internal approvals, and execute on technical implementation?

• Regulatory environment. Markets with mature regulatory frameworks require more documentation, more sign-off, and more external review. This is especially pronounced at larger institutions.

One thing to keep in mind: the actual model development — the build phase — is often not where the time goes. With good data and a clear objective, building a model that outperforms off-the-shelf alternatives is achievable in a relatively short window. The timeline is usually determined by what happens before and after.

Where Things Go Wrong

A few patterns show up repeatedly in projects that stall or struggle.

Data that doesn't match the problem. This is probably the most common source of friction. A lender has data, but it reflects a prior product, a prior customer segment, or a prior policy — not the population the new model will decision. Working through that misalignment takes time, and sometimes it means going back to procurement when the original dataset turns out not to be fit for purpose.

Retroactive study complications. Retros are the right tool for many problems, but they come with real overhead. Poorly specified parameters, unexpected data quality issues, or bureaus returning data that doesn't match what was scoped can set a project back significantly. We've seen retros in emerging markets come back essentially unusable. Getting the requirements right upfront — ideally with the modeling team involved in the bureau conversations — is worth the effort.

Implementation complexity. The harder a client's origination system is to integrate with, the longer implementation takes. Pulling a handful of bureau features into a simple scorecard is one thing. Logging raw data, computing custom features, and piping the results through a complex origination system is another. The more bespoke the feature engineering, the more work on the back end.

Misaligned expectations on performance. This one is easy to overlook. Early in our work at Ensemblex, we came into a project where a client had spent 18 months rebuilding their originations model internally. They then wanted us to come in and beat it in a month or two. That's a difficult ask — and an unfair one. The amount of lift you can expect from a new model, and the timeline you can reasonably promise, have to be calibrated against what's already been done and what the data can actually support.

The Nature of the Work

One thing we try to convey early with any new modeling client: this is an iterative process, not a linear one. Results from early modeling inform the feature set. Feature feedback shapes how you think about the target variable. Preliminary outputs sometimes prompt a rethink of the problem framing entirely.

That's not a sign something has gone wrong. It's how good models get built. But it does mean the process works best when there's a clear feedback loop between the modeling team and the client, and when the client comes in with realistic expectations about what the work involves.

If you're new to custom model builds and trying to decide whether the investment makes sense for your business, we've covered the case for custom models elsewhere. The short version: a model built on your own data, calibrated to your specific population and product, will consistently outperform anything you can buy off the shelf — and that advantage compounds over time. The build process takes real effort, but it's the right kind of effort.

If you're ready to get into specifics, we're happy to walk through what the process would look like for your business.