Why your first ML project should be boring

Every organisation's first machine learning project is chosen the same way: someone lists the places where a model could plausibly help, and the list gets sorted by how impressive each item sounds. Demand forecasting. Churn prediction. Dynamic pricing. Something with a dashboard and a number that goes up.

That sort order is the mistake. The first project is not where you extract value from machine learning — it is where you find out whether your organisation can operate a model at all. Sort by that instead, and the list comes out in a completely different order.

What the first project is actually testing

A model in production is the smallest part of a system that also includes:

• a source of training data that will still exist in six months;
• a way to get features at inference time that match the features at training time;
• somewhere to put predictions where a human or a system will act on them;
• a way to notice that the model has become wrong;
• a person whose job it is to look.

Four of those five have nothing to do with modelling. A first project that gets a good score on a Kaggle-shaped problem and skips them has tested the one part that was never in doubt. When the second project arrives — the one that actually matters — every untested piece fails at once, and the failures are blamed on "AI" rather than on the pipeline that was never built.

"Boring" means the failure mode is obvious

The property to optimise for is not low risk. It is legible risk: when the model is wrong, can somebody tell, immediately, without a statistician?

Consider two candidates for a first project at the same company.

The second one is boring, and it is the better first project by a wide margin. It has a built-in label pipeline — every human correction is a new training example, generated as a by-product of people doing their jobs. It has an obvious baseline: whatever the routing rules do today. And when it degrades, the complaint arrives on its own.

Demand forecasting is not a worse problem. It is a worse first problem, because it fails silently, and a silent failure in the first project teaches the organisation that models are unaccountable.

The baseline you have to beat is not zero

Before the model, write down what the current process achieves — as a number, measured, not estimated. Rules, a spreadsheet, an experienced person guessing: whatever it is, it has a hit rate.

This is uncomfortable, because a surprising fraction of proposed ML projects turn out to be competing against a baseline that is already at 85% and cheap. Learning that in week one costs a week. Learning it after six months of model development costs the credibility of every proposal that follows.

It also settles the argument about whether the model is good enough. "0.91 F1" means nothing to the person who has to approve the rollout. "It gets 91% right where the current rules get 84%, and here are the twelve cases it got wrong that the rules got right" is a conversation a non-technical decision-maker can actually have.

Ship the pipeline before the model

A useful sequencing trick: build and deploy the entire system with the stupidest possible model in the middle. Most-frequent class. Yesterday's value. A hand-written rule.

class Baseline:
    """Predicts the most common label. Deploy this first."""

    def fit(self, X, y):
        self.label_ = Counter(y).most_common(1)[0][0]
        return self

    def predict(self, X):
        return [self.label_] * len(X)

Deploying that teaches you everything the real model would have taught you about integration, latency, permissions, logging and rollback — and it teaches it in a week, while nobody is emotionally invested in the outcome. Swapping in a real model afterwards is a contained change to one component, made by a team that has already been through a deploy.

The alternative — six months of modelling, then integration — puts every infrastructure surprise in the same fortnight as the executive demo.

The one that is worth the trouble

None of this is an argument for small ambitions. It is an argument about order. The ambitious project is the second one, and it goes better because the first one built the data path, the monitoring, the review workflow and the organisational habit of checking.

A first project should be chosen so that its most likely outcome — a model that is only slightly better than the rules it replaced — is still a win, because the pipeline it leaves behind is the actual deliverable.