Your model will optimize for exactly what you tell it to. The problem is, we rarely tell it the right thing.
Cross-validation gave you a stable, consistent estimate. But stability only matters if you are measuring the right thing to begin with. A model can be consistently wrong – low variance across folds, perfectly calibrated confidence, optimizing hard for a number that does not reflect what actually matters in production.
That is the metrics selection problem. And it starts before you write a single line of code.
The domain defines what failure costs
A model only understand relationship between input and output, not business value. Data goes in. A decision comes out.
The same model architecture – Random Forest, anything – pointed at different domains produces fundamentally different decisions:
A fraud detection system says: this transaction is clean. It is not. A fraudster gets through. In payments, that carries direct financial cost, regulatory exposure, and erosion of customer trust.
A cancer screening system says: this scan is clear. It is not. A patient goes undiagnosed. Irreversible. Catastrophic. Human cost.
A house pricing model says: this property is worth X. It is worth X minus 15%. Symmetric financial error – overpricing and underpricing both matter differently to different stakeholders.
Same model family. Completely different measurement framework. The metric must encode the cost of being wrong in the specific way your domain punishes – not the way that is mathematically convenient.
What breaks when you choose the wrong metric
You optimize for the wrong thing — and the model follows perfectly.
A fraud model optimized for accuracy on an imbalanced dataset — where 99% of transactions are legitimate — can learn to predict not fraud for everything and still score 99%. It has learned nothing. It will catch nothing. But the number looks excellent.
Freakonomics taught me to always ask: what are the incentives actually rewarding here? A metric is an incentive structure encoded in mathematics. Whatever you measure is what the model will learn to optimize. If your metric does not reflect what actually matters in production, the model will find the shortest path to a good score — not a good decision.
This is Goodhart’s Law operating silently inside your training loop: once a measure becomes a target, it ceases to be a good measure.
The threshold is where business meets math
Precision and recall are not fixed properties of a model. They are functions of where you draw the decision boundary.
Lower the threshold and you flag more transactions as fraud and recall goes up. You catch more real fraud. But precision drops. More legitimate customers get blocked. Customer support costs rise. User experience hampers and churn increases.
Raise the threshold to only flag high-confidence fraud and precision improves. Fewer false alarms. But recall falls. Fraudster smartly makes low-confidence fraud rings and losses skyrockets.
That tradeoff is not a modeling decision. It is a business decision. How much friction can you introduce to legitimate customers before they leave? How much fraud loss is acceptable before regulators act? Those questions live outside the model. The threshold is where their answers get encoded into math.
Choosing a threshold without that business conversation is choosing who bears the cost of errors without telling them.
One number hides where the model actually fails
A global AUC is an average. It tells you how the model performs across the whole population, not where it quietly breaks.
In payments, a fraud model with strong overall AUC can be dangerously mis-calibrated on specific merchant categories, new customer segments, or low-frequency transaction types. The aggregate number looks fine. The subgroup failure is invisible until fraud/customer complaints exposes it.
This is where post-deployment surprises come from. Not from the model being globally wrong but from it being locally wrong in exactly the segments that matter most.
The mental model to carry forward
Metrics are not properties of models. They are properties of decisions.
Before you ask which model to build, ask: what does being wrong cost and which kind of wrong is more expensive?
Before you trust a number, ask: is this an average hiding a subgroup failure?
Before you set a threshold, ask: who bears the cost if I move it in either direction?
The answers to those questions are your metric strategy. Everything else is implementation.