Somewhere, an ML model is proudly reporting 99.4% accuracy. No training. No feature engineering. No hyperparameter tuning. No GPU trying to heat the neighborhood. The model is correct most of the time but useful none of the time. It catches no fraud and probably still asks for a promotion.
This is the classic trap of imbalanced datasets. In fraud detection, for example, fraudulent transactions might be only 0.6% of all transactions. A model that simply predicts "not fraud" for every transaction will achieve 99.4% accuracy. It will be correct 99.4% of the time and useful 0% of the time.
Precision and Recall
To understand why, we need to look beyond accuracy:
- Precision: Of all the transactions the model flagged as fraud, how many were actually fraud? Low precision means your system keeps blocking genuine customers. Congratulations—you have successfully detected someone buying groceries.
- Recall: Of all the actual fraud cases, how many did the model catch? Low recall means the fraudsters leave with the money while the model celebrates its excellent accuracy.
In an imbalanced dataset, you can have high accuracy while both precision and recall are abysmal. The model is useless for its intended purpose.
The Business Cost
The "best" threshold for a classification model is not only a mathematical choice. It depends on business cost. If false positives (blocking a genuine customer) are expensive, you might want higher precision, even if it means lower recall. If false negatives (missing a fraud case) are more expensive, you want higher recall.
The optimal threshold is where the cost of false positives equals the cost of false negatives. This is a business decision, not a technical one.
The Takeaway
Accuracy is a dangerous metric on imbalanced datasets. Always look at precision, recall, and the confusion matrix. And remember: a model that's 99.4% accurate might be 100% useless.