XGBoost vs. Neural Nets for Purchase Prediction: What We Learned Building “The Cart Knows First”

“The Cart Knows First” is Skubl’s purchase-prediction model: given a shopper’s live session and history, it estimates the probability they will complete a purchase — early enough to act on it. When we started, the fashionable answer was a deep neural network. We ran the bake-off anyway, and on our data the boring answer won. Here is what we actually found.

The problem is tabular, and that matters

Purchase prediction on commerce data is a tabular problem: dozens to low-hundreds of heterogeneous features — recency, frequency, basket value, category affinities, session depth, device, time-of-day. This is exactly the regime where gradient-boosted decision trees have repeatedly beaten deep learning in the literature and in practice. Trees handle mixed feature types, monotonic relationships, and missing values natively; neural nets need heavy preprocessing before they even reach the starting line.

The head-to-head

We evaluated an XGBoost classifier against two neural architectures — a plain multilayer perceptron and a TabTransformer-style attention model — on the same features, same splits, same time-based validation. We ranked on AUC and, more importantly, on calibrated precision at the operating threshold we would actually deploy.

• Accuracy. XGBoost edged both nets on AUC and won clearly on precision-at-threshold once probabilities were calibrated. The MLP could match it only after far more tuning; the transformer never justified its overhead on this feature count.
• Latency. A boosted-tree ensemble scores in well under a millisecond on CPU. “The Cart Knows First” has to fire mid-session, so inference cost is a product constraint, not a footnote — and here the tree model was decisively cheaper.
• Explainability. SHAP values on the tree model gave us per-prediction reasons a merchandiser could read. That is not a nice-to-have; it is what lets a human trust the model enough to let it act.
• Stability. The tree model degraded gracefully as the data drifted week to week. The nets were more sensitive to distribution shift and needed retraining sooner.

Where the neural nets did help

This is not tribalism. Where we had genuinely high-cardinality, sequential signal — long clickstreams, free-text search — learned embeddings from a small net added real lift. The pattern that worked was embeddings from a net, decision from the trees: let the network compress the sequential mess into a handful of dense features, then let XGBoost make the call. We got the representational upside without handing the actual decision to a model we could not fully explain.

The deterministic takeaway

The headline is not “trees beat nets.” It is that for an operational decision you have to weigh accuracy alongside latency, stability, and — above all — whether you can explain and reproduce the output. On tabular commerce data, gradient boosting tends to win that combined scorecard, which is why it sits at the core of a deterministic system: a model whose decisions we can defend, repeat, and improve. Pick the model that lets you be accountable for the decision, not the one with the best press.