why does a model perform well offline but not online?

Original answer: Why does a model train and predict well offline but have no effect after launch?

In recommendation systems, it is common to see major offline gains in metrics such as AUC, precision, and recall, followed by disappointing—or even declining—business metrics after launch, such as online CTR or CVR. This post lists common causes to help diagnose the problem.

1. Inconsistent offline and online features

Feature inconsistency is often the primary reason a model performs poorly online. Its causes can be varied and subtle.

1. Implementation bugs. Offline and online feature ETL is usually not implemented by the same code. Offline features may be computed in SQL on a big-data platform, while online features are produced in C++ or Go. The same logic must therefore be implemented twice, potentially by different people, and inconsistencies are likely without rigorous tests. The fundamental way to ensure consistency is to use the same code and data source for feature extraction. A common industry approach is to log real-time features during online scoring; training then only joins labels to produce positive and negative samples, without a separate feature-assembly step.
2. Feature-update latency. Real-time statistical or sequential features may be simulated correctly offline but fail to update promptly in production, creating a feature-distribution mismatch that is effectively overfitting. Even hourly or daily features can be delayed.

2. Data leakage

Data leakage, sometimes called leakage or time travel, occurs when data used to train a machine-learning model includes information about the outcome being predicted. In other words, information from test data leaks into training data. This information can concern the target label, or data that is available in training but unavailable or invalid in the real world at prediction time.

Types of data leakage

There are two broad categories: training-data leakage and feature leakage. Training-data leakage occurs when test or future data is mixed into training data; feature leakage occurs when features contain information about the true label.

Training-data leakage can occur when preprocessing uses the entire dataset—both training and test data. This includes computing normalization and scaling parameters, finding global minima and maxima to detect or remove outliers, imputing missing training values from distributions over the full dataset, and feature selection. In time-series data, future events may also accidentally be used to compute a feature for a past prediction.

Feature leakage can occur through invalid features. For example, a disease-diagnosis model might use whether a patient underwent surgery for the disease being diagnosed. A CTR model might use interactions on a product-detail page—such as commenting or contacting the seller—that have not happened yet at prediction time.

Detecting data leakage

Before building a model, exploratory analysis can identify features highly correlated with the target; extreme correlations may indicate leakage. After training, inspect highly weighted features for leakage. If a model performs unbelievably well, leakage should be suspected.

A more reliable check is a limited real deployment that compares training performance with behavior in the real environment. A large gap can also be caused by overfitting, however.

3. Distribution mismatch

1. Temporal drift. Distribution changes can result from marketing campaigns, seasons, trends, and changing aesthetic preferences.
2. Geographic drift. At the same moment, people in southern and northern regions may prefer different products. A training set containing only certain locations will mismatch test data from other locations. Cross-border e-commerce must account for the fact that target markets may be in opposite hemispheres and need different seasonal recommendations.
3. Scenario mismatch. Small scenarios often borrow data from high-traffic scenarios because they lack data. A large scenario may use a personalized recommender as its base model, while a small scenario relies on non-personalized popularity ranking. The large scenario dominates the training distribution, so a model trained on it may not work well in the small scenario. Transfer learning needs care.
4. The iceberg effect. This subtle and common recommendation-system mismatch means offline training uses the biased data above the iceberg, while online prediction must score the whole iceberg—including the large amount of data below the surface. In the baseline below, green represents positive samples, red negative samples, and gray represents data never shown online because the existing recommender assigned it low scores.

Offline, an improved model may rank observed positive samples higher, increasing AUC. Online, its ordering of observed samples might remain unchanged, but it may assign higher scores to unseen gray samples. If those samples perform poorly, offline AUC can improve markedly while online CTR declines.

The iceberg effect can be large when an experimental model differs greatly from the baseline—for example, a personalized DNN replacing a popularity strategy. The new model exposes many items that the baseline rarely showed, whose click and conversion performance is uncertain. Initially the new model is fitting samples created by the old one. If launch performance is poor, the data distribution will gradually move toward the new model after iterations, but inefficiently.

Two ways to mitigate it are:

1. Upsample less-biased data. This can be samples from random or exploration traffic, or samples generated by the new model. The goal is to make better use of samples favorable to the new model.
2. Blend online models. Linearly combine the new and old predictions. At launch, choose a small smoothing coefficient $a$, then increase it gradually as the system iterates:

$$pctr = a \cdot pctr_{new} + (1-a) \cdot pctr_{old}.$$

4. Model overfitting

Even when offline test data is isolated from training data, overfitting remains possible.

In Kaggle competitions, a model can rank highly on the public leaderboard and poorly on the private leaderboard because it has overfit the public leaderboard. Likewise, if many hyperparameter variants are evaluated and selected on the same test set, repeating that process can overfit to the test set. The selected model may not perform well online.

Using a test set to select a model is itself equivalent to training a “model-selection model” on that test set, and that process can overfit too. In an extreme case, the chosen model might memorize every test label and fail every sample outside the test set. This is also why some machine-learning models claim to beat humans on benchmark datasets.

5. Inconsistent offline and online metrics

During optimization, we have seen high offline AUC, or a large AUC improvement, with disappointing CTR; we have also seen small AUC gains with large CTR gains. Why? AUC is the probability that a model ranks a positive ahead of a negative for an arbitrary pair from the full sample set. Offline AUC therefore measures ranking across samples from multiple requests, while CTR improvement requires ranking items within the same request. Strong global ranking does not necessarily imply strong within-request ranking.

GAUC (Group AUC) can be closer to online CTR preference because it evaluates samples grouped by user session.

6. Other causes

Experimental results themselves may not be statistically trustworthy; see a checklist for poor recommendation results for more.

Traffic cannibalization and pipeline entanglement can matter too. In marketing, optimizing an upstream push notification, SMS, or fixed-entry advertisement may convert the best customers early, making downstream IVR or human telemarketing more difficult. A model trained on historical data can then improve offline without much online lift. This is common in waterfall processes. Monitor upstream and downstream experiment changes, and use an MVP mechanism that continuously A/B tests the best traffic. Relative improvement then remains measurable. Even if a small module’s metric worsens, the overall business result may still improve.

June 22, 2025, Suzhou