Model drifting

• Tags:: 📝CuratedNotes , Data Engineering

A model is drifting if the joint distribution of features and target of the training data is different from the joint distribution of features and target of the real data in which the model is being used, potentially hurting the model performance: the model was trained in a setting that does not longer reflects reality.

$P(x_s,y_s)\ne P(x_t,y_t)$

Since:

• $P(x,y)=P(x)P(y|x)$, when $X$ causes $Y$.
• $P(x,y)=P(y)P(x|y)$, when $Y$ causes $X$.

there are three types of drifting depending on the distribution that shifts:

• Covariate shift

• Label shift
• Concept drift

Interesting refs

• Chip Huyen on Twitter: “Sooo I wrote a 13,000-word lecture note on data distribution shifts, monitoring, and causes of ML failures. This was very difficult for me to write, because academia & industry literature use very different terminology. Feedback appreciated 🙏 https://t.co/DOTjb1EEGZ https://t.co/cNp8N67uqr” / Twitter