- Tags:: 📝CuratedNotes , Data Science
There are several phenomena we should pay attention to:
- Real Model performance, for those cases where we get the label of live data.
- Early indicators of possible bad performance. We can distinguish between:
- Single prediction indicators :
- Broken data integrity of features Model sanity checks.
- Model outlier detection or trespassed hard limits in features.
- Outliers or trespassed hard limits in predictions.
- Distribution indicators Model drifting:
- Distribution shift of predictions.
- Distribution shift of features.
- Single prediction indicators :
Tools:
- GCP has AI Platform. From https://cloud.google.com/ai-platform/docs/ml-solutions-overview:
- You deploy by uploading your model to GCS.
- 2021-01-08 Monitoring is in beta, and it is called “Continuous Evaluation”: https://cloud.google.com/ai-platform/prediction/docs/continuous-evaluation/view-metrics but in only supports image, text or general classification.