← Back to work

Machine learning · Model reliability

ShiftGuard

Can distribution shift warn us before a classifier fails?

I built ShiftGuard to explore whether changes in incoming data can warn us that a model is becoming less reliable. It compares several ways of measuring those changes.

View repository ↗
Failure detection ROC curves from the full ShiftGuard experiment.
Failure detection ROC curves from the full ShiftGuard experiment.

About the project

What it is

ShiftGuard studies a practical problem: a model can look healthy when it is deployed even while the data it receives is changing. I built a repeatable experiment to test whether those changes can act as an early warning.

The project compares several distance measures across different datasets, classifiers, and types of shift. It also looks at false alarms and warning time, because a signal is only useful when it gives a clear and timely warning.