Experiment Tracking
Intro
Machine learning involves a lot of experimentation. As we work to improve our models we’ll try out different techniques, gather better data, and tweak certain parameters. As we do all this, we look at metrics like accuracy to decide how good a model is.
Experiment tracking is what allows data scientists to compare models over time, identify factors that affect performance, and share experiments with colleagues for easier collaboration.
Typically, we want to track the following:
- The source code version and data version used in the model,
- Model training parameters such as neural network size or epoch count,
- Training performance metrics such as accuracy,
- The final model assets, which are ready to be deployed into production.
Do I need it?
An experiment tracker enables you to reproduce any model from the past. It doesn’t do this alone — to have full reproducibility you also need data version control and source control — but an experiment tracker is the only tool that combines all of the relevant information about a model.
When a model’s performance changes, experiment trackers allow you to go back and understand why, and this in turn means you can make the right decisions to improve your model in the future.
Or, if you have a particular experiment that you’d like to share with a colleague, in order to get their input or a review, an experiment tracker makes it easy for your colleague to see not only the end result but exactly how you got there.
As a tool both for ensuring reproducibility and enabling collaboration, we think experiment tracking is a key piece of your MLOps infrastructure.
What are the options?
There are a lot of experiment trackers out there, but most of them provide:
- A central location where each experiment can be recorded. Often this will be a database or a cloud storage bucket.
- A way for your model training code to interact with the experiment tracker.
- A friendly user interface for data scientists to browse past experiments.
