Notes on AI
Why Experiment Management is the Key to Success in Data Science
If you recognize its advantages, you will adopt it today.
--
To build a machine learning or ML product, you must train a large number of models with different parameter configurations. These models are trained using a training dataset that evolves over time. The performance metrics can also be changed according to various business requirements. Nevertheless, a data scientist or machine learning engineer must manage this complex process and be able to always deploy the best model suitable for each scenario. A technology that helps us manage this process is essential.
In software development, git technology along with various workflow structures are used to track code changes and manage software versions. This technology expedites the development process and enables us to collaborate easily. However, we still do not have a well-adapted management system in machine learning, mostly due to its complexity.
In the machine learning context, we often use “experiment” instead of “version” that is used in software development. An experiment has a broader definition compared to a version. An experiment is a series of research and development tasks to train a model that may fail or succeed. An experiment is tagged with a large number of artifacts such as log files, models, and performance reports. The experiment management system or EMS is the version control system in the machine learning context. In this article, you find answers to questions such as “Why do we need an experiment management system?” or “What are the criteria for a useful EMS?”.
An experiment is a series of research and development tasks to train a model that may fail or succeed. An experiment is tagged with a large number of artifacts such as log files, ML models, and performance reports.
