Databricks, the company behind the commercial development of Apache Spark, is placing its machine learning lifecycle project MLflow under the stewardship of the Linux Foundation.
MLflow provides a programmatic way to deal with all the pieces of a machine learning project through all its phases — construction, training, fine-tuning, deployment, management, and revision. It tracks and manages the the datasets, model instances, model parameters, and algorithms used in machine learning projects, so they can be versioned, stored in a central repository, and repackaged easily for reuse by other data scientists.
MLflow’s source is already available under the Apache 2.0 license, so this isn’t about open sourcing a previously proprietary project. Instead, it’s about giving the project “a vendor neutral home with an open governance model,” according to Databricks’s press release.
Projects for managing entire machine learning pipelines have taken shape over the past couple of years, providing single overarching tools for governing what is typically a sprawling and complex process involving multiple moving parts. Among them is a Google project, Tensorflow Extended, but better known is its descendent project Kubeflow, which uses Kubernetes to manage machine learning pipelines.
MLflow differs from Kubeflow in several key ways. For one, it doesn’t require Kubernetes as a component; it runs on local machines by way of simple Python scripts, or in Databricks’s hosted environment. And while Kubeflow focuses on TensorFlow and PyTorch as its learning systems, MLflow is agnostic — it can work with models from those frameworks and many others.