topic Re: What are the advantages of using Delta if I am using MLflow? How is Delta useful for DS/ML use cases? in Data Engineering

What are the advantages of using Delta if I am using MLflow? How is Delta useful for DS/ML use cases?

Anonymous — Mon, 21 Jun 2021 21:03:47 GMT

I am already using MLflow. What benefit would Delta provide me since I am not really working on Data engineering workloads

Re: What are the advantages of using Delta if I am using MLflow? How is Delta useful for DS/ML use cases?

sean_owen — Tue, 22 Jun 2021 09:58:16 GMT

Because Delta can version data, it becomes useful for reproducibility and debugging of models. Weeks later you could see exactly how the table looked when the model was built. MLflow's "Spark" autologging actually helps automatically capture and log this version information when Delta is used in a Databricks notebook.

Its transactional writes are useful, as a modeling job does not need to worry about other data engineering jobs writing to the same data source at the same time. To a lesser extent, being able to write Delta Live Tables and/or being able to roll back bad writes increases the reliability of upstream data, which helps with downstream reliability of ML jobs.

Re: What are the advantages of using Delta if I am using MLflow? How is Delta useful for DS/ML use cases?

Sebastian — Mon, 13 Sep 2021 13:00:01 GMT

The most important aspect is your experiment can track the version of the data table. So during audits you will be able to trace back why a specific prediction was made.