- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-21-2021 02:03 PM
I am already using MLflow. What benefit would Delta provide me since I am not really working on Data engineering workloads
- Labels:
-
Delta
-
Delta Tables
-
MlFlow
-
MLUseCases
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2021 02:58 AM
Because Delta can version data, it becomes useful for reproducibility and debugging of models. Weeks later you could see exactly how the table looked when the model was built. MLflow's "Spark" autologging actually helps automatically capture and log this version information when Delta is used in a Databricks notebook.
Its transactional writes are useful, as a modeling job does not need to worry about other data engineering jobs writing to the same data source at the same time. To a lesser extent, being able to write Delta Live Tables and/or being able to roll back bad writes increases the reliability of upstream data, which helps with downstream reliability of ML jobs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2021 02:58 AM
Because Delta can version data, it becomes useful for reproducibility and debugging of models. Weeks later you could see exactly how the table looked when the model was built. MLflow's "Spark" autologging actually helps automatically capture and log this version information when Delta is used in a Databricks notebook.
Its transactional writes are useful, as a modeling job does not need to worry about other data engineering jobs writing to the same data source at the same time. To a lesser extent, being able to write Delta Live Tables and/or being able to roll back bad writes increases the reliability of upstream data, which helps with downstream reliability of ML jobs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-13-2021 06:00 AM
The most important aspect is your experiment can track the version of the data table. So during audits you will be able to trace back why a specific prediction was made.