cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Lake as source of images to train a classification model on a local computer

MCosta
New Contributor III

Hi Folks,

I'm evaluating Delta Lake to store image / data version control to be used to train models. I looked at a session explaining how to do this and also using MLflow to manage training (https://databricks.com/session_na21/image-processing-on-delta-lake).

Note: it'd be interesting to have a link to the source code used in the demo.

I have a slightly different scenario, though. Testing is being performed on a local machine following the quick tutorial (https://docs.delta.io/latest/quick-start.html). In this scenario, what is the best way (using as much out-of-the-box components as possible) to "grab" a local folder with images organized into subfolders (classes) and dump them into delta lake and then use a specific snapshot on tensorflow?

Thanks

1 REPLY 1

-werners-
Esteemed Contributor III

I can think of 3 ways for doing this:

  1. using the web UI (the create table option or upload data into DBFS)
  2. using databricks-connect, which bridges your local machine with the remote databricks clusters
  3. using the databricks-cli to copy local files to dbfs

your cloud vendor might also have a tool to copy local data into the cloud environment.

For your purpose (evaluating) the web UI option might be the easiest.

https://docs.databricks.com/data/data.html

https://docs.microsoft.com/en-us/azure/databricks/data/databricks-file-system#file-upload-interface

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group