topic Re: Unstructured Data (Images) training in Databricks in Get Started Discussions

Unstructured Data (Images) training in Databricks

aswinkks — Fri, 02 May 2025 17:12:00 GMT

I'm looking for a solution where

1. Need a delta table that saves the pointers(path) of the images in volume

2. Train a Pytorch, tensorflow or keras model with the data from delta lake.

I tried multiple methods :

1. Storing the data as vectors or binary instead of path - I don't need this as it's not efficient

2. Used deltatorch instead of Pandas as Pytorch doesn't support spark APIs - Seems to have some issues and Dont need pandas it's not supported for Big Data

3. I have created a delta table with pointers(path) to the image, but the issue is that when converted to python objects, It's not able to read the file from "dbfs:"

Can anyone please tell if you have trained a model according to the requirement I have posted

Re: Unstructured Data (Images) training in Databricks

Louis_Frolio — Fri, 02 May 2025 17:37:14 GMT

I managed to find a few solution accelerators that are in the ballpark, albeit not exact, to what you are trying to accomplish. Have a look:

1. https://www.databricks.com/solutions/accelerators/digital-pathology

2. https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/Image-classification-deep-learning

3. https://www.databricks.com/solutions/accelerators/pixels-medical-image-processing

4. https://www.databricks.com/solutions/accelerators/product-quality-inspection

Hope this helps. Big Roux.

Re: Unstructured Data (Images) training in Databricks

aswinkks — Sat, 10 May 2025 12:53:01 GMT

Thanks @Louis_Frolio , the solution helps.