Unstructured Data (Images) training in Databricks

[aswinkks]

I'm looking for a solution where

1. Need a delta table that saves the pointers(path) of the images in volume

2. Train a Pytorch, tensorflow or keras model with the data from delta lake.

I tried multiple methods :

1. Storing the data as vectors or binary instead of path - I don't need this as it's not efficient

2. Used deltatorch instead of Pandas as Pytorch doesn't support spark APIs - Seems to have some issues and Dont need pandas  it's not supported for Big Data

3. I have created a delta table with pointers(path) to the image, but the issue is that when converted to python objects, It's not able to read the file from "dbfs:"

Can anyone please tell if you have trained a model according to the requirement I have posted

[BigRoux · Accepted solution]

I managed to find a few solution accelerators that are in the ballpark, albeit not exact, to what you are trying to accomplish.   Have a look:

1. https://www.databricks.com/solutions/accelerators/digital-pathology

2. https://www.databricks.com/resources/demos/tutorials/data-science-and-ai/Image-classification-deep-l...

3. https://www.databricks.com/solutions/accelerators/pixels-medical-image-processing

4. https://www.databricks.com/solutions/accelerators/product-quality-inspection

Hope this helps. Big Roux.

[aswinkks]

Thanks @BigRoux , the solution helps.