topic Delta tables and YOLO computer vision tasks in Data Engineering

Delta tables and YOLO computer vision tasks

Andrewcon — Sun, 10 Mar 2024 14:09:07 GMT

Hi all,

I would really appreciate if someone could help me out. I feel it’s both a data engineering and ML question.

One thing we use at wo is YOLO for object detection. I’ve managed to run YOLO by loading data from the blob storage, but I’ve seen that the best way to do deep learning tasks in Databricks is to train your ML models on Delta Live Tables.

I currently have my training dataset as a Delta table, and I was wondering if anyone has managed to train computer vision models on Delta tables.

I’ve read the documentations and have seen repos such as petastorm that try to implement training on delta tables, but I can’t for the life of me understand how to actually run yolo this way, especially since YOLO uses yaml for config.

Thank in advance for your help! 😇

Re: Delta tables and YOLO computer vision tasks

jnap — Thu, 21 Nov 2024 00:00:38 GMT

I am also looking for an answer to this question. Did you manage to find a solution @Andrewcon ?

Re: Delta tables and YOLO computer vision tasks

MathieuDB — Wed, 15 Jan 2025 15:29:42 GMT

Hello @Andrewcon and @jnap ,

Have a look at Mosaic Streaming Dataset. You could load your data from your delta table and then train it on your PyTorch YOLO model. In that example, it use mobilenet model but you can adapt it to use YOLO. Petastorm is another option, but the project is a bit outdated.

https://docs.databricks.com/en/machine-learning/load-data/streaming.html