Databricks Community

DataBRObin · 05-30-2023

I am running training of a Keras/Tensorflow deep learning model on a cluster of (for now) 2 workers and 1 driver (T4 GPU, 28GB, 4 core) using the Databricks provided HorovodRunner. It all seems to go well and the performance scales quite nicely over ...

DataBRObin · 08-04-2022

For a computer vision project, my raw data consists of encrypted videos (60fps) stored in Azure Blob Storage. In order to have the data usable for model training, I need to do some preprocessing and for that I need the video split into individual fra...

DataBRObin · 06-13-2023

Hi Sean, thank you for coming back to me and my apologies for my delayed response. Unfortunately, those state callbacks are necessary for Elastic Horovod to work (https://horovod.readthedocs.io/en/stable/elastic_include.html) which I want to use as i...

DataBRObin · 05-30-2023

In the end, I decided to change around the workflow so it is as efficient as I could imagine it:Extract frames of video files in a containerized application somewhere running ffmpeg and storing the resulting frames in a parquet file in blob storage (...

DataBRObin · 09-09-2022

Hi @Vidula Khanna Hope you're doing good too! I haven't really been able to solve my issue beside the workaround mentioned above, with a very low number of videos to process at at time with a large VM, so I still consider this issue unresolved (and ...

DataBRObin · 08-12-2022

In the end, I managed to get it to work. The way I did it is to only have one executor per cluster node (assign executor.cores to be the node core count -2 and memory as much as allowed) and repartition to more partitions than rows in the table and l...

DataBRObin · 08-04-2022

I believe that is true as well, since I am just using the python library implementation of it. If there would be a way to run it sequentially on the driver but once one row has been processed it hands it off to the executors again and materialises it...

Databricks Community

User Stats

User Activity

Running Keras model training with HorovodRunner works until the training function is exited ("The MPI_Query_thread() function was called after MPI_FINALIZE was invoked.")

FFmpeg frame extraction explodes memory, how to mitigate?

Re: Running Keras model training with HorovodRunner works until the training function is exited ("The MPI_Query_thread() function was called after MPI_FINALIZE was invoked.")

Re: FFmpeg frame extraction explodes memory, how to mitigate?

Re: FFmpeg frame extraction explodes memory, how to mitigate?

Re: FFmpeg frame extraction explodes memory, how to mitigate?

Re: FFmpeg frame extraction explodes memory, how to mitigate?