topic Re: parallelizing function call in databricks in Data Engineering

parallelizing function call in databricks

Shivanshu_ — Thu, 18 Jan 2024 09:32:11 GMT

I have a use case where I have to process stream data and have to create categorical table's(500 table count). I'm using concurrent threadpools to parallelize the whole process, but while seeing the spark UI, my code dosen't utilizes all the workers(Cluster configuration: Standard_e8ads type for both driver and worker, and 4 workers 32gb memory and 4 cores each). I'm using 4 threads.

the code sometimes executes on the driver or the worker, I never get utilization more than 40 to 45% for 5million records.

The function I call using threadpool has all spark code in it.

Any help on the issue will be highly appriciated, and thanks in advance.

Re: parallelizing function call in databricks

-werners- — Wed, 31 Jan 2024 14:19:44 GMT

AFAIK threadpool works on a single machine. So by using it you cannot scale out to multiple nodes.
These tables you are talking about, are these spark tables or from a database?

Re: parallelizing function call in databricks

Shivanshu_ — Wed, 31 Jan 2024 14:25:11 GMT

Spark tables

Re: parallelizing function call in databricks

-werners- — Wed, 31 Jan 2024 14:39:44 GMT

why not creating a single table with 500 partitions?
If that is not an option, you could still write the data as a partitioned parquet file and then create tables out of each partition using a small python script.

Re: parallelizing function call in databricks

jose_gonzalez — Wed, 31 Jan 2024 19:03:35 GMT

You can use DLT, read from many-to-one table.