Parallel execution of SQL cell in Databricks Notebooks

[Phani1]

Hi Team,

Please provide guidance on enabling SQL cells  parallel execution in a notebook containing multiple SQL cells. Currently, when we execute notebook and all the SQL cells they run sequentially. I would appreciate assistance on how to execute the SQL cells in parallel using an automated approach. For instance, when we invoke a notebook which consist of multiple SQL cells, it is necessary for those cells to run in parallel.

Regards,

Janga

[Aviral-Bhardwaj]

This will help you - https://community.databricks.com/t5/data-engineering/support-of-running-multiple-cells-at-a-time-in-...

I think its depends on which Runtime versions we are using, @Ajay-Pandey can guide here more

[Ajay-Pandey]

Hi @Phani1 ,

Can you please explain your usecase as databricks notebook support the sequential executions we have to look for workaround so it will great if you can explain it more.

For now you can manually run multiple cell for sql but it's not possible through workflow jobs

[Phani1]

Hi Ajay-Pandey ,

Here is my scenario: In our current situation, we need to load data in multiple stages. Each stage includes a set of insert statements. Instead of running these statements one after another, we can run them in parallel to save time.

Stage1: (below insert statements A,B,C needs to run parallel)
Insert into table A
Insert into table B
Insert into table C

Stage2: (below insert statements D,E needs to run parallel)
Insert into table d based on A,B
Insert into table e based on B,C

Stage 3 (below insert statements F,G needs to run parallel)
Insert into table f based on d
Insert into table g based on e

Stage 4
insert into table h based on above tables

Regards,Janga

[Ajay-Pandey]

Hi @Phani1 

Yes you can achieve this scenario with the help of Databricks Workflow jobs where you can create task and dependencies for each other.