topic Re: Parallel execution of SQL cell in Databricks Notebooks in Data Engineering

Parallel execution of SQL cell in Databricks Notebooks

Phani1 — Tue, 30 Apr 2024 11:58:02 GMT

Hi Team,

Please provide guidance on enabling SQL cells parallel execution in a notebook containing multiple SQL cells. Currently, when we execute notebook and all the SQL cells they run sequentially. I would appreciate assistance on how to execute the SQL cells in parallel using an automated approach. For instance, when we invoke a notebook which consist of multiple SQL cells, it is necessary for those cells to run in parallel.

Regards,

Janga

Re: Parallel execution of SQL cell in Databricks Notebooks

Aviral-Bhardwaj — Tue, 30 Apr 2024 12:38:03 GMT

This will help you - https://community.databricks.com/t5/data-engineering/support-of-running-multiple-cells-at-a-time-in-databricks/td-p/9590

I think its depends on which Runtime versions we are using, @Ajay-Pandey can guide here more

Re: Parallel execution of SQL cell in Databricks Notebooks

Ajay-Pandey — Tue, 30 Apr 2024 13:35:19 GMT

Hi @Phani1 ,

Can you please explain your usecase as databricks notebook support the sequential executions we have to look for workaround so it will great if you can explain it more.

For now you can manually run multiple cell for sql but it's not possible through workflow jobs

Re: Parallel execution of SQL cell in Databricks Notebooks

Phani1 — Wed, 01 May 2024 02:59:56 GMT

Hi Ajay-Pandey ,

Here is my scenario: In our current situation, we need to load data in multiple stages. Each stage includes a set of insert statements. Instead of running these statements one after another, we can run them in parallel to save time.

Stage1: (below insert statements A,B,C needs to run parallel)
Insert into table A
Insert into table B
Insert into table C

Stage2: (below insert statements D,E needs to run parallel)
Insert into table d based on A,B
Insert into table e based on B,C

Stage 3 (below insert statements F,G needs to run parallel)
Insert into table f based on d
Insert into table g based on e

Stage 4
insert into table h based on above tables

Regards,Janga

Re: Parallel execution of SQL cell in Databricks Notebooks

Ajay-Pandey — Thu, 02 May 2024 05:29:16 GMT

Hi @Phani1

Yes you can achieve this scenario with the help of Databricks Workflow jobs where you can create task and dependencies for each other.