cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Parallel execution of SQL cell in Databricks Notebooks

Phani1
Valued Contributor II

Hi Team,

Please provide guidance on enabling SQL cells  parallel execution in a notebook containing multiple SQL cells. Currently, when we execute notebook and all the SQL cells they run sequentially. I would appreciate assistance on how to execute the SQL cells in parallel using an automated approach. For instance, when we invoke a notebook which consist of multiple SQL cells, it is necessary for those cells to run in parallel.

Regards,

Janga

4 REPLIES 4

Aviral-Bhardwaj
Esteemed Contributor III

This will help you - https://community.databricks.com/t5/data-engineering/support-of-running-multiple-cells-at-a-time-in-...

I think its depends on which Runtime versions we are using, @Ajay-Pandey can guide here more

AviralBhardwaj

Ajay-Pandey
Esteemed Contributor III

Hi @Phani1 ,

Can you please explain your usecase as databricks notebook support the sequential executions we have to look for workaround so it will great if you can explain it more.

For now you can manually run multiple cell for sql but it's not possible through workflow jobs

Ajay Kumar Pandey

Phani1
Valued Contributor II

Hi Ajay-Pandey ,

Here is my scenario: In our current situation, we need to load data in multiple stages. Each stage includes a set of insert statements. Instead of running these statements one after another, we can run them in parallel to save time.

Stage1: (below insert statements A,B,C needs to run parallel)
Insert into table A
Insert into table B
Insert into table C

Stage2: (below insert statements D,E needs to run parallel)
Insert into table d based on A,B
Insert into table e based on B,C

Stage 3 (below insert statements F,G needs to run parallel)
Insert into table f based on d
Insert into table g based on e

Stage 4
insert into table h based on above tables

Regards,Janga

Ajay-Pandey
Esteemed Contributor III

Hi @Phani1 

Yes you can achieve this scenario with the help of Databricks Workflow jobs where you can create task and dependencies for each other.

AjayPandey_0-1714627725482.png

 

Ajay Kumar Pandey

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group