Execute Pyspark cells concurrently

[Phani1]

Hi Team,

Hi Team,

Is it feasible to run pyspark cells concurrently in databricks notebooks? If so, kindly provide instructions on how to accomplish this. We aim to execute the intermediate steps simultaneously.

The given scenario entails the simultaneous execution of several PySpark cells based on a condition.

Regards,

Janga

[Miguel_Suarez]

Hi @Phani1, 

Unfortunately, there isn't a way to run cells in a notebook simultaneously. But with your use case needing the parallel execution of code, you can configure a Databricks Workflow with multiple tasks running concurrently: https://learn.microsoft.com/en-us/azure/databricks/jobs/#what-is-a-task

Best

[NandiniN]

Databricks also supports executing SQL cells in parallel. While a command is running and your notebook is attached to an interactive cluster, you can run a SQL cell simultaneously with the current command. The SQL cell is executed in a new, parallel session. However, this feature is limited to SQL cells and does not apply to PySpark cells.

You can use the dbutils.notebook.run command to run other notebooks from within a notebook. This command can be used to trigger multiple notebooks to run concurrently. However, this approach is limited by the number of concurrent notebook runs allowed by your Databricks workspace.

Databricks Workflows also enable you to define and run a sequence of tasks, including notebooks, Python scripts, and SQL queries. You can configure these tasks to run in parallel by setting up dependencies appropriately.

https://docs.databricks.com/en/notebooks/notebooks-code.html