topic Execute Pyspark cells concurrently in Data Engineering

Execute Pyspark cells concurrently

Phani1 — Mon, 29 Apr 2024 16:36:50 GMT

Hi Team,

Is it feasible to run pyspark cells concurrently in databricks notebooks? If so, kindly provide instructions on how to accomplish this. We aim to execute the intermediate steps simultaneously.

The given scenario entails the simultaneous execution of several PySpark cells based on a condition.

Regards,

Janga

Re: Execute Pyspark cells concurrently

Miguel_Suarez — Thu, 23 Jan 2025 03:58:27 GMT

Hi @Phani1,

Unfortunately, there isn't a way to run cells in a notebook simultaneously. But with your use case needing the parallel execution of code, you can configure a Databricks Workflow with multiple tasks running concurrently: https://learn.microsoft.com/en-us/azure/databricks/jobs/#what-is-a-task

Best

Re: Execute Pyspark cells concurrently

NandiniN — Sat, 01 Feb 2025 07:08:07 GMT

Databricks also supports executing SQL cells in parallel. While a command is running and your notebook is attached to an interactive cluster, you can run a SQL cell simultaneously with the current command. The SQL cell is executed in a new, parallel session. However, this feature is limited to SQL cells and does not apply to PySpark cells.

You can use the dbutils.notebook.run command to run other notebooks from within a notebook. This command can be used to trigger multiple notebooks to run concurrently. However, this approach is limited by the number of concurrent notebook runs allowed by your Databricks workspace.

Databricks Workflows also enable you to define and run a sequence of tasks, including notebooks, Python scripts, and SQL queries. You can configure these tasks to run in parallel by setting up dependencies appropriately.

https://docs.databricks.com/en/notebooks/notebooks-code.html