Databricks Community

oye

Hi, I have set up a job of multiple spark python tasks running in parallel. I have only set up one job cluster, single node, data security mode SINGLE_USER, using Databricks Runtime version 14.3.x-scala2.12. These parallel spark python tasks share so...

oye

Hi thanks for replying!In my case, it would be running parallel tasks of type spark python tasks in a lakeflow job. This is a screenshot of the setup: Aside from the fact that the tasks will share the same resource and thus might run slower, I wonder...

Databricks Community

User Stats

User Activity

Using a cluster of type SINGLE_USER to run parallel python tasks in one job

Re: Using a cluster of type SINGLE_USER to run parallel python tasks in one job