- 4537 Views
- 4 replies
- 5 kudos
Resolved! address how to use multiple spark streaming jobs connecting to one job cluster
Hi,We have a scenario where we need to deploy 15 spark streaming applications on databricks reading from kafka to single Job cluster. We tried following approach:1. create job 1 with new job cluster (C1)2. create job2 pointing to C1...3. create job15...
- 4537 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Jin Kim​, Just a friendly follow-up. Do you still need help, or the above responses help you to find the solution? Please let us know.
- 5 kudos
- 15115 Views
- 1 replies
- 0 kudos
How to run multiple spark streaming application on databricks cluster?
I started working on databricks. I need to migrate few streaming jobs from Ambari to Databricks. I deployed one job using jar and it. is working fine. But when I deploy the second job I faced an error " multiple spark streaming context not allowed". ...
- 15115 Views
- 1 replies
- 0 kudos
- 0 kudos
You can run multiple streaming applications in databricks clusters. By default, this would run in the same fair scheduling pool. To enable multiple streaming queries to execute jobs concurrently and to share the cluster efficiently, you can set the q...
- 0 kudos