- 5785 Views
- 2 replies
- 4 kudos
Resolved! address how to use multiple spark streaming jobs connecting to one job cluster
Hi,We have a scenario where we need to deploy 15 spark streaming applications on databricks reading from kafka to single Job cluster. We tried following approach:1. create job 1 with new job cluster (C1)2. create job2 pointing to C1...3. create job15...
- 5785 Views
- 2 replies
- 4 kudos
- 4 kudos
@Hubert Dudek​ , thanks a lot for responding. When we have setup like this, if one tasks fails, it will not terminate the entire job right?Since, the job is continously running as it is streaming app, is it possible to add new task to the job(while i...
- 4 kudos
- 15582 Views
- 1 replies
- 0 kudos
How to run multiple spark streaming application on databricks cluster?
I started working on databricks. I need to migrate few streaming jobs from Ambari to Databricks. I deployed one job using jar and it. is working fine. But when I deploy the second job I faced an error " multiple spark streaming context not allowed". ...
- 15582 Views
- 1 replies
- 0 kudos
- 0 kudos
You can run multiple streaming applications in databricks clusters. By default, this would run in the same fair scheduling pool. To enable multiple streaming queries to execute jobs concurrently and to share the cluster efficiently, you can set the q...
- 0 kudos