cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Jin_Kim
by New Contributor II
  • 3156 Views
  • 4 replies
  • 5 kudos

Resolved! address how to use multiple spark streaming jobs connecting to one job cluster

Hi,We have a scenario where we need to deploy 15 spark streaming applications on databricks reading from kafka to single Job cluster. We tried following approach:1. create job 1 with new job cluster (C1)2. create job2 pointing to C1...3. create job15...

  • 3156 Views
  • 4 replies
  • 5 kudos
Latest Reply
Kaniz
Community Manager
  • 5 kudos

Hi @Jin Kim​, Just a friendly follow-up. Do you still need help, or the above responses help you to find the solution? Please let us know.

  • 5 kudos
3 More Replies
User16790091296
by Contributor II
  • 14710 Views
  • 1 replies
  • 0 kudos

How to run multiple spark streaming application on databricks cluster?

I started working on databricks. I need to migrate few streaming jobs from Ambari to Databricks. I deployed one job using jar and it. is working fine. But when I deploy the second job I faced an error " multiple spark streaming context not allowed". ...

  • 14710 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

You can run multiple streaming applications in databricks clusters. By default, this would run in the same fair scheduling pool. To enable multiple streaming queries to execute jobs concurrently and to share the cluster efficiently, you can set the q...

  • 0 kudos
Labels