cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

VN11111
by New Contributor III
  • 9650 Views
  • 5 replies
  • 6 kudos

Resolved! ERROR: Some streams terminated before this command could finish!

I have a databricks notebook which is to read stream from Azure Event Hub.My code does the following:1.Configure path for Eventhubs2.Read Streamdf_read_stream = (spark.readStream .format("eventhubs") .options(**conf)...

  • 9650 Views
  • 5 replies
  • 6 kudos
Latest Reply
guru1
New Contributor II
  • 6 kudos

I am also facing same issue , using Cluster11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) liberary : com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.21Please help me for sameconf = {}conf["eventhubs.connectionString"] = "Endpoint=sb://xxxx.ser...

  • 6 kudos
4 More Replies
osoucy
by New Contributor II
  • 1134 Views
  • 0 replies
  • 1 kudos

Is it possible to join two aggregated streams of data?

ObjectiveWithin the context of a delta live table, I'm trying to merge two streams aggregation, but run into challenges. Is it possible to achieve such a join?ContextAssume- table trades stores a list of trades with their associated time stamps- tabl...

  • 1134 Views
  • 0 replies
  • 1 kudos
AdamRink
by New Contributor III
  • 4497 Views
  • 3 replies
  • 0 kudos

Try catch multiple write streams on a job

We are having issues with checkpoints and schema versions getting out of date (no idea why), but it causes jobs to fail. We have jobs that are running 15-30 streaming queries, so if one fails, that creates an issue. I would like to trap the checkpo...

  • 4497 Views
  • 3 replies
  • 0 kudos
Latest Reply
AdamRink
New Contributor III
  • 0 kudos

The problem is that on startup if a stream fails, it would never hit the awaitAnyTermination? I almost want to take that while loop and put it on a background thread to start that at the beginning and then fire all the streams afterward... not sure ...

  • 0 kudos
2 More Replies
User16826994223
by Honored Contributor III
  • 1119 Views
  • 1 replies
  • 1 kudos

Resolved! If multiple streams as jar is running and one fail will it fails others also

Hi Team, IHAC who is running multiple stream in the jar based job, one of the streams got terminated, but other streams are processing without termination.Is this know behaviour in case of jar based streaming application? Any insight please? (edited)

  • 1119 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 1 kudos

Failure in any of the active streaming queries causes the active run to fail and terminate all the other streaming queries. You do not need to use  streamingQuery.awaitTermination()or spark.streams.awaitAnyTermination() at the end of your notebook. ...

  • 1 kudos
Labels