-werners-
Esteemed Contributor III

Ok, the only thing I notice is that you have set a termination time which is not necessary for streaming (if you are doing real-time).

I also notice you do not set a checkpoint location, something you might consider.

You can also try to remove the maxEvent and maxRate config.

A snippet from the docs:

Here are the details of the recommended job configuration.

  • Cluster: Set this always to use a new cluster and use the latest Spark version (or at least version 2.1). Queries started in Spark 2.1 and above are recoverable after query and Spark version upgrades.
  • Alerts: Set this if you want email notification on failures.
  • Schedule: Do not set a schedule.
  • Timeout: Do not set a timeout. Streaming queries run for an indefinitely long time.
  • Maximum concurrent runs: Set to 1. There must be only one instance of each query concurrently active.
  • Retries: Set to Unlimited.

https://docs.databricks.com/spark/latest/structured-streaming/production.html

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html