Databricks Streaming: Recommended Cluster Types and Best Practices

[FarhanM]

Hi Community, I recently built some streaming pipelines (Autoloader-based) that extract JSON data from the Data Lake and, after parsing and logging, dump it into the Delta Lake bronze layer. Since these are streaming pipelines, they are supposed to run indefinitely until I deliberately stop them. However, I’ve noticed that the Databricks clusters (All-Purpose Compute) tend to become unstable after a day or two of continuous execution.

To keep things running, I’ve currently implemented an optimizer job that’s scheduled daily to stop the cluster, restart it, and then re-trigger the streaming pipeline.

I feel this might not be a best practice. Could you please suggest what type of clusters are most suitable for streaming jobs/pipelines and what the best practices are for managing streaming systems in Databricks?

[bianca_unifeye · Accepted solution]

When running streaming pipelines, the key is to design for stability and isolation, not to rely on restart jobs.

The first thing to do is run your streams on Jobs Compute, not All-Purpose clusters. If available, use Serverless Jobs. Each pipeline should have its own dedicated job cluster, which ensures clean, isolated runtimes, consistent libraries, and automatic retries, all of which reduce drift and instability.

Choose a recent LTS Databricks Runtime with Photon (e.g., 14.x or 15.x). Photon gives a real boost in JSON parsing and Delta writes. Enable autoscaling with a minimum of at least two workers to prevent executors from churning, avoid min=0 for long-running streams.

You didn’t mention if you’re using Delta Live Tables (now called Declarative Pipelines), but that’s worth exploring. DLT automatically manages cluster lifecycles, recovery, data quality checks, autoscaling, and lineage, all built in.

In short:

Run your workloads through Workflows (Jobs) using Job or Serverless clusters with retries, autoscaling floors, proper checkpoints, file-notification mode, and monitoring.

There’s no need for a separate optimizer job to stop and restart clusters, follow the checkpoint/notifications/small-file/state management hygiene instead. You’ll find detailed guidance in the Databricks documentation on streaming best practices and Auto Loader performance tuning.