Autoscaling with the autoloader without SDP

[HTD360]

Hi there,

I have a question regarding the autoloader without SDP and auto-scaling of clusters. I'm reading the following in the docs:

• Production considerations for Structured Streaming | Databricks on AWS:
  Do not enable autoscaling for compute for Structured Streaming jobs.
• Configure Auto Loader for production workloads | Databricks on AWS: Enhanced autoscaling implements optimization of streaming workloads and adds enhancements to improve the performance of batch workloads. Enhanced autoscaling optimizes costs by adding or removing machines as the workload changes.
• But also: Compute auto-scaling has limitations when scaling down cluster size for structured streaming workloads. Databricks recommends using Lakeflow Spark Declarative Pipelines with enhanced autoscaling for streaming workloads.

We don't use SDP because of serverless limitations. Is it not advised to use enhanced autoscaling for non-SDP jobs? And why is that?

[HTD360]

And to add to the question. What if I have a job with 10 tasks that all use the autoloader. Would that benefit from auto-scaling?

[amirabedhiafi]

Hello !

In DBKS we have 2 different autoscaling mechanisms here:

- normal compute autoscaling on a job or all purpose cluster : this is the autoscaling you enable on a classic job cluster by setting min or max workers and for structured streaming jobs, it is not recommended to enable compute autoscaling because scale down has limitations for streaming workloads. The cluster may not scale down as expected and if you want to resize you will experience latency especially for stateful streams.

- autoscaling for LSDP : this is a pipeline specific autoscaling mode that uses pipeline workload metrics such as task slot usage and queued tasks. It improves streaming workload optimization and can proactively shut down under used nodes while avoiding failed tasks during shutdown.

So shortly :

• for non SDP continuous auto loader jobs you can use fixed size jobs compute
• for non SDP available now auto loader jobs autoscaling can be reasonable
• for streaming autoscaling with better scale down behavior LSDP autoscaling is the recommended option

[HTD360]

Hi, thank you for your answer. Could you elaborate a bit on this?
for non SDP available now auto loader jobs autoscaling can be reasonable

How do you decide on whether it is reasonable or not? Especially you said it is not recommended to enable compute autoscaling because scale down has limitations for streaming workloads

[FelixBlom]

Also interested in this

[ThomazNeto]

Hi,

Two clarifications that I think answer both of your questions.

Why the docs say no to autoscaling on streaming jobs. Classic autoscaling only removes a node when it's idle (or, on Premium, "underutilized over the last 40 seconds" on jobs compute, judged by shuffle file state). A continuous stream with a short trigger never lets the workers go idle, so the cluster scales up when a spike arrives and then has nothing that tells it to scale back down. You end up paying for the peak. The docs put it as "Compute auto-scaling has limitations scaling down cluster size for Structured Streaming workloads" and leave it there, but that's the mechanism. Enhanced autoscaling fixes exactly that: it looks at task slot utilization and queue size, and "proactively shuts down under-utilized nodes while guaranteeing there are no failed tasks during shutdown", versus classic which "scales down nodes only if the node is idle".
https://docs.databricks.com/aws/en/compute/configure#how-autoscaling-behaves
https://docs.databricks.com/aws/en/ldp/auto-scaling

Enhanced autoscaling is a pipelines feature only, you can't switch it on for a plain job cluster. But, and this matters given your reason for avoiding pipelines, it does not require serverless. It works on classic pipeline compute; the docs show the config ("autoscale": {"min_workers": 5, "max_workers": 10, "mode": "ENHANCED"}) and even have a section on monitoring it for classic pipelines. So the serverless limitations don't apply if you run the pipeline on classic clusters.

How to decide for a non-pipeline Auto Loader job. It comes down to the trigger. With Trigger.AvailableNow the stream processes what's there and stops, so workers do go idle at the end and autoscaling behaves like any batch job. That's the case where it's reasonable, and it's also what the Auto Loader production page recommends when cost matters more than latency, ideally with a file arrival trigger. With a continuous trigger (processingTime, or a continuous job), use fixed-size jobs compute and size it for the workload.
https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/production
https://docs.databricks.com/aws/en/structured-streaming/production

For the 10 Auto Loader tasks in one job: if they're AvailableNow and run on a shared job cluster, autoscaling helps because tasks start and finish at different times and the cluster can shrink as they complete. If they're continuous, the docs' advice is one appropriately sized jobs compute per streaming task, no autoscaling.

Hope this helps.