Databricks Community

HTD360

Hi there,I have a question regarding the autoloader without SDP and auto-scaling of clusters. I'm reading the following in the docs:Production considerations for Structured Streaming | Databricks on AWS:Do not enable autoscaling for compute for Struc...

HTD360

Hi, thank you for your answer. Could you elaborate a bit on this?for non SDP available now auto loader jobs autoscaling can be reasonableHow do you decide on whether it is reasonable or not? Especially you said it is not recommended to enable compute...

HTD360

And to add to the question. What if I have a job with 10 tasks that all use the autoloader. Would that benefit from auto-scaling?

Databricks Community

User Stats

User Activity

Autoscaling with the autoloader without SDP

Re: Autoscaling with the autoloader without SDP

Re: Autoscaling with the autoloader without SDP