Databricks Community

Do we have option to pass spark config variables in terms of executors workers etc while using autoloader?

I have created a ETL pipeline with DLT. My first step is to ingest into raw delta table using autoloader file notification. when I have 20k notification pipe line run well across all stages. But when we have surge in number of messages pipeline waits...