https://community.databricks.com/t5/data-engineering/databricks-autoloader-best-practice/m-p/21172#M14390 <P>Databricks Autoloader is a popular mechanism for ingesting data/files from cloud storage into Delta; for a very high throughput source, what are the best practices to be following while scaling up an autoloader based pipeline&nbsp;to the tune of millions of events per minute; </P><P></P><P>While looking at tuning "cloudFiles.fetchParallelism" is something to look at are they any other configurations that need tuning? presumably fetch rate increase should be paired with delete rate from sqs/aqs as well ? </P> Wed, 23 Jun 2021 21:28:35 GMT User16783853501 2021-06-23T21:28:35Z https://community.databricks.com/t5/data-engineering/databricks-autoloader-best-practice/m-p/21172#M14390 <P>Databricks Autoloader is a popular mechanism for ingesting data/files from cloud storage into Delta; for a very high throughput source, what are the best practices to be following while scaling up an autoloader based pipeline&nbsp;to the tune of millions of events per minute; </P><P></P><P>While looking at tuning "cloudFiles.fetchParallelism" is something to look at are they any other configurations that need tuning? presumably fetch rate increase should be paired with delete rate from sqs/aqs as well ? </P> Wed, 23 Jun 2021 21:28:35 GMT https://community.databricks.com/t5/data-engineering/databricks-autoloader-best-practice/m-p/21172#M14390 User16783853501 2021-06-23T21:28:35Z