I have created a ETL pipeline with DLT. My first step is to ingest into raw delta table using autoloader file notification. when I have 20k notification pipe line run well across all stages. But when we have surge in number of messages pipeline waits for first step to complete before processing next steps. this is strange considering my first step is a streaming delta table and next steps has dependency on this. question: is it possible to use rate limit with autoloader file notification implementation?
Welcome to Databricks Community: Lets learn, network and celebrate together
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.