Delta live table blocks pipeline autoloader rate limit
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-16-2023 04:29 AM
I have created a ETL pipeline with DLT. My first step is to ingest into raw delta table using autoloader file notification.
when I have 20k notification pipe line run well across all stages. But when we have surge in number of messages pipeline waits for first step to complete before processing next steps.
this is strange considering my first step is a streaming delta table and next steps has dependency on this.
question: is it possible to use rate limit with autoloader file notification implementation?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-17-2023 02:51 PM
Did you try following options:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-17-2023 03:44 PM
yep set it to 15K. Does these impact when running using file notification?
I am looking at parallel fetch whats the upper limit for this?
How to derive this number, is it based on number of workers, executors or cores?
