lingareddy_Alva
Esteemed Contributor

Hi @ChristianRRL 

Your Assumptions - Partially Correct
You're correct about several key points:

1. File listing overhead: Yes, the trigger does need to list files in the monitored location to detect new arrivals
2. Cloud provider costs: Listing operations do incur costs (though typically minimal per operation)
3. Continuous polling: The trigger checks at regular intervals regardless of whether new files arrive

However, there are some optimizations and considerations that affect the impact:
How File Arrival Triggers Actually Work
Optimization Mechanisms:
1. Incremental Detection: Most implementations use timestamps or other metadata to avoid full scans
2. Efficient Listing: Cloud providers optimize listing operations for performance
3. Batching: Multiple file arrivals within a short window are often batched together

Cost Perspective:
-- Storage listing costs are typically very low (e.g., AWS S3 LIST requests cost $0.0004 per 1,000 requests)
-- For your 100k files example: Even with minute-by-minute checks, the listing cost would be negligible compared to compute costs

When File Arrival Triggers Make Sense
Good Use Cases:
1. Low to Moderate File Volumes (hundreds to low thousands of files)
2. Predictable Arrival Patterns (files arrive regularly but not constantly)
3. Near Real-time Requirements (need to process files within minutes of arrival)
4. Event-driven Architectures (want to trigger downstream processes immediately)

 

 

LR

View solution in original post