Databricks Community

johschmidt42

Hi, I'm using the Auto Loader feature to read streaming data from Delta Lake files and process them in a batch. The trigger is set to availableNow to include all new data from the checkpoint offset but I limit the amount of delta files for the batch ...

johschmidt42

It works when changing "cloudFiles.maxFilesPerTrigger" to "maxFilesPerTrigger". But this is unexpected..

johschmidt42 · 11-17-2023

@RaocoSolutions This works. However, when I asked the Databricks Team, they did not recommed it for production use cases.

Databricks Community

User Stats

User Activity

Autoloader cloudFiles.maxFilesPerTrigger ignored with .trigger(availableNow=True)?

Re: Autoloader cloudFiles.maxFilesPerTrigger ignored with .trigger(availableNow=True)?

Re: Can you use Managed Identities in databricks besides Unity Catalog