12-04-2024 06:14 AM
@Marcin_U Please use the below option in the readStream to load only parquet files
.option("pathGlobfilter", "*.parquet")
Please refer to the below documentation:
https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/options.html#:~:text=Defau...
https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#:~:text=opti...
