AutoLoader - handle spark write transactional (_SUCCESS file) on ADLS

Marcin_U
New Contributor II

‎03-05-2024 06:01 AM

Spark write method (df.write.parquet) to parquet files is transactional. I mean after write is sucessfull file _SUCCESS is created in path where parquet files was loaded.

Marcin_U_0-1709647032623.png

Is it possible to configure AutoLoader to load parquet files only in case when write is done with success (_SUCCESS file was appeared) ?

0 Kudos

Marcin_U
New Contributor II

‎03-12-2024 08:30 AM

I think my question wasn't understood correctly. I meant AutoLoader as the data loading tool provided by Databricks (https://docs.databricks.com/en/ingestion/auto-loader/index.html).

AutoLoader has set of different options to setup (https://docs.databricks.com/en/ingestion/auto-loader/options.html) but I don't find any option to help me achive resault which I described in this topc. Any ideas how to resolve my problem?

0 Kudos

PotnuruSiva
Databricks Employee
Databricks Employee

‎12-04-2024 06:14 AM

@Marcin_U Please use the below option in the readStream to load only parquet files

.option("pathGlobfilter", "*.parquet")

Please refer to the below documentation:

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/options.html#:~:text=Defau...

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#:~:text=opti...