topic Re: AutoLoader - handle spark write transactional (_SUCCESS file) on ADLS in Data Engineering

AutoLoader - handle spark write transactional (_SUCCESS file) on ADLS

Marcin_U — Tue, 05 Mar 2024 14:01:59 GMT

Spark write method (df.write.parquet) to parquet files is transactional. I mean after write is sucessfull file _SUCCESS is created in path where parquet files was loaded.

Is it possible to configure AutoLoader to load parquet files only in case when write is done with success (_SUCCESS file was appeared) ?

Re: AutoLoader - handle spark write transactional (_SUCCESS file) on ADLS

Marcin_U — Tue, 12 Mar 2024 15:30:57 GMT

I think my question wasn't understood correctly. I meant AutoLoader as the data loading tool provided by Databricks (https://docs.databricks.com/en/ingestion/auto-loader/index.html).

AutoLoader has set of different options to setup (https://docs.databricks.com/en/ingestion/auto-loader/options.html) but I don't find any option to help me achive resault which I described in this topc. Any ideas how to resolve my problem?

Re: AutoLoader - handle spark write transactional (_SUCCESS file) on ADLS

PotnuruSiva — Wed, 04 Dec 2024 14:14:32 GMT

@Marcin_U Please use the below option in the readStream to load only parquet files

.option("pathGlobfilter", "*.parquet")

Please refer to the below documentation:

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/options.html#:~:text=Default%20value%3A%20None-,pathGlobFilter,-or%20fileNamePattern

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#:~:text=option(%22-,pathGlobfilter,-%22%2C%20%22*.png