Is it possible to use Autoloader with a daily update file structure?

StephanieAlba — Fri, 08 Apr 2022 16:10:51 GMT

We get new files from a third-p@rty each day. The files could be the same or different. However, each day all csv files arrive in the same dated folder. Is it possible to use autoloader on this structure?We want each csv file to be a table that gets updated each day. Like account table and accounting table....

Re: Is it possible to use Autoloader with a daily update file structure?

Hubert-Dudek — Fri, 08 Apr 2022 16:25:19 GMT

@Stephanie Rivera , You can use pathGlobfilter, but you will need a separate autoloader for which type of file.

df_alert = spark.readStream.format("cloudFiles") \

.option("cloudFiles.format", "binaryFile") \

.option("pathGlobfilter", alert.csv") \

.load(<base_path>)

I think I prefer to set some copy activity first (in Azure Data Factory, for example) to group all files in the same folder on Data Lake. So, for example, alerts.csv is copied to the alert folder and renamed to date, so alerts/2022-04-08.csv (or maybe parquet instead). Then folder I would register in databricks metastore so it will be queryable like SELECT * FROM Alerts, or as Data Live Table to convert it. Then, in the copy activity in Azure Data Factory, you can set it to detect only new files and copy them.

topic Re: Is it possible to use Autoloader with a daily update file structure? in Data Engineering

Is it possible to use Autoloader with a daily update file structure?

Re: Is it possible to use Autoloader with a daily update file structure?