topic Re: DLT: Autoloader Perf in Data Engineering

DLT: Autoloader Perf

Gilg — Fri, 06 Oct 2023 03:07:03 GMT

Hi Team,

I am looking for some advice to perf tune my bronze layer using DLT.

I have the following code very simple and yet very effective.

@dlt.create_table(name="bronze_events", comment = "New raw data ingested from storage account landing zone.") def bronze_events(): df = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .schema(schema) .load("abfss://data@<some storage account>.dfs.core.windows.net/0_Landing") ) return df

that generates this DAG.

Before it was executing quite fast but as days goes by it is becoming more and more slower like from 2 to 5 to 12 mins. Silver and Gold are all executing less than a minute. So wondering what performance tuning I should do with the bronze layer.

Cheers,

Re: DLT: Autoloader Perf

Tharun-Kumar — Fri, 06 Oct 2023 04:30:27 GMT

Hi @Gilg

Is it ingesting the same number of files as before?

Also, you could try using Auto Loader with file notification mode. If there are too many files in the source directory, then significant amount of time would be spent on listing of the directory. We can validate this by analyzing the logs.

Re: DLT: Autoloader Perf

Gilg — Fri, 06 Oct 2023 05:00:08 GMT

It can be from 600 files to up to 1.5k files. The DLT is set to Triggered in Pipeline mode and Continuous in Trigger Type in Workflows.

Re: DLT: Autoloader Perf

Tharun-Kumar — Wed, 11 Oct 2023 09:49:25 GMT

Hi @Gilg

You mentioned that micro-batch time is around 12 minutes recently. Do we also see jobs/stages with 12 minutes in the spark ui. If that is the case, then the processing of the file itself takes 12 minutes. If not, the 12 minutes is spent on listing the directory and maintaining the checkpoint.