Data bricks Write Performance

rami1 — Thu, 29 Jul 2021 14:56:45 GMT

I have a requirement to replay ingestion from landing data and build silver table. I am trying to write delta file from raw Avro files based in landing zone. The raw files are located in folder based on date. I am currently using streaming to read data from files and write into delta file with Z order.

Issue is that it is taking 8-10 hours for for each day[around 20 million records]. Any pointers on how may I can improve performance ?

Few I can think of are:

I can use batch read instead of stream?

Use Parquet instead of Delta ?

Any other?

Thanks in advance for kind help

topic Data bricks Write Performance in Data Engineering

Data bricks Write Performance