Delta Lake Table Daily Read and Write job optimization

AH
New Contributor III

AH_0-1717569489175.png

I have created 7 job for each business system to extract product data from each postgress source then write all job data into one data lake delta table [raw_product].

each business system product table has around 20 GB of data.

do the same thing for 15 table .

is any way to read and write fast in delta tables  

 

one job looks like the one below

AH_1-1717572455868.png

daily day loaded into delta table by using merge command

AH_3-1717572644640.png

 

AH_2-1717572557758.png