topic Re: dlt vs delta table in Data Engineering

dlt vs delta table

pargit2 — Fri, 20 Jun 2025 15:18:00 GMT

Hi,

I'm building gold layer and silver layers.

in bronze I ingest using auto loader.

data is getting updated once a month.

should I save the fd in silver notebooks using delta live table or delta table?

in the past I used simple:

df.write.save("s3.." )

then created a table in uc pointing on path in s3.

what is the best approach? when do I use dlt vs simple delta?

abdominal for my use case what is the best practice?

#dlt

Re: dlt vs delta table

nayan_wylde — Fri, 20 Jun 2025 15:48:23 GMT

I would say if the data is not complex and you are not handling any DQ checks in the pipeline the go for a regular databricks workflow and save it as delta table since you are refreshing the data every 1 month and it is not streaming workload.

Re: dlt vs delta table

pargit2 — Fri, 20 Jun 2025 16:20:07 GMT

thanks for the replay.

I also plan to orchestrated it all from airflow and I prefer not to use databricks workflow. I know dlt work on dlt pipeline I am not sure if and how it should be done.

so you suggest saving in s3 and then create table in uc pointing on s3?

thanks

Re: dlt vs delta table

nayan_wylde — Fri, 20 Jun 2025 16:32:07 GMT

I think if you are registering the table in UC and using delta live table you cannot create an external table in s3. It should be a managed table. DLT, by design, works with managed tables in Unity Catalog.

Re: dlt vs delta table

pargit2 — Fri, 20 Jun 2025 16:52:05 GMT

I meant for my use case maybe it's better to simply save in uc instead of using dlt .what do you think?

Re: dlt vs delta table

nayan_wylde — Fri, 20 Jun 2025 17:03:56 GMT

Yes if your other tables are external. Then no point going for DLT and managed table. You can just write to s3 and register the table in UC.