topic Re: temp tables in Databricks in Data Engineering

temp tables in Databricks

AMadan — Thu, 07 Sep 2023 15:30:38 GMT

Hey Team,

We are in a migration phase from SQL server to Databricks.

In our sql server databricks, lot of times, #temp tables created and dropped like typical Datawarehousing sql.

In Databricks, we can do 2 things. I want to check which one is best approach and why?

1) create temp views in databricks and make use of views later in the query

2) create delta table with temp(prefix) and make use of temp table and at the end drop temp table.

I am looking forward for your response.

Re: temp tables in Databricks

berserkersap — Sat, 16 Sep 2023 15:25:43 GMT

Just to add a bit more from experience

If you are doing to use the temp tables too many times, then using temp views is going to slow down your cluster . It is due to lazy evaluation unless you persist that view (dataframe cache or globaltempview).

Every time your temp view is called, it will execute the view definition/code. This also depends upon your cluster capacity, how complex the view definition is, how much data the temp view has.

Please correct me if I am wrong.