cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Hi Experts : I am new to Databricks please help me on below. Question : How is delta table stored in DBFS ?

BasavarajAngadi
Contributor

If I create delta table the table is stored in parque format in DBFS location ?

and please share how the parque files supports schema evolution if i do DML operation.

As per my understanding : we read data from data lake first in data frame and try to write the dataframe to delta tables and these delta tables when created as tables are stored in parque format ??

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

delta lake is parquet on steroids. The actual data is stored in parquet files, but you get a bunch of extra functionalities (time traveling, ACID, optimized writes, MERGE etc).

check this page for lots of info.

Delta lake does support schema evolution but it has it´s limitations.

About the location: they will indeed be stored in dbfs, but dbfs is a 'virtual' layer on top of your actual storage. So if you mount your data lake into dbfs, dbfs will contain your data (it will link it, not copy it).

View solution in original post

3 REPLIES 3

-werners-
Esteemed Contributor III

delta lake is parquet on steroids. The actual data is stored in parquet files, but you get a bunch of extra functionalities (time traveling, ACID, optimized writes, MERGE etc).

check this page for lots of info.

Delta lake does support schema evolution but it has it´s limitations.

About the location: they will indeed be stored in dbfs, but dbfs is a 'virtual' layer on top of your actual storage. So if you mount your data lake into dbfs, dbfs will contain your data (it will link it, not copy it).

Kaniz
Community Manager
Community Manager

Hi @Basavaraj Angadi​ , Did you get the solution to your problem?

Kaniz
Community Manager
Community Manager

Hi @Werner Stinckens​ , Thank you so much for your contribution to our Community.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.