cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Hi Experts : I am new to Databricks please help me on below. Question : How is delta table stored in DBFS ?

BasavarajAngadi
Contributor

If I create delta table the table is stored in parque format in DBFS location ?

and please share how the parque files supports schema evolution if i do DML operation.

As per my understanding : we read data from data lake first in data frame and try to write the dataframe to delta tables and these delta tables when created as tables are stored in parque format ??

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

delta lake is parquet on steroids. The actual data is stored in parquet files, but you get a bunch of extra functionalities (time traveling, ACID, optimized writes, MERGE etc).

check this page for lots of info.

Delta lake does support schema evolution but it has it´s limitations.

About the location: they will indeed be stored in dbfs, but dbfs is a 'virtual' layer on top of your actual storage. So if you mount your data lake into dbfs, dbfs will contain your data (it will link it, not copy it).

View solution in original post

3 REPLIES 3

-werners-
Esteemed Contributor III

delta lake is parquet on steroids. The actual data is stored in parquet files, but you get a bunch of extra functionalities (time traveling, ACID, optimized writes, MERGE etc).

check this page for lots of info.

Delta lake does support schema evolution but it has it´s limitations.

About the location: they will indeed be stored in dbfs, but dbfs is a 'virtual' layer on top of your actual storage. So if you mount your data lake into dbfs, dbfs will contain your data (it will link it, not copy it).

Kaniz_Fatma
Community Manager
Community Manager

Hi @Basavaraj Angadi​ , Did you get the solution to your problem?

Kaniz_Fatma
Community Manager
Community Manager

Hi @Werner Stinckens​ , Thank you so much for your contribution to our Community.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!