03-26-2022 04:18 AM
If I create delta table the table is stored in parque format in DBFS location ?
and please share how the parque files supports schema evolution if i do DML operation.
As per my understanding : we read data from data lake first in data frame and try to write the dataframe to delta tables and these delta tables when created as tables are stored in parque format ??
03-28-2022 01:24 AM
delta lake is parquet on steroids. The actual data is stored in parquet files, but you get a bunch of extra functionalities (time traveling, ACID, optimized writes, MERGE etc).
check this page for lots of info.
Delta lake does support schema evolution but it has it´s limitations.
About the location: they will indeed be stored in dbfs, but dbfs is a 'virtual' layer on top of your actual storage. So if you mount your data lake into dbfs, dbfs will contain your data (it will link it, not copy it).
03-28-2022 01:24 AM
delta lake is parquet on steroids. The actual data is stored in parquet files, but you get a bunch of extra functionalities (time traveling, ACID, optimized writes, MERGE etc).
check this page for lots of info.
Delta lake does support schema evolution but it has it´s limitations.
About the location: they will indeed be stored in dbfs, but dbfs is a 'virtual' layer on top of your actual storage. So if you mount your data lake into dbfs, dbfs will contain your data (it will link it, not copy it).
04-07-2022 01:32 PM
Hi @Basavaraj Angadi , Did you get the solution to your problem?
04-26-2022 12:51 PM
Hi @Werner Stinckens , Thank you so much for your contribution to our Community.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.