topic Re: Data persistence, Dataframe, and Delta in Data Engineering

Data persistence, Dataframe, and Delta

Vik1 — Fri, 21 Jan 2022 17:30:02 GMT

I am new to databricks platform.

what is the best way to keep data persistent so that once I restart the cluster I don't need to run all the codes again?So that I can simply continue developing my notebook with the cached data.
I have created many dataframes and I want to save them as Delta table using the code

dataFrame.to_delta('/dbfs/Projects/', index_col='index')

then I list the table using the command I see a table with two columns: path, and name. The path column contains the path starting from dbfs:/dbfs/Projects/part-00000-xxxx-snappy.parquet. The name column has only the filename part. How will I later query those two tables if the dataframe name is not saved with the filename. Do I have to query by the extremely long filename.?

Re: Data persistence, Dataframe, and Delta

Anonymous — Fri, 21 Jan 2022 19:50:45 GMT

Hi @Vivek Ranjan! My name is Piper, and I'm a moderator for the community. Welcome to Databricks and the community! Thank you for your question. We give our members time to respond to questions before we circle back.

Thanks in advance for your patience and best wishes on your Databricks journey.

Re: Data persistence, Dataframe, and Delta

-werners- — Tue, 25 Jan 2022 14:32:41 GMT

you can just use spark.read.format("delta").load("path to the parent folder of 'delta_log'-folder")

or save it as a table and read that table.

https://docs.microsoft.com/en-us/azure/databricks/delta/quick-start

Re: Data persistence, Dataframe, and Delta

Anonymous — Wed, 26 Jan 2022 16:42:15 GMT

@Vivek Ranjan - Does werners' response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

Re: Data persistence, Dataframe, and Delta

Anonymous — Thu, 12 May 2022 13:50:49 GMT

Hey there @Vivek Ranjan

Hope you are doing great!

Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.

Thanks!