- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2022 09:30 AM
I am new to databricks platform.
- what is the best way to keep data persistent so that once I restart the cluster I don't need to run all the codes again?So that I can simply continue developing my notebook with the cached data.
- I have created many dataframes and I want to save them as Delta table using the code
dataFrame.to_delta('/dbfs/Projects/', index_col='index')
- then I list the table using the command I see a table with two columns: path, and name. The path column contains the path starting from dbfs:/dbfs/Projects/part-00000-xxxx-snappy.parquet. The name column has only the filename part. How will I later query those two tables if the dataframe name is not saved with the filename. Do I have to query by the extremely long filename.?
- Labels:
-
DataPersistence
-
Delta
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2022 06:32 AM
you can just use spark.read.format("delta").load("path to the parent folder of 'delta_log'-folder")
or save it as a table and read that table.
https://docs.microsoft.com/en-us/azure/databricks/delta/quick-start

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2022 11:50 AM
Hi @Vivek Ranjan! My name is Piper, and I'm a moderator for the community. Welcome to Databricks and the community! Thank you for your question. We give our members time to respond to questions before we circle back.
Thanks in advance for your patience and best wishes on your Databricks journey.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2022 06:32 AM
you can just use spark.read.format("delta").load("path to the parent folder of 'delta_log'-folder")
or save it as a table and read that table.
https://docs.microsoft.com/en-us/azure/databricks/delta/quick-start

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2022 08:42 AM
@Vivek Ranjan - Does werners' response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-12-2022 06:50 AM
Hey there @Vivek Ranjan
Hope you are doing great!
Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.
Thanks!

