โ01-23-2023 06:27 AM
What would be the best plan for independent course creator?
Hi folks! I want to use databrick community edition as the platform to teach online courses. As you may know, for community edition, you need to create a new cluster when the old one terminates. I found out however tables created from the old cluster would disappear as well, so I have to re-create tables every time I start the cluster.
I wonder if anyone knows how to fix this problem. I have a couple tables with static data with 100+ questions. It will be too much repetitive work if students need to upload the data and re-create the steps every time they re-start a new cluster.
Looking forward to your advice!
โ01-23-2023 11:20 PM
Hi, It will be helpful if you provide the screenshots before and after the incident.
โ01-24-2023 04:29 PM
Hi @Lizhi Dongโ,
This might be a limitation from Community Edition. When your cluster gets terminated all your tables will be removed.
โ02-24-2023 03:29 PM
Hi @Lizhi Dongโ,
Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
3 weeks ago
I am also facing the same issue
3 weeks ago
Separation of Storage and Compute:
Cluster Metadata:
Persistence of Data Files:
To persist the metadata (schemas, table definitions, etc.) across cluster restarts, you can use the following methods:
External Hive Metastore:
Example of configuring an external Hive metastore:
Metastore Persistence in DBFS:
Example of saving table definitions and recreating them:
By using these methods, you can ensure that both your data and metadata are preserved across cluster restarts in Databricks Community Edition.
3 weeks ago
I believe only the metadata get's removed from HMS not the delta files from dbfs. Instead of loading the data again and again try using ctas with that dbfs location.
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!