topic The database and tables disappears when I delete the cluster in Data Engineering

The database and tables disappears when I delete the cluster

William_Scardua — Thu, 02 Dec 2021 00:13:39 GMT

Hi guys,

I have a trial databricks account, I realized that when I shutdown the cluster my databases and tables is disappear .. that is correct or thats is because my account is trial ?

Re: The database and tables disappears when I delete the cluster

Prabakar — Thu, 02 Dec 2021 08:17:16 GMT

Hi @William Scardua this is expected. To view the database and tables you need a running cluster. If none of the clusters are running then you won't be able to see the database list and you will see as below.

Once you start a cluster you will be able to see the database and tables.

Re: The database and tables disappears when I delete the cluster

Hubert-Dudek — Thu, 02 Dec 2021 10:38:19 GMT

exactly as @Prabakar Ammeappin said. If you want to keep access to your databases you need general-purpose cluster running. It will not work with (cheaper) job cluster. When you use jdbc/odbc connector to Spark (or some built in like in PowerBI) you also need to have general-purpose cluster running but soon for that purpose new SQL cluster will be introduced (which will be much cheaper than general-purpose cluster).

Re: The database and tables disappears when I delete the cluster

William_Scardua — Thu, 02 Dec 2021 11:59:44 GMT

Thank you @Hubert Dudek and @Prabakar Ammeappin

but it's a bit weird because if i shut down my cluster I lose all my databases and tables

I create a new cluster and a new database

I shutting down/delete my cluster

When I create a new cluster the database is not existe anymore

Can you imagine, if somebody shutting down the cluster by accident or beat the idle time ... in this case you organisation lose all of databases and tables. If that's true, does not worth using the databases in databricks but in a other cloud

Re: The database and tables disappears when I delete the cluster

Hubert-Dudek — Thu, 02 Dec 2021 12:20:45 GMT

If it is community edition free edition it can happen as it is only for education purposed,

In normal edition all data is stored on your blob storage so whatever happened it can not be lost, additionally default storage is delta so it is even transactional. Temporary views are also possible to create.

You can send more details about your edition and code so I will be glad to help.

Additionally you can create database or table using specified location. The best practice is to use own storage and mount it to directory so you will have full control.

Re: The database and tables disappears when I delete the cluster

Prabakar — Thu, 02 Dec 2021 13:04:04 GMT

@William Scardua if it's an external hive metastore or Glue catalog you might be missing the configuration on the cluster.

https://docs.databricks.com/data/metastores/index.html

Also as mentioned by @Hubert Dudek , if it's a community edition then this is possible.