- 8490 Views
- 2 replies
- 4 kudos
METASTORE_DOWN: Cannot connect to metastore
I am trying to view databases and tables, default as well user created but it looks like the cluster created is not able to connect. I am using databricks default hive metastore. Viewing cluster logs provide following ventMETASTORE_DOWN Metastore is...
- 8490 Views
- 2 replies
- 4 kudos
- 4 kudos
@rami​ :If the metastore is down, it means that the Databricks cluster is not able to connect to the metastore. Here are a few things you can try to resolve the issue:Check if the Hive metastore is up and running. You can try to connect to the metast...
- 4 kudos
- 1499 Views
- 2 replies
- 2 kudos
Not able to create a unity metastore in a specified region
Hi Team,I'm not able to create a metastore in a region (us-east-1).It tells me that "This region already contains a metastore. Only a single metastore is allowed per region"But we don't have any metastore. Earlier we had one metastore we had deleted...
- 1499 Views
- 2 replies
- 2 kudos
- 2 kudos
@Rajath C​ can you please re-check if it has been properly deleted and if still old one has been tied to any of workspaces. also try to delete that storage if no data exists and retry
- 2 kudos
- 1918 Views
- 2 replies
- 0 kudos
- 1918 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Debayan,Thank you for answering my question.We were able to successfully attach the 2nd workspace (on a different AWS account) to the existing UC metastore on another AWS account.Unfortunately, we couldn't figure out what we did differently. It se...
- 0 kudos
- 1088 Views
- 1 replies
- 2 kudos
How to sync the meta store info with the real data for external delta table
if I manually delete some parque files in location which the real data is stored in, so spark catalog still has the old version. How can I sync them?Thanks!
- 1088 Views
- 1 replies
- 2 kudos
- 2 kudos
You just need to create a new table and specify the location of the data for your case it's going to be an ADLS, S3...Example​Create table customer using delta location 'mnt/data./'
- 2 kudos
- 2878 Views
- 3 replies
- 4 kudos
Unity through terraform
I am working on automation of Unity through terraform. I have referred below link link to get started :https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/unity-catalog-azureI am facing issue when I create metastore using...
- 2878 Views
- 3 replies
- 4 kudos
- 4 kudos
Not sure if you got this working, but I noticed you are using provider: `databrickslabs/databricks`, hence why this is not avaialable. You should be using new provider: `databricks/databricks`: https://registry.terraform.io/providers/databricks/datab...
- 4 kudos
- 1029 Views
- 0 replies
- 0 kudos
Issues in Metastore Migration using Databricks Migration Tool
Hi Team,As I'm performing the Databricks workspace migration, during Metastore migration I'm facing below issue.As we found differences in the Metastore table count between Legacy and Target workspace, we checked error logs.After going through Failed...
- 1029 Views
- 0 replies
- 0 kudos
- 6539 Views
- 3 replies
- 1 kudos
Metastore - Access validation failed
Hi,When i create a metastore in aws databricks, i always have this error in the picture bellow.Eventhought i follow this link https://docs.databricks.com/data-governance/unity-catalog/get-started.html#cloud-tenant-setup-aws
- 6539 Views
- 3 replies
- 1 kudos
- 1 kudos
@Anatole Cadet​ looks your IAM role is not properly configured, can you please check
- 1 kudos
- 3351 Views
- 1 replies
- 1 kudos
import of tables to a new metastore failed with schema mismatch "specified schema does not match the existing schema"
The error i get when importing certain delta table isThe specified schema does not match the existing schema at dbfs:/mnt/mart/tablenamehowever, when i check the metadata table in the old workspace and the exported file, they match. However, it seems...
- 3351 Views
- 1 replies
- 1 kudos
- 1 kudos
below is example error. however, in existing metadata i still see varchar 100 as the type. Specified metadata for field Percentage is different from existing schema:\n Specified: {}\n Existing: {\"HIVE_TYPE_STRING\":\"varchar(100)\"}\n\nIf your inte...
- 1 kudos
- 2626 Views
- 2 replies
- 2 kudos
How can we connect to the databricks managed metastore
Hi, I am trying to take advantage of the treasure trove of the information that metastore contains and take some actions to improve performance. In my case, the metastore is managed by databricks, we don't use external metastore.How can I connect to ...
- 2626 Views
- 2 replies
- 2 kudos
- 2 kudos
@AKSHAY PALLERLA​ to get the jdbc/odbc information you can get it from the cluster configuration. In the cluster configuration page, under advanced options, you have JDBC/ODBC tab. Click on that tab and it should give you the details you are looking ...
- 2 kudos
- 8965 Views
- 7 replies
- 2 kudos
Resolved! How to upgrade internal hive metadata store version
Is it possible to upgrade the out of the box hive metastore version? running spark.conf.get("spark.sql.hive.metastore.version") indicates that it is running on 0.13.0 However https://docs.microsoft.com/en-us/azure/databricks/release-notes/runtime/7....
- 8965 Views
- 7 replies
- 2 kudos
- 2 kudos
Hello guys!Atanu's post, although correct does not solve the problem. Is there any official documentation on how to upgrade the internal databricks metastore to a greater version? If this is availble then we can try Atanu's solution (not sure if need...
- 2 kudos
- 4824 Views
- 4 replies
- 4 kudos
Resolved! Terraform: get metastore id without creating new metastore
Hello,I want to create database (schema) and tables in my Databricks workspace using terraform.I found this resources: databricks_schemaIt requires databricks_catalog, which requires metastore_id.However, I have databricks_workspace and I did not cre...
- 4824 Views
- 4 replies
- 4 kudos
- 4 kudos
https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/schema I think this is for UC. https://docs.databricks.com/data-governance/unity-catalog/index.html
- 4 kudos
- 2806 Views
- 3 replies
- 4 kudos
Resolved! How to migrate an existing workspace for an external metastore
Currently we're on an azure databricks workspace, we've setup during the POC, a long time ago. In the meanwhile we have built quite a production workload above databricks.Now we want to split workspaces - one for analysts and one for data engineeri...
- 2806 Views
- 3 replies
- 4 kudos
- 4 kudos
From databricks notebook just run mysqldump. Server address and details you can take from logs or configuration.I am including also link to example notebook https://docs.microsoft.com/en-us/azure/databricks/kb/_static/notebooks/2016-election-tweets.h...
- 4 kudos
- 2258 Views
- 2 replies
- 3 kudos
Resolved! How to update external metastore cluster configuration on the fly ?
Hello,In my use case, my data is pushed to an adls gen2 container called ingestAfter some data processing on a databricks cluster of the ingest workspace, I declare the associated table in an external metastore for this workspaceAt the end of this pr...
- 2258 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @Atanu Sarkar​ ,Thank you for your answer. I have created a feature request. I hope, it will be soon accepted ^^
- 3 kudos
- 1346 Views
- 1 replies
- 1 kudos
which table in external hive metastore stores the folder path for delta table's data?
which table in external hive metastore stores the folder path for delta table's data? Is it SDS table?
- 1346 Views
- 1 replies
- 1 kudos
- 1 kudos
Image here can be helpful https://analyticsanvil.files.wordpress.com/2016/08/hive_metastore_database_diagram.png
- 1 kudos
- 2853 Views
- 1 replies
- 2 kudos
Are there any costs or quotas associated with the Databricks managed Hive metastore?
When using the default hive metastore that is managed within the Databricks control plane are there any associated costs? I.e. if I switched to an external metastore would I expect to see any reduction in my Databricks cost (ignoring total costs).Do ...
- 2853 Views
- 1 replies
- 2 kudos
- 2 kudos
There are no costs associated by using the Databricks managed Hive metastore directly. Databricks pricing is on a compute consumption and not on data storage or access. The only real cost would be the compute used to access the data. I would not expe...
- 2 kudos