cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

rami1
by New Contributor II
  • 4795 Views
  • 2 replies
  • 3 kudos

METASTORE_DOWN: Cannot connect to metastore

I am trying to view databases and tables, default as well user created but it looks like the cluster created is not able to connect. I am using databricks default hive metastore. Viewing cluster logs provide following ventMETASTORE_DOWN Metastore is...

  • 4795 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@rami​ :If the metastore is down, it means that the Databricks cluster is not able to connect to the metastore. Here are a few things you can try to resolve the issue:Check if the Hive metastore is up and running. You can try to connect to the metast...

  • 3 kudos
1 More Replies
RC
by Contributor
  • 810 Views
  • 2 replies
  • 2 kudos

Not able to create a unity metastore in a specified region

Hi Team,I'm not able to create a metastore in a region (us-east-1).It tells me that "This region already contains a metastore. Only a single metastore is allowed per region"But we don't have any metastore. Earlier we had one metastore we had deleted...

  • 810 Views
  • 2 replies
  • 2 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 2 kudos

@Rajath C​ can you please re-check if it has been properly deleted and if still old one has been tied to any of workspaces. also try to delete that storage if no data exists and retry

  • 2 kudos
1 More Replies
Tomz
by New Contributor II
  • 1121 Views
  • 2 replies
  • 0 kudos
  • 1121 Views
  • 2 replies
  • 0 kudos
Latest Reply
Tomz
New Contributor II
  • 0 kudos

Hi Debayan,Thank you for answering my question.We were able to successfully attach the 2nd workspace (on a different AWS account) to the existing UC metastore on another AWS account.Unfortunately, we couldn't figure out what we did differently. It se...

  • 0 kudos
1 More Replies
andrew0117
by Contributor
  • 591 Views
  • 1 replies
  • 2 kudos

How to sync the meta store info with the real data for external delta table

if I manually delete some parque files in location which the real data is stored in, so spark catalog still has the old version. How can I sync them?Thanks!

  • 591 Views
  • 1 replies
  • 2 kudos
Latest Reply
youssefmrini
Honored Contributor III
  • 2 kudos

You just need to create a new table and specify the location of the data for your case it's going to be an ADLS, S3...Example​Create table customer using delta location 'mnt/data./'

  • 2 kudos
saurabh12521
by New Contributor II
  • 1390 Views
  • 3 replies
  • 4 kudos

Unity through terraform

I am working on automation of Unity through terraform. I have referred below link link to get started :https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/unity-catalog-azureI am facing issue when I create metastore using...

image
  • 1390 Views
  • 3 replies
  • 4 kudos
Latest Reply
Pat
Honored Contributor III
  • 4 kudos

Not sure if you got this working, but I noticed you are using provider: `databrickslabs/databricks`, hence why this is not avaialable. You should be using new provider: `databricks/databricks`: https://registry.terraform.io/providers/databricks/datab...

  • 4 kudos
2 More Replies
Harsh1
by New Contributor II
  • 551 Views
  • 0 replies
  • 0 kudos

Issues in Metastore Migration using Databricks Migration Tool

Hi Team,As I'm performing the Databricks workspace migration, during Metastore migration I'm facing below issue.As we found differences in the Metastore table count between Legacy and Target workspace, we checked error logs.After going through Failed...

  • 551 Views
  • 0 replies
  • 0 kudos
fundat
by New Contributor II
  • 2108 Views
  • 4 replies
  • 1 kudos

Metastore - Access validation failed

Hi,When i create a metastore in aws databricks, i always have this error in the picture bellow.Eventhought i follow this link https://docs.databricks.com/data-governance/unity-catalog/get-started.html#cloud-tenant-setup-aws

metastore_databricks_error
  • 2108 Views
  • 4 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Anatole Cadet​, We haven't heard from you on the last response from @karthik p​, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others....

  • 1 kudos
3 More Replies
nbakh
by New Contributor II
  • 1701 Views
  • 1 replies
  • 1 kudos

import of tables to a new metastore failed with schema mismatch "specified schema does not match the existing schema"

The error i get when importing certain delta table isThe specified schema does not match the existing schema at dbfs:/mnt/mart/tablenamehowever, when i check the metadata table in the old workspace and the exported file, they match. However, it seems...

  • 1701 Views
  • 1 replies
  • 1 kudos
Latest Reply
nbakh
New Contributor II
  • 1 kudos

below is example error. however, in existing metadata i still see varchar 100 as the type. Specified metadata for field Percentage is different from existing schema:\n Specified: {}\n Existing: {\"HIVE_TYPE_STRING\":\"varchar(100)\"}\n\nIf your inte...

  • 1 kudos
AP
by New Contributor III
  • 1467 Views
  • 2 replies
  • 2 kudos

How can we connect to the databricks managed metastore

Hi, I am trying to take advantage of the treasure trove of the information that metastore contains and take some actions to improve performance. In my case, the metastore is managed by databricks, we don't use external metastore.How can I connect to ...

  • 1467 Views
  • 2 replies
  • 2 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 2 kudos

@AKSHAY PALLERLA​ to get the jdbc/odbc information you can get it from the cluster configuration. In the cluster configuration page, under advanced options, you have JDBC/ODBC tab. Click on that tab and it should give you the details you are looking ...

  • 2 kudos
1 More Replies
AlexDavies
by Contributor
  • 5231 Views
  • 9 replies
  • 2 kudos

Resolved! How to upgrade internal hive metadata store version

Is it possible to upgrade the out of the box hive metastore version? running spark.conf.get("spark.sql.hive.metastore.version") indicates that it is running on 0.13.0 However https://docs.microsoft.com/en-us/azure/databricks/release-notes/runtime/7....

  • 5231 Views
  • 9 replies
  • 2 kudos
Latest Reply
pantelis_mare
Contributor III
  • 2 kudos

Hello guys!Atanu's post, although correct does not solve the problem. Is there any official documentation on how to upgrade the internal databricks metastore to a greater version? If this is availble then we can try Atanu's solution (not sure if need...

  • 2 kudos
8 More Replies
LukaszJ
by Contributor III
  • 2630 Views
  • 6 replies
  • 4 kudos

Resolved! Terraform: get metastore id without creating new metastore

Hello,I want to create database (schema) and tables in my Databricks workspace using terraform.I found this resources: databricks_schemaIt requires databricks_catalog, which requires metastore_id.However, I have databricks_workspace and I did not cre...

  • 2630 Views
  • 6 replies
  • 4 kudos
Latest Reply
Kaniz
Community Manager
  • 4 kudos

Hi @Łukasz Jaremek​ , Just a friendly follow-up. Do you still need help, or @Hubert Dudek (Customer)​ and @Atanu Sarkar​ 's response help you to find the solution? Please let us know.

  • 4 kudos
5 More Replies
alonisser
by Contributor
  • 1814 Views
  • 3 replies
  • 4 kudos

Resolved! How to migrate an existing workspace for an external metastore

Currently we're on an azure databricks workspace, we've setup during the POC, a long time ago. In the meanwhile we have built quite a production workload above databricks.Now we want to split workspaces - one for analysts and one for data engineeri...

  • 1814 Views
  • 3 replies
  • 4 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

From databricks notebook just run mysqldump. Server address and details you can take from logs or configuration.I am including also link to example notebook https://docs.microsoft.com/en-us/azure/databricks/kb/_static/notebooks/2016-election-tweets.h...

  • 4 kudos
2 More Replies
Oliver_Floyd
by Contributor
  • 1219 Views
  • 3 replies
  • 4 kudos

Resolved! How to update external metastore cluster configuration on the fly ?

Hello,In my use case, my data is pushed to an adls gen2 container called ingestAfter some data processing on a databricks cluster of the ingest workspace, I declare the associated table in an external metastore for this workspaceAt the end of this pr...

  • 1219 Views
  • 3 replies
  • 4 kudos
Latest Reply
Oliver_Floyd
Contributor
  • 4 kudos

Hello @Atanu Sarkar​ ,Thank you for your answer. I have created a feature request. I hope, it will be soon accepted ^^

  • 4 kudos
2 More Replies
prasadvaze
by Valued Contributor
  • 838 Views
  • 1 replies
  • 1 kudos

which table in external hive metastore stores the folder path for delta table's data?

which table in external hive metastore stores the folder path for delta table's data? Is it SDS table?

  • 838 Views
  • 1 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Image here can be helpful https://analyticsanvil.files.wordpress.com/2016/08/hive_metastore_database_diagram.png

  • 1 kudos
Anonymous
by Not applicable
  • 1692 Views
  • 1 replies
  • 2 kudos

Are there any costs or quotas associated with the Databricks managed Hive metastore?

When using the default hive metastore that is managed within the Databricks control plane are there any associated costs? I.e. if I switched to an external metastore would I expect to see any reduction in my Databricks cost (ignoring total costs).Do ...

  • 1692 Views
  • 1 replies
  • 2 kudos
Latest Reply
Ryan_Chynoweth
Honored Contributor III
  • 2 kudos

There are no costs associated by using the Databricks managed Hive metastore directly. Databricks pricing is on a compute consumption and not on data storage or access. The only real cost would be the compute used to access the data. I would not expe...

  • 2 kudos
Labels