cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

problem with workspace after metastore deleted

sg-vtc
New Contributor III

I am completely new to Databricks AWS and start working on it a week ago.  Pls excuse me if I ask or did something silly.

I created a workspace and a single node cluster for testing. 
A metastore was created from Databricks quickstart and it was automatically assigned to the workspace which I did not realize.  I thought the workspace is using default metastore from dbfs.

I did some basic testing using this database and able to create external delta tables using cvs file from non-aws object storage.  This is my main goal of test to ensure databricks can integrate with non-aws s3 compatible object storage. 

However, from time to time I saw this error:

parent external location for path 's3://databricks-workspace-stack-xxxxxx-metastore-bucket/b11fb1a0-a462-4dfb-b91b-e0795fde10b0’  does not exist. 
From my aws account, I can see the bucket databricks-workspace-stack-xxxxxx-metastore-bucket created by Databricks quickstart with appropriate IAM role and policy.  However, I did not see any folder or object in this bucket. 

I created this b11fb1a0-a462-4dfb-b91b-e0795fde10b0 folder in the S3 metastore bucket but still get the error re the parent external location does not exist. 

Since I did not see object/folder in this bucket, I deleted the metastore from Databricks console.  After that I start having problem with the workspace since it did not have any metastore assigned to it. 

I created another s3 bucket and follows the naming convention and IAM role/policy that was created by Databricks before for databricks-workspace-stack-xxxxxx-metastore-bucket

I created the new Databricks metastore, when assigned it to the workspace that I am using.  From the workspace details, I can see the new metastore path. However, when I run some python commands in this workspace, I always get this error, it is still expecting the previous metastore that was assigned but deleted by me.

ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.

How to resolve the problem.  I prefer not to delete my existing workspace. 

Any help is greatly appreciated. 

 

1 REPLY 1

sg-vtc
New Contributor III

I restarted the compute node and this problem went away.
ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.

New question: I am testing Databricks with non-AWS S3 object storage.  I can access the non-AWS S3 bucket by setting these parameters:

sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", "XXXXXXXXXXXXXXXXXXXX")
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", "XXXXXXXXXXXXXXXXXXXXXXXXXXXX")
sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "XXXXXXXXXXXX.com")

I can read the csv files in the bucket 

spark.read.format("csv").option("inferschema","true").option("header","true").option("sep","|").load("s3://deltalake/10g_csv/reason.csv")
When trying to create external table from this csv, got AWS Security token service invalid error.  Since I am not using AWS s3 bucket, is there a way to skip this checking. 
sgvtc_4-1697652626812.png 
I can see Databricks created parquet file and _delta_log folder in this external bucket location but it did not complete the delta table creation.  It did not create 00000000000000000000.crc and 00000000000000000000.json in the _delta_log folder.

 

sgvtc_0-1697652437393.pngsgvtc_2-1697652510601.png

 

 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!