problem with workspace after metastore deleted

[sg-vtc]

I am completely new to Databricks AWS and start working on it a week ago.  Pls excuse me if I ask or did something silly.

I created a workspace and a single node cluster for testing. 
A metastore was created from Databricks quickstart and it was automatically assigned to the workspace which I did not realize.  I thought the workspace is using default metastore from dbfs.

I did some basic testing using this database and able to create external delta tables using cvs file from non-aws object storage.  This is my main goal of test to ensure databricks can integrate with non-aws s3 compatible object storage. 

However, from time to time I saw this error:

parent external location for path 's3://databricks-workspace-stack-xxxxxx-metastore-bucket/b11fb1a0-a462-4dfb-b91b-e0795fde10b0’  does not exist. 
From my aws account, I can see the bucket databricks-workspace-stack-xxxxxx-metastore-bucket created by Databricks quickstart with appropriate IAM role and policy.  However, I did not see any folder or object in this bucket. 

I created this b11fb1a0-a462-4dfb-b91b-e0795fde10b0 folder in the S3 metastore bucket but still get the error re the parent external location does not exist. 

Since I did not see object/folder in this bucket, I deleted the metastore from Databricks console.  After that I start having problem with the workspace since it did not have any metastore assigned to it. 

I created another s3 bucket and follows the naming convention and IAM role/policy that was created by Databricks before for databricks-workspace-stack-xxxxxx-metastore-bucket

I created the new Databricks metastore, when assigned it to the workspace that I am using.  From the workspace details, I can see the new metastore path. However, when I run some python commands in this workspace, I always get this error, it is still expecting the previous metastore that was assigned but deleted by me.

ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.

How to resolve the problem.  I prefer not to delete my existing workspace. 

Any help is greatly appreciated. 

[sg-vtc]

I restarted the compute node and this problem went away.
ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.

New question: I am testing Databricks with non-AWS S3 object storage.  I can access the non-AWS S3 bucket by setting these parameters:

sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", "XXXXXXXXXXXXXXXXXXXX")
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", "XXXXXXXXXXXXXXXXXXXXXXXXXXXX")
sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "XXXXXXXXXXXX.com")

I can read the csv files in the bucket 

spark.read.format("csv").option("inferschema","true").option("header","true").option("sep","|").load("s3://deltalake/10g_csv/reason.csv")

When trying to create external table from this csv, got AWS Security token service invalid error.  Since I am not using AWS s3 bucket, is there a way to skip this checking. 

 

I can see Databricks created parquet file and _delta_log folder in this external bucket location but it did not complete the delta table creation.  It did not create 00000000000000000000.crc and 00000000000000000000.json in the _delta_log folder.