Missing Delta-live-Table in hive-metastore catalog

[BobCat62]

Hi experts,

I defined my delta table in an external location as following:

%sql
CREATE OR REFRESH STREAMING TABLE pumpdata (
Body string,
EnqueuedTimeUtc string,
SystemProperties string,
_rescued_data string,
Properties string
)
USING DELTA
LOCATION 'abfss://mdwh@XXXX.dfs.core.windows.net/Bronze/pumpdata'

I have a delta live table pipeline with theses settings:

As you can see, I have defined the same external location and set hive Metastore as storage option:

and this definition:

import dlt
from pyspark.sql.functions import col
json_path = f"abfss://schachtwasser@XXXX.dfs.core.windows.net/XXXX/*/*/*/*/*.JSON"
@dlt.create_table(
name="pumpdata",
table_properties={
"quality": "raw"
},
comment="Data ingested from an ADLS2 storage account."
)
def pumpdata():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "JSON")
.load(json_path)
)

I can successfully run my DLT and parquet files are put in the storage account, but in the catalog under hive-meta store, I cannot see my table :

[KaranamS]

Hi @BobCat62 , Try these steps 

1. Try manually registering the table in hive_metastore. Run this in a databricks notebook 

CREATE TABLE hive_metastore.default.pumpdata USING DELTA LOCATION 'abfss://mdwh@XXXX.dfs.core.windows.net/Bronze/pumpdata/tables/pumpdata';

2. Then verify the table by running this 

SHOW TABLES IN hive_metastore.default

This should create a table under the hive_metastore

[ashraf1395 · Accepted solution]

Hey @BobCat62 , This might help

dlt will be in direct publishingmode by default. If you select hive_metstore you must specify the default schema in the dlt pipeline setting. If not done there. At the time of defining the dlt table pass the schema_name.pumpdata.

example default.pumpdata. - This will store the table in default schema onf hive_metastore