topic Missing Delta-live-Table in hive-metastore catalog in Data Engineering

Missing Delta-live-Table in hive-metastore catalog

BobCat62 — Wed, 19 Feb 2025 15:35:01 GMT

Hi experts,

I defined my delta table in an external location as following:

%sql
CREATE OR REFRESH STREAMING TABLE pumpdata (
Body string,
EnqueuedTimeUtc string,
SystemProperties string,
_rescued_data string,
Properties string
)
USING DELTA
LOCATION 'abfss://mdwh@XXXX.dfs.core.windows.net/Bronze/pumpdata'

I have a delta live table pipeline with theses settings:

As you can see, I have defined the same external location and set hive Metastore as storage option:

and this definition:

import dlt
from pyspark.sql.functions import col
json_path = f"abfss://schachtwasser@XXXX.dfs.core.windows.net/XXXX/*/*/*/*/*.JSON"
@dlt.create_table(
name="pumpdata",
table_properties={
"quality": "raw"
},
comment="Data ingested from an ADLS2 storage account."
)
def pumpdata():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "JSON")
.load(json_path)
)

I can successfully run my DLT and parquet files are put in the storage account, but in the catalog under hive-meta store, I cannot see my table :

Re: Missing Delta-live-Table in hive-metastore catalog

KaranamS — Fri, 21 Feb 2025 21:37:37 GMT

Hi @BobCat62 , Try these steps

1. Try manually registering the table in hive_metastore. Run this in a databricks notebook

CREATE TABLE hive_metastore.default.pumpdata USING DELTA LOCATION 'abfss://mdwh@XXXX.dfs.core.windows.net/Bronze/pumpdata/tables/pumpdata';

2. Then verify the table by running this

SHOW TABLES IN hive_metastore.default

This should create a table under the hive_metastore

Re: Missing Delta-live-Table in hive-metastore catalog

ashraf1395 — Sat, 22 Feb 2025 05:59:37 GMT

Hey @BobCat62 , This might help

dlt will be in direct publishingmode by default. If you select hive_metstore you must specify the default schema in the dlt pipeline setting. If not done there. At the time of defining the dlt table pass the schema_name.pumpdata.

example default.pumpdata. - This will store the table in default schema onf hive_metastore