source db and target db in DLT

alj_a — Tue, 07 Nov 2023 07:06:29 GMT

Hi,

Thanks in advance.

I am new in DLT, the scenario is i need to read the data from cloud storage(ADLS) and load it into my bronze table. and read it from bronz table -> do some DQ checks and load the cleaned data into my silver table. finally populate the data into gold table which is dimensioanl model.

i tried it and everyhting is working. but all the tables are getting created and populated in single database in hive metastore.

what im looking for is, i already have three databse(schema) like bronze_db, silver_db and gold_db. but i am stuck in how to specify the database name or schema name in dlt.

#Load only latest file to Bronze table from landing zone @dlt.create_table( comment="The raw leads data, ingested from ADLS landing zone.", table_properties={ "dlt_lead.quality": "bronze", "pipelines.autoOptimize.managed": "true" } ) def leads(): #return (spark.readStream.csv(file_location, header=True)) return df_bronze_leads

this table should be created under bronze databse/schema

Re: source db and target db in DLT

alj_a — Tue, 07 Nov 2023 11:39:10 GMT

Thank you Kaniz.

I tried and got the error.

"Please remove the database qualifier from table"

topic Re: source db and target db in DLT in Data Engineering

source db and target db in DLT

Re: source db and target db in DLT