Hi,
Thanks in advance.
I am new in DLT, the scenario is i need to read the data from cloud storage(ADLS) and load it into my bronze table. and read it from bronz table -> do some DQ checks and load the cleaned data into my silver table. finally populate the data into gold table which is dimensioanl model.
i tried it and everyhting is working. but all the tables are getting created and populated in single database in hive metastore.
what im looking for is, i already have three databse(schema) like bronze_db, silver_db and gold_db. but i am stuck in how to specify the database name or schema name in dlt.
#Load only latest file to Bronze table from landing zone
@dlt.create_table(
comment="The raw leads data, ingested from ADLS landing zone.",
table_properties={
"dlt_lead.quality": "bronze",
"pipelines.autoOptimize.managed": "true"
}
)
def leads():
#return (spark.readStream.csv(file_location, header=True))
return df_bronze_leads
this table should be created under bronze databse/schema
