Databricks Community

alj_a · ‎11-06-2023

Hi,

Thanks in advance.

I am new in DLT, the scenario is i need to read the data from cloud storage(ADLS) and load it into my bronze table. and read it from bronz table -> do some DQ checks and load the cleaned data into my silver table. finally populate the data into gold table which is dimensioanl model.

i tried it and everyhting is working. but all the tables are getting created and populated in single database in hive metastore.

what im looking for is, i already have three databse(schema) like bronze_db, silver_db and gold_db. but i am stuck in how to specify the database name or schema name in dlt.

#Load only latest file to Bronze table from landing zone
@dlt.create_table(
  comment="The raw leads data, ingested from ADLS landing zone.",
  table_properties={
    "dlt_lead.quality": "bronze",
    "pipelines.autoOptimize.managed": "true"
  }
)
def leads():
    #return (spark.readStream.csv(file_location, header=True))
  return df_bronze_leads