cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Purpose of DLT Table table_properties > quality:medallion

ChristianRRL
Contributor III

Hi there, silly question here but can anyone help me understand what practical purpose does labelling the table_properties with "quality":"<specific_medallion>"? For example:

 

@Dlt.table(
  comment="Bronze live streaming table for Test data",
  name="bronze_test_table",
  table_properties={
    "quality": "bronze"
  }
)

 

I've seen examples where quality is set to bronze/silver/gold, but what can this be used for?

3 REPLIES 3

Priyanka_Biswas
Databricks Employee
Databricks Employee

Hi @ChristianRRL Thank you for reaching out. 

Labeling table properties with "quality": "<specific_medallion>" (like bronze, silver, gold) in DLT serves a practical purpose in a data architecture pattern known as the "Medallion Architecture."

This architecture typically involves three layers of tables - Bronze, Silver, and Gold - each representing a different stage of data processing and refinement.

Bronze Tables: These tables are raw data ingests.

Silver Tables: Data in silver tables has undergone some level of cleaning, transformation, and enrichment. It's more structured and usable than bronze data but may not be fully business-ready.

Gold Tables: These tables contain business-level, aggregated, or summarized data. This data is often the most refined and is used for high-level reporting and analytics

By knowing the "quality" of the table, data engineers and scientists can quickly identify the maturity level of the data and its readiness for various use cases.

Hi @Priyanka_Biswas, I understand the Medallion Architecture and I'm all for it.. I guess my question is more specifically meaning as to in this specific case where we specify the table_properties > quality, how can this be practically used in a real world scenario?

For example, is there some kind of metadata table that I'm not aware of that I can query and filter by table quality? This alone would be more than enough of a reason for me to label my tables with their respective quality indicator. And if this functionality exists, would it work with the Hive Metastore or does it require Unity Catalog?

Hi @Retired_mod , so far this has been a very nice and engaging community that has helped me answer some of my questions. I try and give Kudos often when appropriate, and I have selected some responses to my posts as valid "Solutions", but I may not have gotten a reasonable solution response to all of my posts so sometimes I'm waiting to see if someone will follow-up with another answer that is a better fit.

I can go back and review some of my posts to check if maybe I missed some solutions that are valid. Additionally, can posts have more than one valid solution (if multiple people answer)?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group