Purpose of DLT Table table_properties > quality:medallion

[ChristianRRL]

Hi there, silly question here but can anyone help me understand what practical purpose does labelling the table_properties with "quality":"<specific_medallion>"? For example:

@Dlt.table(
  comment="Bronze live streaming table for Test data",
  name="bronze_test_table",
  table_properties={
    "quality": "bronze"
  }
)

I've seen examples where quality is set to bronze/silver/gold, but what can this be used for?

[Priyanka_Biswas]

Hi @ChristianRRL Thank you for reaching out. 

Labeling table properties with "quality": "<specific_medallion>" (like bronze, silver, gold) in DLT serves a practical purpose in a data architecture pattern known as the "Medallion Architecture."

This architecture typically involves three layers of tables - Bronze, Silver, and Gold - each representing a different stage of data processing and refinement.

Bronze Tables: These tables are raw data ingests.

Silver Tables: Data in silver tables has undergone some level of cleaning, transformation, and enrichment. It's more structured and usable than bronze data but may not be fully business-ready.

Gold Tables: These tables contain business-level, aggregated, or summarized data. This data is often the most refined and is used for high-level reporting and analytics

By knowing the "quality" of the table, data engineers and scientists can quickly identify the maturity level of the data and its readiness for various use cases.

[ChristianRRL]

Hi @Priyanka_Biswas, I understand the Medallion Architecture and I'm all for it.. I guess my question is more specifically meaning as to in this specific case where we specify the table_properties > quality, how can this be practically used in a real world scenario?

For example, is there some kind of metadata table that I'm not aware of that I can query and filter by table quality? This alone would be more than enough of a reason for me to label my tables with their respective quality indicator. And if this functionality exists, would it work with the Hive Metastore or does it require Unity Catalog?

[Kaniz]

@ChristianRRL, Welcome, and thank you for being a part of our community! 

We highly value your inquiries, and we applaud your active participation. We kindly invite you to take some time to review the responses provided and select the solution that best suits your needs. Not only will you gain access to the most pertinent information, but you will also be contributing to the wealth of knowledge within the community. 

To show our gratitude, we will be awarding a special badge to users who consistently engage in selecting solutions. 

Thank you for being an integral part of our community, and please do not hesitate to reach out if you require any further assistance!

[ChristianRRL]

Hi @Kaniz , so far this has been a very nice and engaging community that has helped me answer some of my questions. I try and give Kudos often when appropriate, and I have selected some responses to my posts as valid "Solutions", but I may not have gotten a reasonable solution response to all of my posts so sometimes I'm waiting to see if someone will follow-up with another answer that is a better fit.

I can go back and review some of my posts to check if maybe I missed some solutions that are valid. Additionally, can posts have more than one valid solution (if multiple people answer)?