Purpose of DLT Table table_properties > quality:medallion

[ChristianRRL]

Hi there, silly question here but can anyone help me understand what practical purpose does labelling the table_properties with "quality":"<specific_medallion>"? For example:

@Dlt.table(
  comment="Bronze live streaming table for Test data",
  name="bronze_test_table",
  table_properties={
    "quality": "bronze"
  }
)

I've seen examples where quality is set to bronze/silver/gold, but what can this be used for?

[Priyanka_Biswas]

Hi @ChristianRRL Thank you for reaching out. 

Labeling table properties with "quality": "<specific_medallion>" (like bronze, silver, gold) in DLT serves a practical purpose in a data architecture pattern known as the "Medallion Architecture."

This architecture typically involves three layers of tables - Bronze, Silver, and Gold - each representing a different stage of data processing and refinement.

Bronze Tables: These tables are raw data ingests.

Silver Tables: Data in silver tables has undergone some level of cleaning, transformation, and enrichment. It's more structured and usable than bronze data but may not be fully business-ready.

Gold Tables: These tables contain business-level, aggregated, or summarized data. This data is often the most refined and is used for high-level reporting and analytics

By knowing the "quality" of the table, data engineers and scientists can quickly identify the maturity level of the data and its readiness for various use cases.

[ChristianRRL]

Hi @Priyanka_Biswas, I understand the Medallion Architecture and I'm all for it.. I guess my question is more specifically meaning as to in this specific case where we specify the table_properties > quality, how can this be practically used in a real world scenario?

For example, is there some kind of metadata table that I'm not aware of that I can query and filter by table quality? This alone would be more than enough of a reason for me to label my tables with their respective quality indicator. And if this functionality exists, would it work with the Hive Metastore or does it require Unity Catalog?

[ChristianRRL]

Hi @Retired_mod , so far this has been a very nice and engaging community that has helped me answer some of my questions. I try and give Kudos often when appropriate, and I have selected some responses to my posts as valid "Solutions", but I may not have gotten a reasonable solution response to all of my posts so sometimes I'm waiting to see if someone will follow-up with another answer that is a better fit.

I can go back and review some of my posts to check if maybe I missed some solutions that are valid. Additionally, can posts have more than one valid solution (if multiple people answer)?