topic Re: Purpose of DLT Table table_properties > quality:medallion in Get Started Discussions

Purpose of DLT Table table_properties > quality:medallion

ChristianRRL — Wed, 24 Jan 2024 21:10:20 GMT

Hi there, silly question here but can anyone help me understand what practical purpose does labelling the table_properties with "quality":"<specific_medallion>"? For example:

@Dlt.table( comment="Bronze live streaming table for Test data", name="bronze_test_table", table_properties={ "quality": "bronze" } )

I've seen examples where quality is set to bronze/silver/gold, but what can this be used for?

Re: Purpose of DLT Table table_properties > quality:medallion

Priyanka_Biswas — Thu, 25 Jan 2024 00:18:36 GMT

Hi @ChristianRRL Thank you for reaching out.

Labeling table properties with "quality": "<specific_medallion>" (like bronze, silver, gold) in DLT serves a practical purpose in a data architecture pattern known as the "Medallion Architecture."

This architecture typically involves three layers of tables - Bronze, Silver, and Gold - each representing a different stage of data processing and refinement.

Bronze Tables: These tables are raw data ingests.

Silver Tables: Data in silver tables has undergone some level of cleaning, transformation, and enrichment. It's more structured and usable than bronze data but may not be fully business-ready.

Gold Tables: These tables contain business-level, aggregated, or summarized data. This data is often the most refined and is used for high-level reporting and analytics

By knowing the "quality" of the table, data engineers and scientists can quickly identify the maturity level of the data and its readiness for various use cases.

Re: Purpose of DLT Table table_properties > quality:medallion

ChristianRRL — Thu, 25 Jan 2024 16:28:16 GMT

Hi @Priyanka_Biswas, I understand the Medallion Architecture and I'm all for it.. I guess my question is more specifically meaning as to in this specific case where we specify the table_properties > quality, how can this be practically used in a real world scenario?

For example, is there some kind of metadata table that I'm not aware of that I can query and filter by table quality? This alone would be more than enough of a reason for me to label my tables with their respective quality indicator. And if this functionality exists, would it work with the Hive Metastore or does it require Unity Catalog?

Re: Purpose of DLT Table table_properties > quality:medallion

ChristianRRL — Thu, 25 Jan 2024 16:32:18 GMT

Hi @Retired_mod , so far this has been a very nice and engaging community that has helped me answer some of my questions. I try and give Kudos often when appropriate, and I have selected some responses to my posts as valid "Solutions", but I may not have gotten a reasonable solution response to all of my posts so sometimes I'm waiting to see if someone will follow-up with another answer that is a better fit.

I can go back and review some of my posts to check if maybe I missed some solutions that are valid. Additionally, can posts have more than one valid solution (if multiple people answer)?

Re: Purpose of DLT Table table_properties > quality:medallion

nochimo — Wed, 15 Jan 2025 20:50:51 GMT

I'm with the same doubt @ChristianRRL, did you figured out something related to it?
My doubt is to check if it's possible to apply any kind of access control based on this property.