Medallion Architecture and Databricks Assistant
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-16-2024 12:28 PM - edited 12-16-2024 12:48 PM
I am in the process of rebuilding the data lake at my current company with databricks and I'm struggling to find comprehensive best practices for naming conventions and structuring medallion architecture to work optimally with the Databricks assistant.
I've been reading about the assistant and what sources it uses to determine what fields it should use in what table etc. Most of the examples I have read shows descriptive table names but without any prefixes or suffixes. The problem is I usually just organize medallion architecture, as well as other things like residency and ingestion source, either using prefixes or suffixes in the table names. For example, bronze_marketing_campaign_response_us_cdc . The documentation I am reading makes it seem like this is not going to be very optimal but I can't seem to find what the 'right' way actually is. Does all of the other information need to happen at the catalog or schema level? Is there something I can do in Unity Catalog to set this up so the assistant can interpret the other information in the table names?