Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Is there a command to hide the details of DLT/materialized views from users in UC?
Hi @dataminion01,As far as I know, Unity Catalog doesn't currently provide a direct way to hide DLT or materialized view definitions. Users with SELECT access can still view the logic via the "Details" tab.You can work around this by:Moving sensitive...
Hi there,I was checking though the Databricks Runtime 17.0 features while testing:Support ALL CATALOGS in SHOW SCHEMASI faced below issues, is that anticipated or an issue which needs to be fixed?This is how it was before Runtime 17.0 I was expectin...
I was able to reproduce same issue, I am working with the team, as this DBR is in Beta version we might see some changes on the behavior, so I will confirm if anything has changed around this topic
We have implemented our own delta sharing server, following the delta sharing protocol. We are able to use the delta-sharing connectors to read shared data from our own server. However, when we tried importing our share credential file in Databricks ...
Fixed
Hi,If I update a Tag on a table within a Noetbook as follows:sql = f"""ALTER TABLE {silver_catalog}.public.tablename SET TAGS ('Is_published' = 'Y');"""spark.sql(sql)Then the User ID and Username, nor Notebook is populated when I look in the History ...
Hi team,You should be able to see the change in the History tab when changes are made manually through the UI or changes made through APIs that explicitly push user metadata. Programmatic Spark SQL updates may not automatically attach full interactiv...
I have created a catalog, a schema and several tables within unity catalog, when I run a notebook that makes a merge to one of the tables the lineage of this is correct with all its dependencies, but if I run the notebook of another different table t...
Hello @andalo! This might be happening because lineage for MERGE operations is not fully captured by default. Could you try setting the Spark property spark.databricks.dataLineage.mergeIntoV2Enabled to true?Just a heads-up, enabling this setting may ...
Hello All,We have recently created a databricks with UC enabled. All the VNET Pairing is completed. Now, while I am trying to create a table [Managed] under a catalog / schema - I am not able to query the table. I am getting below error org.apache.ha...
Hi @dipanjannet , This could be due to the code falling back to the hive metastore, which could be due to a number of reasons, but the first thing that caught my eye was that the table name in your saveAsTable() call isn't fully qualified. Can you ad...
Hi,I have tried to execute Vectorized udfs in SQL editor of Databricks, I am getting the following error[FEATURE_UNAVAILABLE] Pandas UDF is not supported in your environment. To use this feature, please contact Databricks Support. SQLSTATE: 56038Coul...
Hi Avinash, I believe this is not supported on Sql editor yet. However, it is available in DBR rollout 16.3.1. Hope this helps!
According to the docs, we should be able to use SYNC TABLE to 'upgrade' a table from a Hive metastore to Unity Catalog. We are using AWS Glue as our Hive Metastore, but tables created in Glue do not seem to be set up in a format that Databricks likes...
Hi team,The issue arises because Unity Catalog's SYNC command cannot process tables that were created using the Hive SerDe format in the hive_metastore catalog. This means that any attempts to upgrade such tables will fail. You need to convert your H...
Is it possible to update the schema/change the type of individual feature in a feature store?
Hi Henrik, Please check this documentation and let me know if it answers your question -> https://docs.databricks.com/gcp/en/machine-learning/feature-store/uc/feature-tables-uc#update-a-feature-table-in-unity-catalog Thank you!
Hi there,When creating an external table in Unity Catalog using an existing Delta Lake table with a comment on the table itself, the comment isn't imported to the `Comment` key's value in the Unity Catalog table.Could you explain why?
Hi team, Have you tried using SQL command to update the column comment?ALTER TABLE <table_name> ALTER COLUMN <column_name> COMMENT '<comment_text>';I believe this should resolve the issue
Unity Catalog (UC) tracks the metadata and your cloud storage accounts store the your data. This python script will extract the metadata from {catalog}.information_schema into folders in a storage location. Take this and put into a notebook. Make da...
Hi All, Can anyone point me to either documentation or personally tried and tested method of backing up (and restoring) Unity Catalog and its associated managed tables? We're running on Azure and using ADLS Gen2.Regards,Ashley
Unity Catalog (UC) tracks the metadata and your cloud storage accounts store the your data. This python script will extract the metadata from {catalog}.information_schema into folders in a storage location. Take this and put into a notebook.Make data...
Hi,I am working for a large company that is implementing a Databricks solution. We have multiple domains, each responsible for its own data products, following a data mesh approach.As part of a federated governance model, we need a way to communicate...
If a model is registered in Unity Catalog, then what all APIs and SDKs could be used to build the entire lineage for a model? I'm trying to figure out what all do I query to ensure I don't miss any element of the model lineage.Now a model can have be...
@rashij You're tackling an important challenge in ML governance. Building a comprehensive lineage for models registered in Unity Catalog requires piecing together information from multiple APIs. Let me outline a more complete approach based on your i...
I am trying to set up Unity Catalog in Azure Databricks following this documentation Unity Catalog, but I do not find "Create Metastore" and "Create Metastore" on Databricks.Please help with the same, how can I enable Unity Catalog.
This issue is resolved. Thank you @Camilla Lok​ and @Harun for the suggestions. It works fine after enabling Global administrators in Azure Active Directory .
