Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Without streaming everything works fine. When I try to set up some example structured streaming in my company's new Databricks workspace (we're trying it out) I can't seem to figure out the correct way to save it/write to storage. Please see the pict...
Hi, This can be an issue where Storage Blob Data contributor role is missing. Please refer: https://docs.databricks.com/external-data/azure-storage.html
Hi,I've got Unity Catalog working but i cant create an external Table.We have several Workspaces (with customer managed VPCs), all with Private Link connected to the Control Plane. Our Data S3 Buckets are secured via Bucket Policy (in addition to KM...
Hi, Could you please confirm if you have followed this: https://docs.databricks.com/data-governance/unity-catalog/get-started.html , also, please confirm the exact error received?
Hi there,I am facing an issue when setting up the metastore.I use an account admin user, and the user is part of a group called Unity admin.The Unity admin group contains 2 users: me and another user, also account admin.Without Unity catalog, for now...
Hi, Could you please refer https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#transfer-ownership, please let us know if this helps.
Checking the docs I noticed this statement under Azure storage access page:[https://learn.microsoft.com/en-us/azure/databricks/external-data/azure-storage]Deprecated patterns for storing and accessing data from Azure DatabricksThe following are depre...
You don't need to mount to dbfs. The security issue with mounting is that as soon as you've mounted using a credential, everyone with access to the workspace now has access to the data at that mounted location.The recommended way that isn't mounting ...
Unity Catalog and Delta Sharing Workshop - Register Now!Join us on February 2nd, 2023 as we take Unity Catalog for a spin. Register Here!!Unity Catalog provides a unified governance solution for data, analytics and AI on any cloud. With Unity Catalog...
Unity Catalog Understanding In DatabricksThe Unity Catalog is a feature of the Databricks platform that allows users to organize and discover data assets within their organization. It provides a central repository for storing and managing data assets...
Understanding DLT pipe lines in DatabricksIn Databricks, a DLT (Data Live Table) pipeline is a set of data transformations that are applied to data assets in a defined sequence, in order to clean, enrich, and prepare data for analysis or other purpos...
Hi Guys,I'd like to know how you have adopted the Unity Catalog. To me it seems rather limited in terms of that you are only allowed to have one UC per region. We are building out a data warehouse and we have three workspaces - dev, test and prod. Th...
Hi @Carsten Klausman​ ,How is your journey with the Unity Catalog?I am seeing more and more limitations recently. It's pity, we had this freedom to work with Spark - ACID and optimized.We are being forced to do things with the Unity Catalog, and seas...
We are attempting to setup Unity Catalog and our security team is requesting justification on why this level of access is required. Why must the first Azure Databricks account admin must be an Azure Active Directory Global Administrator at the time t...
Hi @Matthew Dalesio​ From our eng. team:"The high privileged is only used to make sure only highly privileged users get access to Databricks account admin role as this is a highly-privileged role and they can make anyone else an account admin. This i...
Now that Unity Catalog is GA, I was wondering how y'all implemented it (or planning to do so) on a workspace level.Do you create a separate 'one workspace to rule them all' with the Unity Catalog, or do you create it on an already existing workspace ...
Are you thinking about creating totally separate workspace just to manage Unity Catalog?I've been playing a bit with the Unity Catalog and I am currently thinking about managing it from one of the existing workspaces. I don't think there is need for ...
Is any one came across below issue, based on azure external table upgrade article. what ever tables are being supported in unity catalog they can get upgraded. but other than below "UNSUPPORTED_FILE_SCHEME" Indication, we are not seeing much informat...
Hi @karthik p​ you can raise a support request to databricks they will let you know the limitation of this.You can use the below link for general QnA-Link
Hi all, I've gone through the deployment guide to set up Unity catalog as outlined in https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/get-started.I tried creating a table once I had it set up with the below command.CR...
Legend! This solved it for me. I kept checking permissioning issues but I missed a key step in setting up the storage account in the following guide: https://learn.microsoft.com/en-us/azure/storage/blobs/create-data-lake-storage-account.
I have images, zipped files, and arbitrary binary files in ADLS gen 2. In Unity Catalog I created a storage credential and an external location to the container that contains these files in directories.Is there any way I can view them within Unity Ca...
Hi, it is a known bug that External Locations do not work for Azure Storage Account root locations. Please create a folder in the root location and use that as the external location.With the External Location and its permissions properly set up, you ...
HI Team,once after implementing unity catalog and start migrating external tables from legacy hive to unity catalog, i am seeing in articles that we need to make changes to our workloads to be in sync with 3 level name spaceeg: if i have 50 notebooks...
@Aviral Bhardwaj​ this will be normal priority and it takes lot of time to get response.team, we are looking for downtime possibilities and how to handle or change workflows or notebooks to use unity catalog 3 level namespace. do we have any utility ...
HiWe have faced an issue recently (everything worked well 1 week before)We are using exnernal hive metastore:"spark.hadoop.datanucleus.connectionPool.maxPoolSize" : "10" "spark.sql.hive.metastore.jars" : "/dbfs/ivyjars/*" "spark.sql.hive.metastor...
Generally, this should not be a case. Please raise a support request to databricks
