cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

AdnanFaridKhan
by • New Contributor II
  • 414 Views
  • 4 replies
  • 1 kudos

Masking Columns for a Client Workspace When You Can’t Use Catalog Binding

Hi Everyone,I am Adnan and I am a Cloudera Platform expert recently joined the Databricks Community. I will be sharing all my implementations and experience I have gained as Big Data Expert and any solutions I implement to address my current business...

  • 414 Views
  • 4 replies
  • 1 kudos
Latest Reply
suranga
Databricks Partner
  • 1 kudos

Hi @rachelwilson,I'd do both, because they catch different failures.In the sync job, as the last step. It runs right after every sync, so a tag that failed to copy is caught in the same run, before anyone queries the table.A separate sweep on its own...

  • 1 kudos
3 More Replies
steff_horemans
by • New Contributor
  • 526 Views
  • 1 replies
  • 0 kudos

First time community member working on a project

Hi everyone,Don't know where to put this specific question. I'm working on a reference data mesh implementation to connect and combine datasets to find matching trials for patients with a specific genetic profile. - Do you know anyone that might be i...

  • 526 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @steff_horemans, My guess is that the Free Edition is probably not the best place to demonstrate true external sharing. Free Edition is positioned as a serverless-only, quota-limited, non-commercial environment, with one workspace, one metastore, ...

  • 0 kudos
APJESK
by • Contributor
  • 2804 Views
  • 3 replies
  • 0 kudos

Resolved! Unity catalog management

How is Unity Catalog managed in real time at an enterprise scale, including workspace-level restrictions, privilege-based ACLs, row- and column-level security, ABAC, and tag-driven governance, and which languages or tools are used to manage the entir...

  • 2804 Views
  • 3 replies
  • 0 kudos
Latest Reply
APJESK
Contributor
  • 0 kudos

Thank you... Which roles ( workspace admin or Metastore Admin) are recommended to securely create and manage Unity Catalog objects (Storage Credentials, External Locations, Catalogs, Schemas, and Delta Sharing)?” and why ?

  • 0 kudos
2 More Replies
cobba16
by • New Contributor II
  • 3284 Views
  • 2 replies
  • 1 kudos

Hive metastore CANNOT access storage

I'm new to Azure Databricks and I'm facing an issue when trying to create a schema or table that points to my Azure Storage account. I keep getting this error:```[EXTERNAL_METASTORE_CLIENT_ERROR.OPERATION_FAILED] Client operation failed: org.apache.h...

  • 3284 Views
  • 2 replies
  • 1 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 1 kudos

Hi @cobba16, This is a common scenario on Azure Databricks when using the legacy hive_metastore with external storage. The root cause is that dbutils.fs.ls and Spark read operations use the credentials you set in the Spark session config, but the Hiv...

  • 1 kudos
1 More Replies
ArunKhandelwal
by • New Contributor II
  • 6927 Views
  • 1 replies
  • 1 kudos

Whitepaper - leverage Databricks to build a robust data governance framework

I am excited to share that I recently published a whitepaper on how to leverage Databricks to build a robust data governance framework. This paper explores key strategies and best practices for implementing a strong governance model using Databricks'...

  • 6927 Views
  • 1 replies
  • 1 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 1 kudos

Hey @ArunKhandelwal ,  Thanks for sharing this with the community! Data governance is top of mind for so many of our customers, and having practical guidance on how to bring policy and control frameworks into Databricks is incredibly valuable. I’m lo...

  • 1 kudos
ajai_duraisamy
by • New Contributor
  • 1207 Views
  • 1 replies
  • 1 kudos

Unity catalogue sync problem

Hi Everyone ,while I am creating sync table in UC, I got an error and failed . then I am not able to recreate the same table.when I checked the catalogue table name is there but not able to open , and I am querying the same in the notebook .it's givi...

  • 1207 Views
  • 1 replies
  • 1 kudos
Latest Reply
nayan_wylde
Esteemed Contributor II
  • 1 kudos

@ajai_duraisamy Here are few checks you can do:1. SHOW TABLES IN <catalog>.<schema> LIKE '<table>'; This command will show if the table exists in UC.2. Make sure your compute is UC‑enabled.If you’re on a cluster/SQL Warehouse without UC enabled (or c...

  • 1 kudos
kktim
by • New Contributor II
  • 5709 Views
  • 2 replies
  • 1 kudos

Resolved! Accessing unity catalog volumes from a databricks web application

Hello,I am trying to deploy a gradio app (app.py) in databricks, but I am having problem accessing data stored in a volume in unity catalog. It seems like that I cannot access the data using path like "/Volumes/catalog/schema.../my_data", which works...

  • 5709 Views
  • 2 replies
  • 1 kudos
Latest Reply
MMRDUS
New Contributor II
  • 1 kudos

Unlike notebooks, Databricks Apps does not support mounting Unity Catalog volumes and directly reading and writing files. As this code snippet demonstrates, each file needs to be downloaded to the app compute before being able to manipulate it.

  • 1 kudos
1 More Replies
Col1ns
by • New Contributor
  • 3787 Views
  • 2 replies
  • 2 kudos

Filtering Partitioned Data in Databricks Delta Share

We have a Delta Share that includes a partitioned table, and we want our recipient to be able to retrieve data only from a specific partition.I reviewed the Delta Sharing server documentation and found that it's possible to use SQL expressions for fi...

  • 3787 Views
  • 2 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 2 kudos

Hi @Col1ns ,All the things mentioned by @lingareddy_Alva are correct. Some additional things to consider. Check if your implementation of Delta Sharing supports predicate hints. Also, if your server supports them then your predicate should use restri...

  • 2 kudos
1 More Replies
katiej02
by • New Contributor II
  • 10221 Views
  • 1 replies
  • 0 kudos

Error: cannot import share credential file from implemented delta sharing server in Unity Catalog

We have implemented our own delta sharing server, following the delta sharing protocol. We are able to use the delta-sharing connectors to read shared data from our own server. However, when we tried importing our share credential file in Databricks ...

  • 10221 Views
  • 1 replies
  • 0 kudos
Latest Reply
katiej02
New Contributor II
  • 0 kudos

Fixed

  • 0 kudos
yvishal519
by • Contributor
  • 10364 Views
  • 5 replies
  • 3 kudos

Resolved! Accessing Databricks Delta Live Tables (DLT) in MS Fabric with Unity Catalog Integration

Hi Community,I’ve been working on a data pipeline using Databricks Delta Live Tables (DLT) with Unity Catalog. My setup saves data into Azure Data Lake Storage (ADLS) for streaming tables. However, I’ve noticed that Unity Catalog encrypts various com...

  • 10364 Views
  • 5 replies
  • 3 kudos
Latest Reply
yvishal519
Contributor
  • 3 kudos

Hi Community,I previously reached out regarding creating shortcuts in Microsoft Fabric for Databricks Delta Live Tables (DLT) managed through Unity Catalog, specifically when the data resides in Azure Data Lake Storage (ADLS) and appears encrypted.Af...

  • 3 kudos
4 More Replies
Anske
by • New Contributor III
  • 7212 Views
  • 2 replies
  • 0 kudos

Delta Sharing - 'No data' in schema

When creating a data share in my Databricks workspace, in the step Add data assets, one of my schemas in Unity Catalog shows 'No data' when expanding. If I expand the same schema in Unity Catalog explorer, it shows lots of tables and all tables conta...

  • 7212 Views
  • 2 replies
  • 0 kudos
Latest Reply
MathieuDB
Databricks Employee
  • 0 kudos

Hello @Anske , This feature is currently in private preview. You should contact your account team to have it enabled in your workspace(s). Requirements Databricks Runtime (DBR) 13.3 or aboveRecipients must use Serverless SQL warehouse to query a shar...

  • 0 kudos
1 More Replies
F_Goudarzi
by • New Contributor III
  • 5965 Views
  • 2 replies
  • 0 kudos

Naming conventions for delta sharing

Hi All,A question for those using Delta Sharing, how are you defining naming conventions for Shares and Recipient names? What best practices or standards are you following? Thanks, Fatima

  • 5965 Views
  • 2 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

For platform-to-platform environments, I nest all shared tables under a specially named Schema with a prefix of DS-.  For example, my schema name could be ds-mfgschema. This is a 'flag' that any tables nested under this schema are being delta shared ...

  • 0 kudos
1 More Replies
pernilak
by • New Contributor III
  • 2388 Views
  • 1 replies
  • 0 kudos

Permissions needed for creating a new (delta) share

What permissions are actually needed in order to create a delta share?I have read that you either need to be metastore admin, or aUser with delegated permissions or ownership on both the share and the recipient objects ((USE SHARE + SET SHARE PERMISS...

  • 2388 Views
  • 1 replies
  • 0 kudos
Latest Reply
MoJaMa
Databricks Employee
  • 0 kudos

Hi,  Hope the doc link helps: https://docs.databricks.com/en/delta-sharing/create-share.html#requirements Specifically this statement should alleviate your immediate point. Have the USE CATALOG and USE SCHEMA privilege on the catalog and schema that...

  • 0 kudos
Gilg
by • Contributor II
  • 5115 Views
  • 3 replies
  • 0 kudos

Delta Sharing within the same region

Hi Team,I have setup UC and have a Metastore for AUE. Assigned 2 workspaces to the AUE Metastore and both with different set of users and group . 1 workspace is the data provider and the other is a data consumerNow since 2 workspaces are in one Meta...

  • 5115 Views
  • 3 replies
  • 0 kudos
Latest Reply
NikunjKakadiya
New Contributor II
  • 0 kudos

Do we have any updates on this in terms of how we can share the data between different workspaces part of the same metastore?

  • 0 kudos
2 More Replies
Labels