cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

udittyagi1994
by New Contributor II
  • 4456 Views
  • 5 replies
  • 0 kudos

Unity Catalog - Databricks

I have a unity catalog which is binded to multiple workspaces (Attached to Metastore), In both workspaces I have created different tables under this catalog will the Metastore admin will have access to these tables and data in those tables in differe...

  • 4456 Views
  • 5 replies
  • 0 kudos
Latest Reply
udittyagi1994
New Contributor II
  • 0 kudos

Unity Catalog acts as a single point of access for managing data across Databricks workspaces, which means with unity catalog a Metastore admin can manage the access permissions on the Data which comes under Unity catalog umbrella.Data inside differe...

  • 0 kudos
4 More Replies
ossinova
by Contributor II
  • 3687 Views
  • 3 replies
  • 0 kudos

Fine grained control of volumes

Is it possible to provide fine grained control (folder level/file level) for a given volume?I have two SCIM integrated groups who have read volume access at the catalog level, but those two groups need different permissions on a lower level. Preferab...

  • 3687 Views
  • 3 replies
  • 0 kudos
Latest Reply
rkalluri-apex
New Contributor III
  • 0 kudos

Can you define the external location at the Landing level and create two Volumes one for PDF and other for CSV and provide access to the respective groups 1 and 2.

  • 0 kudos
2 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 26005 Views
  • 1 replies
  • 1 kudos

Sql to Excel

All data finish in Excel, but do it with some dignity. Official #databricks dbsqlcli allows it to be done in the old-style mysqldump way.https://docs.databricks.com/en/dev-tools/databricks-sql-cli.html

sql.png
  • 26005 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Thank you for sharing this @Hubert-Dudek 

  • 1 kudos
databriccone
by New Contributor
  • 2846 Views
  • 1 replies
  • 0 kudos

unity catalog guidelines for internal/external tables and multiple workspaces

We're setting up from scratch the Unity Catalog in our infrastructure in Azure, that is both- multi region ( europe, us)- multi env (dev, qa, prod)So, we did setup 2 metastore, one for each region, one in west europe and one for south central us.So f...

  • 2846 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @databriccone, Setting up Unity Catalog in your Azure infrastructure is a crucial step for better data governance and operational efficiency. Let’s address your questions: Naming Conventions: For naming conventions, consider adding a suffix li...

  • 0 kudos
23940829381
by New Contributor II
  • 4390 Views
  • 2 replies
  • 0 kudos

Integrating Unity Catalog ML Models into Data Lineage

I saw a really nice article (https://www.databricks.com/blog/announcing-public-preview-volumes-databricks-unity-catalog) on the incorporation of various elements of data lineage within Unity Catalog. In my own exploration, I've been able to replicate...

  • 4390 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @23940829381, Thank you for sharing your interest in data lineage within Unity Catalog! It’s a powerful feature that allows you to track the flow of data across various elements in your Databricks environment. Let’s delve into this further: Un...

  • 0 kudos
1 More Replies
dejans
by New Contributor II
  • 2693 Views
  • 4 replies
  • 0 kudos

Can AAD group membership be used as object Owner in UC to edit object description?

I wanted to set description of a schema in Unity Catalog.Description was greyed out and on pencil icon I got: "Requires Ownership of SCHEMA 'xxxxx'."Owner is set to the Azure Active Directory group that I am member of.I was able to change the owner t...

  • 2693 Views
  • 4 replies
  • 0 kudos
Latest Reply
alexcary83
New Contributor II
  • 0 kudos

AAD group membership cannot be used as the object owner in UC to edit object description directly. Object ownership typically refers to individual user accounts, not group memberships, in most systems.

  • 0 kudos
3 More Replies
SergeSmertin
by New Contributor II
  • 2048 Views
  • 0 replies
  • 1 kudos

Unleashing UCX v0.15.0: A Game-Changer for Upgrading to Unity Catalog

We're thrilled to introduce UCX v0.15.0, packed with cutting-edge features, enhancements, and deprecations that'll take your upgrading experience to new heights!   Release Highlights: AWS S3 support: We've added AWS S3 support to the migrate-location...

Data Governance
Databricks Labs
ucx
  • 2048 Views
  • 0 replies
  • 1 kudos
BobBubble2000
by New Contributor II
  • 2403 Views
  • 1 replies
  • 0 kudos

Who or what is System user?

I noticed in the Catalog Explorer of an Unity Catalog integrated workspace that there is a default catalog named 'system' owned by 'System user'. Who is this system user? It is not listed in the admin dashboard of all workspace users. 

  • 2403 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @BobBubble2000, The ‘System user’ in the context of Unity Catalog within an Azure Databricks workspace plays a crucial role in managing data governance and access control.    Unity Catalog: Unity Catalog is a fine-grained governance solution for d...

  • 0 kudos
TanushM
by New Contributor
  • 4282 Views
  • 3 replies
  • 0 kudos

Unity Catalog- Informatica Cloud Data Governance & Catalog Integration

We are attempting to create a comprehensive Data Governance solution using Unity Catalog & INFA CDGC tool. The objective is to onboard the business assets on the Informatica DG platform and use the unity catalog to trace technical assets and lineage ...

  • 4282 Views
  • 3 replies
  • 0 kudos
Latest Reply
Starry
New Contributor II
  • 0 kudos

Yes, Informatica now has native integration to Unity Catalog with its cloud data catalogue and governance service. You can scan Unity catalog Metadata and propagate this in CDGC and apply policies, business terms, overlay data quality scores, display...

  • 0 kudos
2 More Replies
-werners-
by Esteemed Contributor III
  • 12737 Views
  • 7 replies
  • 2 kudos

Resolved! external volume: overlap error with empty catalog?

Hi all,I try to create an external volume in a catalog/schema on a subpath of the external location.This works fine, however for a certain path I cannot create an external volume.  I always getInput path url overlaps with managed storage within 'Crea...

  • 12737 Views
  • 7 replies
  • 2 kudos
Latest Reply
owlleg6
New Contributor III
  • 2 kudos

I have the same issue. I have created a Catalog using External Location, deleted it and now i can't create Volume for this Location even thought Catalog is deleted long ago

  • 2 kudos
6 More Replies
owlleg6
by New Contributor III
  • 12793 Views
  • 6 replies
  • 1 kudos

Models signature inaccessible in Unity Catalog

I am using Unity Catalog to store my models. My Unity Catalog meta-store is hosted in an Azure Storage Account, which has public network access disabled. Access is restricted to certain IP ranges and private endpoint connections only.Recently, my dev...

Data Governance
azure
models
networking
Unity Catalog
  • 12793 Views
  • 6 replies
  • 1 kudos
Latest Reply
lbdatauser
New Contributor II
  • 1 kudos

I have the same problem. It must be a permissions issue. In my case, it happens when the models are under a schema that was created without indicating a specific "MANAGED LOCATION". In those cases, I also have problems trying to load a model. When th...

  • 1 kudos
5 More Replies
Ashley1
by Contributor
  • 10416 Views
  • 8 replies
  • 9 kudos

Backup Unity Catalog and managed tables

Hi All, Can anyone point me to either documentation or personally tried and tested method of backing up (and restoring) Unity Catalog and its associated managed tables? We're running on Azure and using ADLS Gen2.Regards,Ashley

  • 10416 Views
  • 8 replies
  • 9 kudos
Latest Reply
prasad_vaze
New Contributor III
  • 9 kudos

Our UC managed  tables are stored on  prod ADLS storage which is different from UC root storage account.  So what's the best way to backup and restore UC managed tables into different region?   One option is deep clone tables, copy ADLS folders to an...

  • 9 kudos
7 More Replies
GMB
by New Contributor II
  • 1993 Views
  • 2 replies
  • 0 kudos

User ID, Username and Notebook not populated in History Tab

Hi,If I update a Tag on a table within a Noetbook as follows:sql = f"""ALTER TABLE {silver_catalog}.public.tablename SET TAGS ('Is_published' = 'Y');"""spark.sql(sql)Then the User ID and Username, nor Notebook is populated when I look in the History ...

  • 1993 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
1 More Replies
Dp15
by Contributor
  • 3530 Views
  • 2 replies
  • 1 kudos

Resolved! Schema enforcement from External Location

Hi, This might be a silly question to ask but is it possible to enforce schema on a table created from external location, I am trying to create a table from a s3 location and my command looks like thisquery = """CREATE TABLE IF NOT EXISTS {} ...

  • 3530 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution. 

  • 1 kudos
1 More Replies
Yevhen
by New Contributor II
  • 3039 Views
  • 1 replies
  • 1 kudos

Metastore ErrorClass=QUOTA_EXCEEDED limit: 100000

I faced with error "[RequestId=b567c3bb-05fc-4ac8-b4db-2d826b2c00be ErrorClass=QUOTA_EXCEEDED] Cannot create 1 Table(s) in Metastore 95157916-5539-4dc4-a7d6-e9a854ba0591 (estimated count: 100088, limit: 100000). " but I didn't  see  here this limit  ...

  • 3039 Views
  • 1 replies
  • 1 kudos
Latest Reply
arpit
Valued Contributor
  • 1 kudos

@Yevhen Here is the doc for quota limit: https://docs.databricks.com/en/data-governance/unity-catalog/index.html#resource-quotas

  • 1 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels