cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

MadelynM
by Databricks Employee
  • 676 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Data Governance | Navigate the explosion of AI, data and tools

Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...

Screenshot 2024-07-03 at 9.27.29 AM.png
  • 676 Views
  • 0 replies
  • 0 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 5241 Views
  • 1 replies
  • 1 kudos

Enable lineage data system tables

Execute the below Python code in your databricks workspace to enable lineage system tables import requests ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() api_url = ctx.tags().get("browserHostName").get() api_token = ctx.apiTo...

ezgif-1-9a34272eba.gif
  • 5241 Views
  • 1 replies
  • 1 kudos
Latest Reply
iplantevin
New Contributor III
  • 1 kudos

Great, I wondered why I didn't see them

  • 1 kudos
MiroFuoli
by New Contributor II
  • 7374 Views
  • 1 replies
  • 1 kudos

Unity Catalog - Limited Options for Connection Objects

I’m currently trying to create a Foreign Catalog based on a Connection object of type SQLSERVER. This would allow me to directly access our on-premises MS SQL database from within Azure Databricks using Unity Catalog.As I’m part of a large organizati...

Data Governance
Connection
Foreign Catalog
JDBC
SQL Server
Unity Catalog
  • 7374 Views
  • 1 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
I’m currently trying to create a Foreign Catalog based on a Connection object of type SQLSERVER. This would allow me to directly access our on-premises MS SQL database from within Azure Databricks using Unity Catalog.As I’m part of a large organizati...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
OwenDB
by New Contributor III
  • 8452 Views
  • 2 replies
  • 1 kudos

Migration and deletion of metastore

Good afternoon,We are in the process of setting up Unity Cat, we started in our existing workspace and set it up fine. We are particularly interested in federating existing warehouses at this point. We were able to federate one datawarehouse (dwh1) b...

  • 8452 Views
  • 2 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Good afternoon,We are in the process of setting up Unity Cat, we started in our existing workspace and set it up fine. We are particularly interested in federating existing warehouses at this point. We were able to federate one datawarehouse (dwh1) b...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
1 More Replies
-werners-
by Esteemed Contributor III
  • 14091 Views
  • 6 replies
  • 2 kudos

multiple storage credentials/external locations to same physical location

Hi all,we are in the process of rolling out a new unity-enabled databricks env with 2 tiers: dev and prod.Initially we had the plan to completely decouple dev and prod, each with their own data lake as storage.While this is the safest option, it does...

  • 14091 Views
  • 6 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

That is the way I am working at right now.  Assign workspace to the catalog and set to read-only if necessary.It would be easier though if it was possible to define a 2nd external location in read-only, as this cannot break anything (of course in rea...

  • 2 kudos
5 More Replies
karthik_p
by Esteemed Contributor
  • 2689 Views
  • 0 replies
  • 0 kudos

Managed Loactions overlap Issue DLT UC

Did any one tried UC DLT, we are trying to create multiple catalogs for UC DLTnote: as per limitations it won't support Locations, supports managed locationsOur use case: we want to create 2 catalogs one for dev and other for production. Dev catalog ...

  • 2689 Views
  • 0 replies
  • 0 kudos
TienDat
by New Contributor III
  • 9341 Views
  • 5 replies
  • 0 kudos

Filter sensitive data on nested column

Dear all,We are working on the column masking topic recently, using the column filter feature of Unity Catalog.We recently face the problem of masking a nested column (a sub-column within a STRUCT type column).We just wonder if this is even possible ...

  • 9341 Views
  • 5 replies
  • 0 kudos
Latest Reply
DucNguyen
New Contributor II
  • 0 kudos

Same as my concern. I try to mask with Decimal datatype but It doesn't work as well. The example of DBX for column mask maybe work well with simple datatype like string. Somehow it doesn't meet our requirements for data governance.

  • 0 kudos
4 More Replies
Kavi_007
by New Contributor III
  • 12609 Views
  • 5 replies
  • 1 kudos

Resolved! Unable to open the account console

I'm the account admin for the subscription where the databricks workspace is created. However If I open the account console, its prompting me to select one of the databricks workspaces. Please note that its my own subscription via visual studio benef...

  • 12609 Views
  • 5 replies
  • 1 kudos
Latest Reply
Kavi_007
New Contributor III
  • 1 kudos

If there is an existing metastore for the region, doesn't mean my organization store all the meta data across the workspaces in a specific metastore or data lake ? Databricks suggests to have one metastore per region. How the project specific metadat...

  • 1 kudos
4 More Replies
Pritesh2
by New Contributor II
  • 3019 Views
  • 0 replies
  • 0 kudos

How Can Companies Organize Their Data Through Data Governance?

When managing and safeguarding your internal data, data governance is crucial. It works like insurance, ensuring that all the data you gather is appropriately disseminated and kept safe within your company.Let's discuss data governance and how to imp...

Tech2_0-1702985203853.png Tech2_1-1702985203952.png Tech2_2-1702985203894.png
Data Governance
data engineering
Data Governance
  • 3019 Views
  • 0 replies
  • 0 kudos
Kris2
by New Contributor II
  • 3518 Views
  • 1 replies
  • 0 kudos

Unable to create a Managed table in Unity Catalog default location

We have setup the metastore with Manged Identity and when trying to create a managed table in the default location I am hitting below error. The storage is ADLS Gen2. AbfsRestOperationException: Operation failed: "This request is not authorized to pe...

  • 3518 Views
  • 1 replies
  • 0 kudos
Latest Reply
RebeccaVC
Databricks Employee
  • 0 kudos

Have you followed all the steps in here to create the metastore? https://app.getreprise.com/launch/wy1Y2ly/Do you have all the necessary permissions granted to create a Managed Table? https://docs.databricks.com/en/data-governance/unity-catalog/manag...

  • 0 kudos
Amber-26
by New Contributor III
  • 6963 Views
  • 4 replies
  • 0 kudos

System Tables not showing up in PowerBI

Hello,I am currently working with System Tables on Unity Catalogue part. I have loaded all the schemas in the catalogue and I am using PowerBI to directly access these tables. But while connecting PowerBI to Databricks, I am not able to see System Ta...

Amber26_0-1699456599708.png
  • 6963 Views
  • 4 replies
  • 0 kudos
Latest Reply
Chinu
New Contributor III
  • 0 kudos

I had/have the same issue with the Tableau desktop. I'm not able to select the "Billing" schema because I don't see the "System" catalog. However, I found a workaround to resolve this issue. In databricks console, go to the "Catalog Explore" and sele...

  • 0 kudos
3 More Replies
AdamMcGuinness
by New Contributor III
  • 33391 Views
  • 4 replies
  • 1 kudos

Metastore - One per Account/Region Limitation

Looking at Databricks’ suggested use of catalogs. My instincts are now leading me to the conclusion having separate metastore for each SDLC environment (dev, test, prod) is preferable. I think if this pattern were followed, this means due to current ...

  • 33391 Views
  • 4 replies
  • 1 kudos
Latest Reply
SSundaram
Contributor
  • 1 kudos

You can create multiple metastores for each region within an account. This is not a hard constraint, reach out to account team and they can make an exception. Before doing that, consider what kind of securable sharing you will need between dev, test ...

  • 1 kudos
3 More Replies
marc88
by New Contributor II
  • 3569 Views
  • 1 replies
  • 0 kudos

Only read access to databricks delta tables

Hi,Is it possible to make delta shares work for a Matlab client (Just the way we share delta tables to a PowerBI workbench)?We have Unity Catalog enabled and I wanted to explore more about data governance with aforesaid integration.Wanted to discuss ...

  • 3569 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi,Is it possible to make delta shares work for a Matlab client (Just the way we share delta tables to a PowerBI workbench)?We have Unity Catalog enabled and I wanted to explore more about data governance with aforesaid integration.Wanted to discuss ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
stiaangerber
by New Contributor III
  • 5539 Views
  • 3 replies
  • 2 kudos

MLlib load from UC Volume: IllegalArgumentException: Cannot access the UC Volume path...

I'm trying to store MLlib instances in Unity Catalog Volumes. I think volumes are a great way to keep things organized.I can save to a volume without any issues and I can access the data using spark.read and with plain python open(). However, when I ...

  • 5539 Views
  • 3 replies
  • 2 kudos
Latest Reply
slimexy
New Contributor II
  • 2 kudos

Just to supplement that if the ML model is saved and then loaded within the same execution, calling load() will not cause the mentioned exception. Copying the model directory from UC volume to ephemeral storage attached to the driver node is also a w...

  • 2 kudos
2 More Replies
prasad_vaze
by New Contributor III
  • 4774 Views
  • 0 replies
  • 0 kudos

Is there a way to import table , column descriptions into unity catalog?

I have a spreadheet containing table & column descriptions (comments) Is there a way to upload this against the schema in unity catalog?  Basically instead of running  'Alter table <> alter column <>  comment "description"  '  command for every colum...

Data Governance
Unity Catalog
  • 4774 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels