cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

MadelynM
by Databricks Employee
  • 752 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Data Governance | Navigate the explosion of AI, data and tools

Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...

Screenshot 2024-07-03 at 9.27.29 AM.png
  • 752 Views
  • 0 replies
  • 0 kudos
Mado
by Valued Contributor II
  • 37524 Views
  • 8 replies
  • 1 kudos

Resolved! Databricks Audit Logs, Where can I find table usage information or queries?

Hi,I want to access the Databricks Audit Logs to check the table usage information.I created a Databricks workspace on the premium pricing tier and enabled it for the Unity Catalogue.I configured Audit logs to be sent to Azure Diagnostic log delivery...

image image
  • 37524 Views
  • 8 replies
  • 1 kudos
Latest Reply
Mado
Valued Contributor II
  • 1 kudos

Thanks @Suteja Kanuri​ Could you guide me on how to setup and configure Table Access Control (TAC)?

  • 1 kudos
7 More Replies
Dp15
by Contributor
  • 10161 Views
  • 1 replies
  • 1 kudos

Refresh a External table metadata

Hi,I have an external table which is created out of a S3 bucket. The first time I am creating the table I am using the following command : query = """CREATE TABLE IF NOT EXISTS catalog.schema.external_table_s3           USING PARQUET            LOCAT...

  • 10161 Views
  • 1 replies
  • 1 kudos
Latest Reply
Dp15
Contributor
  • 1 kudos

Hi @Retired_mod thank you for reply, how can we handle the schema changes in the external location, what if there are additions or deletions on the schema, will the refresh table work then too? 

  • 1 kudos
karthik_p
by Esteemed Contributor
  • 6805 Views
  • 2 replies
  • 0 kudos

How to configure storage credentials for aws s3 buckets which use access keys

We have a requirement to configure storage credentials in UC for s3 buckets which have been previously mounted by using access keys  mounts has been depreciated and we need to configure storage credentials when we use UC. But while configuring storag...

  • 6805 Views
  • 2 replies
  • 0 kudos
Latest Reply
nkraj
Databricks Employee
  • 0 kudos

Hello @karthik_p ,  The UC setup provides the feasibility to add the S3 bucket as a volume when compared to the non-UC concept Mounts.When added as Volume, the access to the Volume in UC is Governed via the Storage Credentials.Regarding Storage crede...

  • 0 kudos
1 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 5306 Views
  • 1 replies
  • 1 kudos

Enable lineage data system tables

Execute the below Python code in your databricks workspace to enable lineage system tables import requests ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext() api_url = ctx.tags().get("browserHostName").get() api_token = ctx.apiTo...

ezgif-1-9a34272eba.gif
  • 5306 Views
  • 1 replies
  • 1 kudos
Latest Reply
iplantevin
New Contributor III
  • 1 kudos

Great, I wondered why I didn't see them

  • 1 kudos
MiroFuoli
by New Contributor II
  • 7845 Views
  • 1 replies
  • 1 kudos

Unity Catalog - Limited Options for Connection Objects

I’m currently trying to create a Foreign Catalog based on a Connection object of type SQLSERVER. This would allow me to directly access our on-premises MS SQL database from within Azure Databricks using Unity Catalog.As I’m part of a large organizati...

Data Governance
Connection
Foreign Catalog
JDBC
SQL Server
Unity Catalog
  • 7845 Views
  • 1 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
I’m currently trying to create a Foreign Catalog based on a Connection object of type SQLSERVER. This would allow me to directly access our on-premises MS SQL database from within Azure Databricks using Unity Catalog.As I’m part of a large organizati...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
OwenDB
by New Contributor III
  • 8633 Views
  • 2 replies
  • 1 kudos

Migration and deletion of metastore

Good afternoon,We are in the process of setting up Unity Cat, we started in our existing workspace and set it up fine. We are particularly interested in federating existing warehouses at this point. We were able to federate one datawarehouse (dwh1) b...

  • 8633 Views
  • 2 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Good afternoon,We are in the process of setting up Unity Cat, we started in our existing workspace and set it up fine. We are particularly interested in federating existing warehouses at this point. We were able to federate one datawarehouse (dwh1) b...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
1 More Replies
-werners-
by Esteemed Contributor III
  • 14361 Views
  • 6 replies
  • 2 kudos

multiple storage credentials/external locations to same physical location

Hi all,we are in the process of rolling out a new unity-enabled databricks env with 2 tiers: dev and prod.Initially we had the plan to completely decouple dev and prod, each with their own data lake as storage.While this is the safest option, it does...

  • 14361 Views
  • 6 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

That is the way I am working at right now.  Assign workspace to the catalog and set to read-only if necessary.It would be easier though if it was possible to define a 2nd external location in read-only, as this cannot break anything (of course in rea...

  • 2 kudos
5 More Replies
karthik_p
by Esteemed Contributor
  • 2729 Views
  • 0 replies
  • 0 kudos

Managed Loactions overlap Issue DLT UC

Did any one tried UC DLT, we are trying to create multiple catalogs for UC DLTnote: as per limitations it won't support Locations, supports managed locationsOur use case: we want to create 2 catalogs one for dev and other for production. Dev catalog ...

  • 2729 Views
  • 0 replies
  • 0 kudos
TienDat
by New Contributor III
  • 9478 Views
  • 5 replies
  • 0 kudos

Filter sensitive data on nested column

Dear all,We are working on the column masking topic recently, using the column filter feature of Unity Catalog.We recently face the problem of masking a nested column (a sub-column within a STRUCT type column).We just wonder if this is even possible ...

  • 9478 Views
  • 5 replies
  • 0 kudos
Latest Reply
DucNguyen
New Contributor II
  • 0 kudos

Same as my concern. I try to mask with Decimal datatype but It doesn't work as well. The example of DBX for column mask maybe work well with simple datatype like string. Somehow it doesn't meet our requirements for data governance.

  • 0 kudos
4 More Replies
Kavi_007
by New Contributor III
  • 12811 Views
  • 5 replies
  • 1 kudos

Resolved! Unable to open the account console

I'm the account admin for the subscription where the databricks workspace is created. However If I open the account console, its prompting me to select one of the databricks workspaces. Please note that its my own subscription via visual studio benef...

  • 12811 Views
  • 5 replies
  • 1 kudos
Latest Reply
Kavi_007
New Contributor III
  • 1 kudos

If there is an existing metastore for the region, doesn't mean my organization store all the meta data across the workspaces in a specific metastore or data lake ? Databricks suggests to have one metastore per region. How the project specific metadat...

  • 1 kudos
4 More Replies
Pritesh2
by New Contributor II
  • 3227 Views
  • 0 replies
  • 0 kudos

How Can Companies Organize Their Data Through Data Governance?

When managing and safeguarding your internal data, data governance is crucial. It works like insurance, ensuring that all the data you gather is appropriately disseminated and kept safe within your company.Let's discuss data governance and how to imp...

Tech2_0-1702985203853.png Tech2_1-1702985203952.png Tech2_2-1702985203894.png
Data Governance
data engineering
Data Governance
  • 3227 Views
  • 0 replies
  • 0 kudos
Kris2
by New Contributor II
  • 3587 Views
  • 1 replies
  • 0 kudos

Unable to create a Managed table in Unity Catalog default location

We have setup the metastore with Manged Identity and when trying to create a managed table in the default location I am hitting below error. The storage is ADLS Gen2. AbfsRestOperationException: Operation failed: "This request is not authorized to pe...

  • 3587 Views
  • 1 replies
  • 0 kudos
Latest Reply
RebeccaVC
Databricks Employee
  • 0 kudos

Have you followed all the steps in here to create the metastore? https://app.getreprise.com/launch/wy1Y2ly/Do you have all the necessary permissions granted to create a Managed Table? https://docs.databricks.com/en/data-governance/unity-catalog/manag...

  • 0 kudos
Amber-26
by New Contributor III
  • 7163 Views
  • 4 replies
  • 0 kudos

System Tables not showing up in PowerBI

Hello,I am currently working with System Tables on Unity Catalogue part. I have loaded all the schemas in the catalogue and I am using PowerBI to directly access these tables. But while connecting PowerBI to Databricks, I am not able to see System Ta...

Amber26_0-1699456599708.png
  • 7163 Views
  • 4 replies
  • 0 kudos
Latest Reply
Chinu
New Contributor III
  • 0 kudos

I had/have the same issue with the Tableau desktop. I'm not able to select the "Billing" schema because I don't see the "System" catalog. However, I found a workaround to resolve this issue. In databricks console, go to the "Catalog Explore" and sele...

  • 0 kudos
3 More Replies
AdamMcGuinness
by New Contributor III
  • 33707 Views
  • 4 replies
  • 1 kudos

Metastore - One per Account/Region Limitation

Looking at Databricks’ suggested use of catalogs. My instincts are now leading me to the conclusion having separate metastore for each SDLC environment (dev, test, prod) is preferable. I think if this pattern were followed, this means due to current ...

  • 33707 Views
  • 4 replies
  • 1 kudos
Latest Reply
SSundaram
Contributor
  • 1 kudos

You can create multiple metastores for each region within an account. This is not a hard constraint, reach out to account team and they can make an exception. Before doing that, consider what kind of securable sharing you will need between dev, test ...

  • 1 kudos
3 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels