Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
In the Databricks one demo, and even in official website I seehttps://www.databricks.com/blog/introducing-databricks-one"Domains", how to enable it ? Or add "Domain" to data?
We are currently managing our permissions via Terraform (including cluster creation, UC governance, etc.). We have a specific `data_engineer` role, and we need everyone with this role to be able to view and manage all of our SDPs.The Issue: Currently...
Our Solution: We moved job and pipeline permissions to DAB configuration files for streamlined enforcement. Terraform will remain the source of truth for workspace-level permissions only.
Hi!By default it seems users can only see tables and views in Unity Catalog that they have SELECT permission/privilege on. However, we would like to use Unity Catalog as a data catalog of tables we have. They wouldn't then be able to request access t...
Hi All, We are in a process of migrating to UC from HMS and also changing the DBR from 11.3 to 16.4 LTS. When i am running sync schema from HMS to UC i am getting "Parquet does not support date. See HIVE-6384" error. Trying to resolve this, i have di...
@yashojhaIf you are doing HMS to UC migration. It is worth checking UCX an automated Unity catalog migration tool.https://github.com/databrickslabs/ucx
We are attempting to create a comprehensive Data Governance solution using Unity Catalog & INFA CDGC tool. The objective is to onboard the business assets on the Informatica DG platform and use the unity catalog to trace technical assets and lineage ...
Today there’s no native, single-UI integration between Informatica CDGC and Unity Catalog. Most teams use Unity Catalog for technical metadata and lineage on Databricks assets, and INFA CDGC for business glossary and stewardship. Metadata can be sync...
Hello,Is SQL warehouses managed by Unity Catalog? My understanding is that since SQL Warehouse is part of the compute layer, Unity Catalog doesn't manage it as it only manages data layers.
Hi @Senga98 ,Your understanding is correct. Unity Catalog governs:Data objects (catalogs, schemas, tables, views, functions)Permissions (grants on the above)LineageGoverned storage locations & external locationsModel serving endpoints (UC Volumes / A...
HelloIs SQL warehouses managed by Unity Catalog? My understanding is that since SQL Warehouse is part of the compute layer, Unity Catalog doesn't manage it as it only manages data layers.
Hi @Senga98 ,Your understanding is correct. Unity Catalog governs:Data objects (catalogs, schemas, tables, views, functions)Permissions (grants on the above)LineageGoverned storage locations & external locationsModel serving endpoints (UC Volumes / A...
I have an S3 account in which I have full administrator privileges. In that account I have a databricks workspace and an EFS filesystem setup. I created an interactive compute inside databricks workspace with the default config. How do I configure m...
if you’re trying to mount EFS directly to an interactive cluster, you’ll usually need to handle it through init scripts since EFS requires the NFS client to be installed and mounted at cluster startup. One thing to double-check is whether your worksp...
I have a Python-based FastAPI application with a React frontend, which I would like to deploy to Databricks apps. I'm able to do it, but having trouble with file system access. In addition to database access, my application relies on some file operat...
I am running into the same issue: UC volumes are simply not mounted inside the app.According to the documentation this should be possible but it is simply isn't.https://docs.databricks.com/aws/en/dev-tools/databricks-apps/uc-volumesIn my case, the wo...
For some reason the SCIM API endpoint to get all the groups doesn't return any members information.According to the docs it should contain all members that are part of a group.Anyone know how to fix this?
Hi @ECCBV ,Which endpoint do you use? The following one will list for you all groups:List groups /api/2.1/accounts/{account_id}/scim/v2/Groups Now, if you want to list a members of the group you need to use following endpoint and provide group_id par...
Does removing the ADLS path metastore location affect the data or cause any critical issues? the ADLS path tied to a project and if people from other projects wanted to use the metastore for catalog creation. We do not want to incur any payments for ...
Hi @VRaj, removing the ADLS path from a metastore won’t delete existing data, but any references to that path for catalog creation will fail. To prevent cross-project storage charges, you could create separate metastores per project or use access con...
I am using Databricks Apps, and I want to grant privileges to the auto-generated service principal from the app to read specific Unity Catalog schemas. I can't find a way to do this automatically using Asset Bundles.I prefer to query using a service ...
Why not to run a databricks cli script for first assign proper permissions (https://docs.databricks.com/aws/en/dev-tools/cli/reference/account-commands) and then run DAB? This was my approach to deploy some components not already available in previou...
Hello Team, We are building different KPIs on the Databricks Dashboard, the issue is: We have many pages which are part of the Dashboard, they are working but the issue is we are not able to navigate the pages with mouse if they are moving out of the...
Hello @saravjeet! You can check out this similar post: Databricks Dashboard Issue: No Mouse-Based Navigation When Dashboard Tabs Exceed the Top Ribbon If you’re using a mouse, you can navigate horizontally by: Holding Shift and scrolling with the mou...
In my organization, we are using Databricks unity catalog and we have a metastore created for our region which holds all of our workspaces. When we created the metastore last year, we set a metastore root location for it (If I remember correctly, met...
Currently, Databricks Unity Catalog does not allow removing the metastore root once it’s set. The main options are:Create a new metastore without a root path and migrate catalogs, external locations, and workspaces into it. This approach works but re...
Data Profile on a table is not a securable object in Unity Catalog or at Workspace level. This make the management of Data Profiles difficult for workspace admins.Why isn’t “profile” a securable object in Databricks? It makes sense to require “Manage...
Data profiling is a configuration you attach to a table. The output metric tables are Unity Catalog objects that you choose the schema for and can govern with UC privileges like any other table. The profile’s configuration and refresh history are wor...
