Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
Looking at Databricks’ suggested use of catalogs. My instincts are now leading me to the conclusion having separate metastore for each SDLC environment (dev, test, prod) is preferable. I think if this pattern were followed, this means due to current ...
Glad to see its not just me thinking about multiple metastores. Separate metastores by environment makes total sense. This would have complete isolation between environments, also in your dev, stg, prod you can reuse catalog names without having to u...
try build Delta sharing for PowerBI and OCI Automouns Database. connection it work but when pull data show error. So doesn't know if i use free version. go to Databricks delta sharing setting , add recipient show i need permission
Hello @FanMichelleTW ,Yes, this is most likely a limitation of the Databricks Free Edition. According to the documentation, this edition is intended for non-commercial use and includes several feature restrictions.In Delta Sharing, a recipient is an ...
Hey Databricks Community,I have applied USE CATALOG, USE SCHEMA and SELECT privilege to a table in the unity catalog. The grants shows only SELECT but I am able to do INSERT as well. Is this the expected behaviour ? or am missing something ?DELETE an...
Hello @old_school Good day!I dont think its expected behaviour. Table Privileges SummaryPrivilegeAllowsPrerequisites SELECTRead data (SELECT * FROM table)USE CATALOG + USE SCHEMAMODIFYWrite data: INSERT, UPDATE, DELETE, MERGESELECT + USE CATALOG + US...
Hi there,We are using Databricks and are in the early stages of adopting it. Recently, I noticed something in Databricks that caught my attention.I implemented column-level security, which works such that if you are a member of a particular group, yo...
Hey @F_Goudarzi , here are some things to think about: Is this a security issue? This is expected behavior: workspace admins have broad authority over workspace assets (including viewing notebook content). Row filters and column masks apply at quer...
hi..I want to use 1 all purpose cluster and use dynamic taging in each notebook.for example tag project and department.I want to be able to change the tag from the notebook so I will be able to understand the costs for each project and departme...
Greetings @pargit , Why Your Approach Isn't Working Cluster usage tags cannot be dynamically modified at runtime from within a notebook. The `spark.databricks.clusterUsageTags.` configurations are read-only properties set when the cluster is created...
Example: I wantColumns tagged sensitive_level = pii → masked for everyone.But if column is classification = email → allow users in group "see_email_group" to see it.Similar to tag:value classification = phone, email, tax_code,... Column Tag Descripti...
Something like following should work for your scenario. CREATE FUNCTION mask_email_tiered(value STRING) RETURN STRING RETURN CASE WHEN IS_MEMBER('admin') THEN value -- Full access WHEN IS_MEMBER('da_email') THEN value ...
I am excited to share that I recently published a whitepaper on how to leverage Databricks to build a robust data governance framework. This paper explores key strategies and best practices for implementing a strong governance model using Databricks'...
Hey @ArunKhandelwal , Thanks for sharing this with the community! Data governance is top of mind for so many of our customers, and having practical guidance on how to bring policy and control frameworks into Databricks is incredibly valuable. I’m lo...
We have been using the feature MLflow Model Registry Webhooks on Azure Databricks . Webhooks are in public preview, but oddly are deprecated at the same time with the introduction of Unity Catalog. The documentation says "For an alternative, see Can ...
Greetings @bragoarefur , Thanks for laying out the context so clearly—your read of the current state is spot on. Below is what the Databricks docs actually say, and a recommendation with an implementation path that avoids polling while staying cloud...
Hi everyone!I would love to know whether Databricks has the following capabilities and what are the core functionalities of the tool:Data glossary: provides clear definitions of business terms, ensuring that everyone in the organization uses the same...
Hi @DG1 - You can use AI to generate comments for your Unity Catalog Database objects. For the Data glossary, you can have specific roles in your organization to establish those standards. We do have business semantics (UC metrics Views) for a centra...
Hi everyone,I want to apply Row Filters to restrict access to Unity Catalog tables using a dedicated cluster running Databricks Runtime 15.4 LTS (Spark 3.5.0, Scala 2.12).I followed the official documentation which says that row filters are supported...
Hi @ESN, let's first understand why you need to use a dedicated cluster with FGAC (Fine-Grained Access Control). You should ideally use a standard cluster where you do not need the serverless component for filtering. Is there a specific piece of your...
I have been tasked with governing the data within our Databricks instance. A large part of this is adding Comments or Descriptions, and Tags to our Schemas, Tables and Columns in Unity Catalog.For most objects this has been straight-forward, but one ...
It is currently not possible to reliably add or persist comments or descriptions directly to Streaming Tables created via the SQL Server Data Ingestion Wizard in Databricks using the Data Ingestion UI or Jobs & Pipelines UI. All metadata management—i...
Hi Databricks team,Quick heads-up and request for guidance. While collaborating on a notebook in my personal workspace, a colleague who had READ permission on a workspace-backed Secret Scope was able to delete a secret key via terminal/CLI from the s...
Hi @guidotognini - This is not expected behavior if your colleague truly has only READ and their CLI was using their own identity; deletion requires WRITE/MANAGE on the scope. The most probable explanation is that the CLI call was authenticated as an...
Hello Team,Could you please let me know what Databricks recommended way to move a job from lower env to higher , especially when certain parameters will be dynamic with respect to env .DAB support templating ?
@arun_6482 Please accept the solution if your issue is solved, thank you!
The future of data and AI sharing should be open, secure, and seamless not locked into costly proprietary systems. What is Delta Sharing?Delta Sharing is the world’s first open protocol for secure, real-time data sharing at scale, developed by Databr...
As a way to minimize storage costs, my team and I want to create views instead of tables in the Gold layer.We always try to improve the experience of our users by adding comments to the columns. the problem is that views do not inherit comments from ...
you can use COMMENT ON to set or update the comment on an view-- View-level comment (use TABLE for views)COMMENT ON TABLE catalog.schema.my_view IS 'View for user info';-- Column-level comment on a viewCOMMENT ON COLUMN catalog.schema.my_view.name IS...
