If you're catching up on Data + AI Summit 2026, here’s a simple Unity Catalog recap for data governance. Three themes stand out: Control, Context, and Choice. Control: Keep governance consistent as AI moves from access to action. AI governance at Dat...
I gone through resources supported by DABs but cannot find notebooks deployment via DABs. Is it any possible way we can deploy notebooks using DABs? What the alternative approach?
Hi @Akash30307 ,Thank you for your question!You can also deploy notebooks via our Terraform provider.You can find more information on our website.You can also work with databricks_notebook and databricks_notebook_paths data sources.
We are looking to send messages using databricks to an azure service bus topic and would like to connect to the service bus using a managed identity vs a connection string. Is this possible in databricks. The only thing I could find regarding datab...
Is there any update to use managed identity instead of dbmanaged identity?
Problem: Queries on our 100M+ record Iceberg tables were taking 45+ seconds.Solution: Implemented lightweight zonemap indexing that tracks min/max values per file.Quick Implementationdef apply_zonemap_pruning(table_path, predicate_value):# Load zonem...
Hi! @ck7007 @WiliamRosa ,I have a question — why are you actually doing this? I’m not fully familiar with your exact setup (Iceberg), but my understanding is that Iceberg already stores these stats (min/max) in the manifests, and Spark should be able...
Hello there!After a year working with Databricks, we have found that we could improve our catalog and schemas organization, so we would like to perform a 'migration' process where we move tables, volumes, models, etc from one catalog to another.The p...
Hi @-werners- Exactly, “It is not possible to have overlap within/between catalogs.” I’m sharing below the official documentation that confirms this behavior:https://docs.databricks.com/aws/en/volumes/paths
Currently our Databricks platform is been audited against ITGC standards. ITGC 101 which is create access, we need to provide a first login date after access grant. The system audit table provides all logins for the last 180 days but our period is gr...
Hi @MatthewSandstro ,You can create a pipeline that will historize data from the audit system. This way, you can bypass the limit and have the full history of user logins
Hi Everyone,I am trying to write data to a delta table created on Unity Catalog with external location. I am using AWS EMR and below are my table and spark properties.#### Spark Shell```spark-shell \--conf "spark.sql.defaultCatalog=<catalog_name>" \-...
Hi @SP_6721 ,I am able to read data from Spark (as it is using delta logs) and able to view the schema/catalog details on Trino over EMR.I am also able to write data to delta files in s3 using UC. However, the metadata/manifest file for iceberg is no...
Understanding DLT pipe lines in DatabricksIn Databricks, a DLT (Data Live Table) pipeline is a set of data transformations that are applied to data assets in a defined sequence, in order to clean, enrich, and prepare data for analysis or other purpos...
We have a large Databricks instance, and we are performing a technical audit of Databricks to identify (1) the full list of users, service principals, and groups; (2) the full list of objects (e.g. catalogs, schemas, jobs, notebooks, etc.); and (3) t...
Hi @Databricks1126,I understand that you’re looking to capture permissions across a wide variety of Databricks objects. These can generally be grouped into three main categories:- Data objects (Unity Catalog–governed) – catalogs, schemas, tables, vie...
I'm trying to identify events in the audit logs that correspond to granting permissions for dashboards, genie spaces, and folders. Based on the documentation, my understanding is that I should be looking for the following:serviceName: 'unityCatalog'a...
Can we Move delta live tables amongst schemas within a catalog
Do you have a mechanism to do this ?Any aspects that are accessible to allow this to be possible?
no information on character_maximum_length and is_identity in unity catalog table system.information_schema.columnstable definition exampleCREATE TABLE tyable_name ( UniqueID bigint not null GENERATED ALWAYS AS IDENTITY , ID varchar(256))looks like ...
if you look at the official document here: https://docs.databricks.com/gcp/en/sql/language-manual/information-schema/columnsmany of them are "Always NULL, reserved for future use."so I think the feature simply isn't available to use yet. It's seems i...
i try to find out SQL for RLS , the databricks geni give this feedback "In Databricks SQL, the CREATE ROW ACCESS POLICY statement is not supported" . it means we can set RLS (role level security) for managed table or external table ?
We have RLS applied on a table and we would like to enable lake house monitoring for a quality profile metrics ,My Questions:Which identity is used by Lakehouse Monitoring and how can I find it, so that I can exclude it for unrestricted access to tab...
Hi @karunakaran_r In Databricks Lakehouse Monitoring, the profiling and drift metric collection runs as a service principal that’s tied to the Databricks system itself, not as your own user account.That means when the monitor queries your table, it w...
Hi,I am working with Databricks Serverless Compute and All-Purpose Compute and have several questions regarding tagging, system tables, and budget policies. Specifically, I need clarification on how these work together in Databricks, and how I can ma...
I am now more inclined towards the serverless and tagging, would be more helpful if you could take a real time use case and elaborate on it. Ty
Unity Catalog: create the first metastoreThe great benefit of the Unity catalog is that data is ours and stored in an open format on cloud storage in our container. To install the unity catalog, we need to create storage and give databricks access to...
