If you're catching up on Data + AI Summit 2026, here’s a simple Unity Catalog recap for data governance. Three themes stand out: Control, Context, and Choice. Control: Keep governance consistent as AI moves from access to action. AI governance at Dat...
I'm currently working with Databricks within Azure and encountering an issue with Unity Catalog in a newly created workspace.My colleague initially deployed a Databricks workspace where Unity Catalog was provisioned and enabled by default. I recently...
Hello like szymon suggested just try to check with the admin also as you share vnet injected workspace was it done recently if yes! ThenMight be you were missing two private endpoint of storage account you needed to create 2 private endpoints, both ...
Hi Everyone ,while I am creating sync table in UC, I got an error and failed . then I am not able to recreate the same table.when I checked the catalogue table name is there but not able to open , and I am querying the same in the notebook .it's givi...
@ajai_duraisamy Here are few checks you can do:1. SHOW TABLES IN <catalog>.<schema> LIKE '<table>'; This command will show if the table exists in UC.2. Make sure your compute is UC‑enabled.If you’re on a cluster/SQL Warehouse without UC enabled (or c...
I'm trying to use the Spark-Nvidia integrations in order to train Spark ML models using GPU.While trying to follow these instructions: https://docs.nvidia.com/spark-rapids/user-guide/23.12.1/getting-started/databricks.htmlI could not execute the init...
Since cluster initialization happens before the "libraries" section installs Maven artifacts, the plugin isn’t available at the required time, causing the error. Workaround Strategies 1. Internal Artifactory or DBFS Manual Upload Upload the RAPIDS j...
Is there anyone can provide reference that AWS Databricks comply with FISC (The Center for Financial Industry Information Systems) standard which is a standard for financial industry in Japan?I can't find any official information yet.or if you find i...
Hi @tana_sakakimiya ,As you said, there's no official documentation on databricks regarding this.But when we consider below FAQ from AWS FISC docs page we can see that from audit perspective they will focus mainly on SOC 1, SOC 2 and SOC 3And those a...
Hello,I am trying to deploy a gradio app (app.py) in databricks, but I am having problem accessing data stored in a volume in unity catalog. It seems like that I cannot access the data using path like "/Volumes/catalog/schema.../my_data", which works...
Unlike notebooks, Databricks Apps does not support mounting Unity Catalog volumes and directly reading and writing files. As this code snippet demonstrates, each file needs to be downloaded to the app compute before being able to manipulate it.
Hi everyone,I'm a Cloud Engineer working on a multi-environment Databricks setup (Dev, QA, Prod), and I've received a request from our Data Engineering team that's a bit unconventional — they are asking for access to Production Unity Catalogs from lo...
Hi @Charansai ,That's a great question! In general, granting Dev/QA users direct access to Production catalogs is not considered best practice. The main risks you already mentioned (governance, compliance, and accidental writes) usually outweigh the ...
I gone through resources supported by DABs but cannot find notebooks deployment via DABs. Is it any possible way we can deploy notebooks using DABs? What the alternative approach?
Hi @Akash30307 ,Thank you for your question!You can also deploy notebooks via our Terraform provider.You can find more information on our website.You can also work with databricks_notebook and databricks_notebook_paths data sources.
We are looking to send messages using databricks to an azure service bus topic and would like to connect to the service bus using a managed identity vs a connection string. Is this possible in databricks. The only thing I could find regarding datab...
Is there any update to use managed identity instead of dbmanaged identity?
Problem: Queries on our 100M+ record Iceberg tables were taking 45+ seconds.Solution: Implemented lightweight zonemap indexing that tracks min/max values per file.Quick Implementationdef apply_zonemap_pruning(table_path, predicate_value):# Load zonem...
Hi! @ck7007 @WiliamRosa ,I have a question — why are you actually doing this? I’m not fully familiar with your exact setup (Iceberg), but my understanding is that Iceberg already stores these stats (min/max) in the manifests, and Spark should be able...
Hello there!After a year working with Databricks, we have found that we could improve our catalog and schemas organization, so we would like to perform a 'migration' process where we move tables, volumes, models, etc from one catalog to another.The p...
Hi @-werners- Exactly, “It is not possible to have overlap within/between catalogs.” I’m sharing below the official documentation that confirms this behavior:https://docs.databricks.com/aws/en/volumes/paths
Currently our Databricks platform is been audited against ITGC standards. ITGC 101 which is create access, we need to provide a first login date after access grant. The system audit table provides all logins for the last 180 days but our period is gr...
Hi @MatthewSandstro ,You can create a pipeline that will historize data from the audit system. This way, you can bypass the limit and have the full history of user logins
Hi Everyone,I am trying to write data to a delta table created on Unity Catalog with external location. I am using AWS EMR and below are my table and spark properties.#### Spark Shell```spark-shell \--conf "spark.sql.defaultCatalog=<catalog_name>" \-...
Hi @SP_6721 ,I am able to read data from Spark (as it is using delta logs) and able to view the schema/catalog details on Trino over EMR.I am also able to write data to delta files in s3 using UC. However, the metadata/manifest file for iceberg is no...
Understanding DLT pipe lines in DatabricksIn Databricks, a DLT (Data Live Table) pipeline is a set of data transformations that are applied to data assets in a defined sequence, in order to clean, enrich, and prepare data for analysis or other purpos...
We have a large Databricks instance, and we are performing a technical audit of Databricks to identify (1) the full list of users, service principals, and groups; (2) the full list of objects (e.g. catalogs, schemas, jobs, notebooks, etc.); and (3) t...
Hi @Databricks1126,I understand that you’re looking to capture permissions across a wide variety of Databricks objects. These can generally be grouped into three main categories:- Data objects (Unity Catalog–governed) – catalogs, schemas, tables, vie...
I'm trying to identify events in the audit logs that correspond to granting permissions for dashboards, genie spaces, and folders. Based on the documentation, my understanding is that I should be looking for the following:serviceName: 'unityCatalog'a...
