Learn, Level Up, and Get Rewarded! Save the dates: October 10 – October 31, 2025! We're celebrating learning, growth, and career advancement with an exciting opportunity to level up in data engineering, data analysis, machine learning, and genera...
- 41007 Views
- 57 replies
- 37 kudos
08-07-2025
From discussions to new resources, we had it all — here’s a quick roundup of what's being celebrated in the Databricks Community this week. Top Resolved Questions Why Does Azure Databricks Consume So Much Memory When Running in the Background? Job...
- 674 Views
- 1 replies
- 6 kudos
Wednesday
We are kicking off our Solution Accelerator Series with a powerful healthcare use case — Automated PHI Removal Why this matters:Healthcare organizations must comply with HIPAA regulations to protect sensitive Protected Health Information (PHI). But ...
- 539 Views
- 0 replies
- 2 kudos
Tuesday
We are back with another round of appreciation for members who lit up the Databricks Community this week with their contributions, positivity, and support! Here are the Sparks for this week: szymon_dybczak –Consistently demonstrates a positive att...
- 39 Views
- 1 replies
- 3 kudos
an hour ago
We are thrilled to spotlight Benjamin Stringer ( @BS_THE_ANALYST ) as our Databricks Community Champion for August! From teaching mathematics in Jersey to competing internationally as a professional gamer, and now thriving as a data consultant at Bul...
- 2353 Views
- 18 replies
- 16 kudos
a week ago
Community Activity
- 39 Views
- 1 replies
- 4 kudos
🌟 Community Sparks of the Week | September 5 - 11 🌟
We are back with another round of appreciation for members who lit up the Databricks Community this week with their contributions, positivity, and support! Here are the Sparks for this week: szymon_dybczak –Consistently demonstrates a positive att...
- 39 Views
- 1 replies
- 4 kudos
Latest Reply
- 4 kudos
Thank you for the recognition and for appreciating the community work @Sujitha .
- 4 kudos
- 10 Views
- 0 replies
- 0 kudos
Event-driven Architecture with Lake Monitoring without "Trigger on Arrival" on DABs
AWS databricksI want to create data quality monitoring and event-driven architecture without trigger on file arrival but once at deploy.I plan to create a job which trigger once at deploy.The job run this tasks sequentially.1. run script to create ex...
- 10 Views
- 0 replies
- 0 kudos
- 13 Views
- 0 replies
- 0 kudos
Problems with cluster shutdown in DLT
[Issue] DLT finishes processing, but cluster remains active due to log write errorHi everyone, I'm running into a problem with my DLT pipeline and was hoping someone here could help or has experienced something similar.Problem DescriptionThe pipeline...
- 13 Views
- 0 replies
- 0 kudos
- 76 Views
- 6 replies
- 6 kudos
MATERIALZIED VIEW TRIGGER ON UPDATE with external table as upstream table
goal: implement event driven architecture without trigger on file arrivalI would like to know whether materialzied view can update itself when its source table which is external table updated.given that the source external table referencing data in d...
- 76 Views
- 6 replies
- 6 kudos
Latest Reply
- 6 kudos
@Khaja_Zaffer That is what is mentioned in the Databricks Docs. I have asked our SA to enable this feature will test and share more details.
- 6 kudos
- 35 Views
- 4 replies
- 0 kudos
Azure not getting listed in create external location
Hi,I'm trying to create a pipeline using Azure, however, Azure is not getting listed in the drop down of Catalog Explorer -> Create External Location. I'm using community version for practice. Please advice.
- 35 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
@AanchalSoni Yes Databricks free edition have this limitation you cannot create customized external location It has only support for s3 now and that location is managed by databricks. Look at the limitations below.
- 0 kudos
- 34 Views
- 1 replies
- 1 kudos
Issue with Databricks Jobs: SQLSTATE: XXKST
Hi,we have our Databricks Jobs deployed via DABs, and they have been running fine for a while now (approximately 1 month since we migrated from ADF). However, since yesterday, we are getting a weird issue while writing. See error below:[STREAM_FAILED...
- 34 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @BMex There are more users reporting similar issues recently:https://community.databricks.com/t5/data-engineering/intermittent-failures-with-auto-loader-on-azure/m-p/131716I’ll wait for others in the community to help shed more light on this.
- 1 kudos
- 62 Views
- 5 replies
- 5 kudos
Resolved! Databricks Free Edition - AI Agent Export Limitation
Hi, I am following the Databricks AI Agent guide to create a custom agent using my functions. However, in the Free Edition, I do not see the option to export the agent from the Playground. Is this feature unavailable in Free Edition, or is there a wo...
- 62 Views
- 5 replies
- 5 kudos
Latest Reply
- 5 kudos
Ok @atharvaghodekar , so there's no export button (neither on premium workspace as @WiliamRosa stated above).I think they refactor UI and now the same functionality is under Tools -> Create agent notebook. So, in tutorial export button is reposonsib...
- 5 kudos
- 838 Views
- 2 replies
- 0 kudos
Materialized Views Incremental Load
My question is: can materialized views be updated incrementally. For example:In my case, we store the data in Iron layer and it gets flattened in bronze and silver layer as separate tables. The required transformations happen from silver to gold laye...
- 838 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
So, I am using DLT declarative framework to work with MV in the gold layer. I am sharing code for the sample. So, can someone tell me that how we can do only the incremental refresh, as this code is doing a full refresh daily. I don't to want do agai...
- 0 kudos
- 1943 Views
- 4 replies
- 1 kudos
Incremental refresh of materialized view in serverless DLT
Hello, Every time that I run a delta live table materialized view in serverless , I get a log of "COMPLETE RECOMPUTE" . How can I achieve incremental refresh in serverless in DLT pipelines?
- 1943 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
So, I am using DLT declarative framework to work with MV in the gold layer. I am sharing code for the sample. So, can someone tell me that how we can do only the incremental refresh, as this code is doing a full refresh daily. I don't to want do agai...
- 1 kudos
- 436 Views
- 5 replies
- 0 kudos
PERMISSION_DENIED: Endpoint databricks-claude-3-7-sonnet is not allowed
I am working on a use case in my personal Databricks account - AgenticDataPipeline. I am getting below error when using Anthropic Claude 3.7 Sonnet AI model15:52:56.267 | ERROR | [Pipeline] Task failed: ACCOUNT DATA NORMALIZATION: ops_bronze.account...
- 436 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @mohnishbhagde,As @szymon_dybczak mentioned, there have been some instabilities when using this endpoint. As an alternative, you could try another model such as Llama or Claude 4, or even configure an external model pointing directly to the Anthro...
- 0 kudos
- 149 Views
- 3 replies
- 0 kudos
Intermittent failures with auto loader on Azure
We've been using auto loader to ingest data from a storage account on Azure (format "cloudFiles").Today, we're starting to see failures during the setup of event notification:25/09/11 19:06:28 ERROR MicroBatchExecution: Non-interrupted exception thro...
- 149 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
These are the current role assignments of this service principal:Seems to be right and also:This is just an intermittent error;There's an event subscription on the storage queue (with a matching query id from the error message).Could it be that someh...
- 0 kudos
- 71054 Views
- 194 replies
- 17 kudos
Suspended Exam? Don't worry! Our support team can help!
Taking exams is so stressful already, and if your exam gets suspended it is extra stressful. Don't worry! Our support team is awesome and will help you get back on track. The most important step is to file a ticket in our Help Center. Be sure to inc...
- 71054 Views
- 194 replies
- 17 kudos
Latest Reply
- 17 kudos
Dear Team,I have faced the same issue of exam suspension of Databricks Machine Learning Certified Professional.I've raised the ticket through help center with request00737502.Can someone help me to re-schedule my exam on 12th September 2025 any time ...
- 17 kudos
- 55 Views
- 2 replies
- 1 kudos
Permission issue for pandas to read local files
I can use pandas to read local files in a notebook, such as those located in tmp.However, when I run two consecutive notebooks within the same job and read files with pandas in both, I encounter a permission error in the second notebook stating that ...
- 55 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @liu The fact that you're getting a permissions issue when trying to access the same location in different notebooks is indeed confusing. Here's my take on what could be the cause of this issue:Jobs in databricks default to creating their own s...
- 1 kudos
- 125 Views
- 4 replies
- 6 kudos
Resolved! Databricks clusters unresponsive
Hello everyone,we are experiencing issues on one of our Databricks workspaces:Notebooks and SQL queries are executing, but results are not returned to the UI.On the screenshots you can see examples where cells in notebooks and queries in a SQL wareho...
- 125 Views
- 4 replies
- 6 kudos
Latest Reply
- 6 kudos
Thanks a lot for the quick replies!It seems the issue was related to a faulty iPython version (or something similar inside the workspace). The problem resolved itself later the same day in the evening, and everything is working normally again now.
- 6 kudos
- 170 Views
- 6 replies
- 0 kudos
Create external tables with properties set in delta log and no collation
There is an external delta lake table that need to be mounted on to the unity catalogIt has some properties configured in the _delta_log folder alreadyWhen try to create table using CREATE TABLE catalog_name.schema_name.table_name USING DELTA LOCATIO...
- 170 Views
- 6 replies
- 0 kudos
Latest Reply
- 0 kudos
@santosh-santosh Did you execute the steps part by part i shared in the DM ? Step 0: Define your external tables, Step 1: Inspect external Delta table schema & properties Success Check:Ensure all expected properties are captured Step 2: Inspect Uni...
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2088 | |
| 1018 | |
| 939 | |
| 670 | |
| 625 |
Latest from our Blog
[CUSTOMER BLOG] Scaling Geospatial Analytics: Efficient Visualization of Massive Spatial Datasets
Introduction If you’ve ever tried visualizing vast geospatial datasets—such as billions of points—you’ve likely run into performance bottlenecks or high costs. At S&P Global Commodity Insights, we set...
FinOps and Databricks System Tables
Original post published on 30 May 2024, here.FinOps and Databricks System Tables Dear all, I have to admit that FinOps and System Tables are not the most interesting topic to talk about it, but everyt...
How to Structure Unity Catalog Like a Pro: Real-World Hierarchy Patterns for Data Teams
After working with dozens of organizations migrating to Unity Catalog, I've seen the same pattern repeat: teams start with good intentions but end up with a tangled mess of catalogs, schemas, and tabl...
Integrating ASPICE, the V-Model, and AI-Driven Data Analytics in Automotive Software Development
Modern vehicles have shifted from mechanical machines to sophisticated software-driven systems. This transformation demands rigorous processes and innovative tools to ensure software quality, safety, ...
