Hey Databricks Community! Join us for a free community-sponsored BrickTalk where we’ll explore Multi-Cloud Data Sharing with OpenSharing SecureConnect & Global Distribution. BrickTalks is a community event series where Databricks experts share real-w...
- 231 Views
- 0 replies
- 0 kudos
Wednesday
Databricks Advanced Learning Festival November 10 – December 8, 2026 Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are ...
- 37803 Views
- 58 replies
- 11 kudos
10-08-2026
NOVEMBER · THE TOUR IS ON THE MOVE Nine stops. Three regions.One epic November. THIS MONTH'S ROUTE Munich → Atlanta → Amsterdam → London → Toronto → Los Angeles (Anaheim) → New York City → Milan → Tokyo A free, one-day event on how organizations...
- 3233 Views
- 1 replies
- 1 kudos
3 weeks ago
DATA + AI LIVE Time: AMER Nov 11 / 9 AM PT / 12 PM ET APAC Nov 12 / 9:30 AM IST / 12 PM SGT / 1 PM JST-KST / 3 PM AEDT EMEA Nov 17 / 9 AM GMT / 10 AM CET Register for the Workshop Note: Marking RSVP on this Community event does not register you fo...
- 2446 Views
- 0 replies
- 2 kudos
3 weeks ago
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 4600 Views
- 0 replies
- 4 kudos
09-01-2026
Predicting patient disease risk and measuring quality of care are important to precision health. This Solution Accelerator demonstrates a machine learning workflow using healthcare data to identify clinical and demographic factors associated with dis...
- 112 Views
- 0 replies
- 0 kudos
Wednesday
Recap • September 14 – October 11 Community Pulse Four weeks of articles, learning content, and conversations – the highlights, all in one read. Community Stars The members who showed up and made a real difference : @ivanvyd @data_pulse @Tho...
- 1040 Views
- 0 replies
- 4 kudos
a month ago
Community Activity
- 147 Views
- 4 replies
- 0 kudos
Lakebase Data API
We need to bring the json data from external API and need to load it into lakebase API tables.Please suggest in connecting this.
- 147 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Jack17,One small addition to the service principal advice above, because it makes testing a lot easier. Since 12 November, OAuth on-behalf-of for service principals is generally available, so you can test the Data API locally as the same service ...
- 0 kudos
- 150 Views
- 2 replies
- 1 kudos
How do you test data quality in a Delta Lake pipeline?
I'm exploring practical approaches to validating data quality in Databricks pipelines.For pipelines that ingest data into Delta tables, what checks do you typically automate at each stage?I'm particularly interested in handling:Schema changes and sch...
- 150 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @AlexMorgan,Mateus covered the checks really well, so I'll just add how I'd answer the PySpark tests vs expectations part, because the two do different jobs:Tests check your code. Unit-test the transformation logic in CI on small, hand-made DataFr...
- 1 kudos
- 240 Views
- 2 replies
- 0 kudos
Masking Columns for a Client Workspace When You Can’t Use Catalog Binding
Hi Everyone,I am Adnan and I am a Cloudera Platform expert recently joined the Databricks Community. I will be sharing all my implementations and experience I have gained as Big Data Expert and any solutions I implement to address my current business...
- 240 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @AdnanFaridKhan,Really nice write-up, and @rachelwilson's question is the one I'd worry about too. With tag-driven policies, a tag that never arrives fails open: the table simply shows every row.What I'd add to the sync job is a fail-closed step. ...
- 0 kudos
- 307 Views
- 4 replies
- 8 kudos
Memory Profiling
How do you debug OOM on long-running streaming jobs?Hi All,One of my Structured Streaming jobs has started dying with OutOfMemoryError. Runs fine for about approx 40 hours, then falls over, restarts, and does the same thing again. I had introduced fe...
- 307 Views
- 4 replies
- 8 kudos
Latest Reply
- 8 kudos
Hi Kiran,That's a really thorough round of checks. A flat heap floor, flat RocksDB and a small state store rule out the usual suspects, which narrows things down a lot.The one thing I'd grab next is the exact OutOfMemoryError line from the driver log...
- 8 kudos
- 131 Views
- 2 replies
- 0 kudos
Genie agents and Unity Catalog wikis: what I tested, and what to use instead
I tested whether a Genie agent picks up a business rule from a Unity Catalog wiki (Beta). Short answer: not today. A Genie agent works from the data, the table and column comments, and the context you configure in the agent. Here's the test and what ...
- 131 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Really useful test, especially the revenue example. It highlights an important gap between documenting business rules and making sure Genie actually follows them. I agree that metric views are a much better approach for defining reusable business log...
- 0 kudos
- 58 Views
- 1 replies
- 0 kudos
Issue with CDC ingestion pipeline
Hello everyoneI'm trying to setup an ingestion pipeline from a mysql server and keep running into an issue. The setup was validated and the pipeline created. When I run the pipeline the destination tables are created in the schema, but then the follo...
- 58 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @MichaelAnckaert,The interesting part is the path in the error: s3:/ with nothing after it. That isn't coming from MySQL. It looks like the pipeline tried to resolve a storage location in Unity Catalog and got back an empty S3 path, and since the ...
- 0 kudos
- 100 Views
- 2 replies
- 1 kudos
Resolved! SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading
Hello everyone,Around 3PM EST yesterday (2026-11-12), dozens of Lakeflow jobs started failing on both classic and serverless compute with distinct symptoms1. On classic compute (DBR 15.4), reading a file from S3 would result in a PATH NOT FOUND error...
- 100 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @ifishbein, I dug in and here is what I found. The serverless behavior is explainable, and your workaround is the right one. Serverless runs on Spark Connect, and in the Spark Connect flavor of PySpark SparkSession.getActiveSession() is backed ...
- 1 kudos
- 65 Views
- 1 replies
- 0 kudos
Databricks best practices for ad-hoc analytics alongside CI/CD
Hi everyone,We currently use CI/CD (Databricks Asset Bundles + Git) for deploying notebooks, jobs and pipelines across DEV and PROD.However, we also have frequent ad-hoc requests from business teams (customer lists, segmentation, one-time analysis), ...
- 65 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello billie17, I took a look at both internal and external documentation and here is what I found. Short version: yes, binding the PROD catalog read-only to a non-prod workspace is a documented pattern. The binding docs list "read-only access to pro...
- 0 kudos
- 1150 Views
- 11 replies
- 14 kudos
How to Connect Databricks to Snowflake & Query Live Data?
Hi everyone,I am looking for a method to connect Databricks to Snowflake so I can query and transform the data directly in a notebook. I specifically want to avoid methods that just perform a one-time copy of the tables into Databricks, as I need the...
- 1150 Views
- 11 replies
- 14 kudos
Latest Reply
- 14 kudos
@raj , if you are happy with the answer and it addresses your issue please mark "Accept as Solution" so that others can have confidence in the response. Cheers, Lou.
- 14 kudos
- 383 Views
- 2 replies
- 2 kudos
Data Classification on Foreign Catalog
Can Data Classification be enabled for a foreign catalog? I haven't found any documentation that says if we are able to or not. Thank you,Mark
- 383 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
@mark_o , if you are happy with the answer and it addresses your issue please mark "Accept as Solution" so that others can have confidence in the response. Cheers, Lou.
- 2 kudos
- 199 Views
- 4 replies
- 2 kudos
Serverless declarative pipeline ingestion from ADLS is running x4 SLOWER randomly
Hi all,A full refresh of one Auto Loader streaming table in a serverless #Lakeflow (Declarative) pipeline has become about 4x slower (all to do with a single streaming table), and I'd like help understanding why. Nothing on our side changed. The onl...
- 199 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @rjmcbride what if you fiddle around wit max worksers for your serverless here
- 2 kudos
- 68 Views
- 1 replies
- 0 kudos
- 68 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
It's a green pigment in leaves performing photosynthesis
- 0 kudos
- 94 Views
- 0 replies
- 0 kudos
Announcement | The new Genie: autonomous AI for complex enterprise work
Genie is evolving into an autonomous AI coworker that can take on long-running enterprise tasks, not just answer one-off questions. It can monitor progress, follow up when information is missing, and act as new information arrives, within the guardra...
- 94 Views
- 0 replies
- 0 kudos
- 47 Views
- 0 replies
- 0 kudos
Issues making requests to Databricks from AWS Lambda
Hello,I am getting acquainted with Databricks, of which I have signed up for a free account. I have a Lambda function using Python 3.14 and the databricks-sql-connector library that runs on a schedule, to take data from a file in S3 and put the diffe...
- 47 Views
- 0 replies
- 0 kudos
- 197 Views
- 1 replies
- 6 kudos
Private Connectivity from Databricks Serverless on AWS to Your On-Prem Data Center
How to connect Databricks serverless workloads to on-premises databases and services over a private network path, using Network Connectivity Configurations and AWS PrivateLink. Why private connectivity to on-prem matters With serverless compute, you ...
- 197 Views
- 1 replies
- 6 kudos
Latest Reply
- 6 kudos
Great post, Jose! Thanks for sharing it!
- 6 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1516 | |
| 1255 | |
| 1190 | |
| 961 |
Latest from our Blog
From Kyriba to Lakehouse: A Near-Real-Time Treasury Data Pipeline on Databricks
The two-speed problem of treasury data Corporate treasury asks the same question at two speeds. Where is our money right now? And where was it at the close of business? For many treasury teams the ans...
How to Query Unity Catalog Views from External Engines
Summary External engines can now query Unity Catalog views. Views over Delta tables are served through the Unity REST API (Generally Available). Views over Iceberg tables are served through the Iceber...
Private Connectivity from Databricks Serverless on AWS to Your On-Prem Data Center
How to connect Databricks serverless workloads to on-premises databases and services over a private network path, using Network Connectivity Configurations and AWS PrivateLink. Why private connectivit...
Reinforcement Learning on Databricks: Building a Contextual Bandit
Is MLflow really built only for supervised learning, or can it handle something like reinforcement learning? A customer asked me exactly this. They're running a contextual-bandit-style system on-premi...
