Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, October 29! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and ...
- 636 Views
- 0 replies
- 2 kudos
2 weeks ago
Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, November 5! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and ...
- 224 Views
- 0 replies
- 2 kudos
Thursday
NOVEMBER · THE TOUR IS ON THE MOVE Nine stops. Three regions.One epic November. THIS MONTH'S ROUTE Munich → Atlanta → Amsterdam → London → Toronto → Los Angeles (Anaheim) → New York City → Milan → Tokyo A free, one-day event on how organizations...
- 117 Views
- 0 replies
- 0 kudos
12 hours ago
DATA + AI LIVE Time: AMER Nov 11 / 9 AM PT / 12 PM ET APAC Nov 12 / 9:30 AM IST / 12 PM SGT / 1 PM JST-KST / 3 PM AEDT EMEA Nov 17 / 9 AM GMT / 10 AM CET Register for the Workshop Note: Marking RSVP on this Community event does not register you fo...
- 588 Views
- 0 replies
- 1 kudos
Wednesday
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 3048 Views
- 0 replies
- 4 kudos
09-01-2026
Community Champion • September 2026 Meet Venugopal Dabbara Every month, our Community Champion Program recognizes a member who makes the Databricks Community better for everyone, through their knowledge, their willingness to help, and their commitmen...
- 866 Views
- 3 replies
- 7 kudos
3 weeks ago
Responsible gaming initiatives are becoming increasingly important as regulation, gameplay experiences and the betting and gaming industry continue to expand. The Responsible Gaming Solution Accelerator, developed in partnership with AWS, shows how t...
- 185 Views
- 1 replies
- 3 kudos
Thursday
Recap • September 14 – October 11 Community Pulse Four weeks of articles, learning content, and conversations – the highlights, all in one read. Community Stars The members who showed up and made a real difference : @ivanvyd @data_pulse @Tho...
- 514 Views
- 0 replies
- 4 kudos
2 weeks ago
Community Activity
- 308 Views
- 5 replies
- 0 kudos
Feature Request: Cross-job dependencies with business-date awareness
SummaryWorkflows can chain tasks inside one job, but there is no native way to say: “Run job X once per day, after jobs A, B, C, D and E have each completed successfully for the same business date, even though they are independent jobs with their own...
- 308 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Nice design. One option for the gate: instead of paging through the Jobs API, query the jobs system tables. It's one SQL query and works across workspaces:SELECT job_id FROM system.lakeflow.job_run_timeline WHERE job_id IN (<dimjob1..5 ids>) AND resu...
- 0 kudos
- 393 Views
- 13 replies
- 3 kudos
MS Fabric Direct Lake, works with delta tables only?
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
- 393 Views
- 13 replies
- 3 kudos
Latest Reply
- 3 kudos
On the open question about an existing MV: the docs only describe setting those two properties when the MV is created. So I'd treat it as a create-time setting and not count on ALTER working on an existing MV. What I'd do:1. Add the properties to the...
- 3 kudos
- 150 Views
- 4 replies
- 0 kudos
Databricks Unity Catalog Foreign Catalog – Timezone Mismatch with AWS RDS PostgreSQL Affecting Query
Hi Databricks Community,I'm facing a timezone-related issue with a Unity Catalog Foreign Catalog connected to AWS RDS PostgreSQL and would appreciate some guidance on the best way to handle it centrally.Current Setup:Source: AWS RDS PostgreSQL (confi...
- 150 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Agree there's no timezone option on the connection or foreign catalog. Rather than casting in every query, I'd fix it once in a governed view:CREATE VIEW main.gold.orders_v ASSELECTorder_id,to_utc_timestamp(created_at, 'America/Chicago') AS created_a...
- 0 kudos
- 375 Views
- 1 replies
- 0 kudos
[FGAC_ON_DEDICATED_COMPUTE_FAILED] Fine-grained access control (FGAC) on dedicated compute failed
Hi everyone, Starting this thread to check if anyone else has seen a similar issue, and sharing the solution because in our case, it was a detail we hadn't noticed it in the documentation. https://docs.databricks.com/aws/en/compute/single-user-fgac P...
- 375 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Your point about the delegation hop is a critical detail. It means that even if you want to keep workloads on dedicated classic compute, you still have to enable the serverless path workspace-wide just to handle policy evaluation.We didn’t hit this o...
- 0 kudos
- 165 Views
- 1 replies
- 2 kudos
Beyond Bronze, Silver, and Gold: When Does a Separate Stage Make Sense?
I recently published my first technical blog! I was a little nervous about sharing it, but I’m happy to take this first step. I hope to keep sharing what I learn and improve my writing along the way.For this article, I chose a question I’ve encounter...
- 165 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Congrats on the first post, Juan. Treating logic, storage, and publishing as three separate decisions is the most useful part, because most layer debates go wrong when those get collapsed into one.My default is simplicity. Start with bronze, silver a...
- 2 kudos
- 76 Views
- 2 replies
- 0 kudos
Best pattern for ingesting large (multi-GB) zip files from an external SFTP into Databricks
What I'm trying to doPull one zip file per day from an external vendor's SFTP server into Databricks. Most are a few hundred MB, but some are several GB. The zips contain .txt files. Ultimately I just need the zip landed somewhere Databricks can read...
- 76 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
@ASH1243434 For this workload, I'd recommend a scheduled Databricks Job using Paramiko → UC Volume → ZIP extraction → Auto Loader.Although the native Lakeflow SFTP connector supports binaryFile, it doesn't provide a direct raw ZIP-to-Volume transfer ...
- 0 kudos
- 22 Views
- 0 replies
- 0 kudos
Unable to see trainings for Azure Databricks Platform Architect accreditation
Hello, I am wanting to get the Azure Platform Architect Badge as mentioned here: https://www.databricks.com/learn/certification/databricks-on-azure. However, the self-paced training it says to take for this accreditation (Azure Databricks Data + AI P...
- 22 Views
- 0 replies
- 0 kudos
- 54 Views
- 1 replies
- 0 kudos
Cluster environment variables not visible to Java Dataset.foreachPartition / mapPartitions
We run a Spark JAR task (Java) on a standard access mode cluster. Credentials come from cluster environmentvariables set as secret references in the cluster's Environment variables field, e.g.MY_PASSWORD={{secrets/<scope>/<key>}}.The variables are vi...
- 54 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @capoorhimanshu On standard access mode this is expected. Scala/Java closures (foreachPartition, mapPartitions, UDFs) don't run in the executor JVM where the cluster env vars are set. They run in an isolated per-user sandbox process with a fixe...
- 0 kudos
- 66 Views
- 0 replies
- 0 kudos
End-to-End Cyber Lakehouse Streaming via Zerobus Ingest, Spark Declarative Pipelines, and OCSF
Streaming raw events into a cyber lakehouse and normalizing them to a standard schema is a common pattern for security analytics. This blog walks through an end-to-end architecture that uses Databricks Zerobus to push events into a bronze Delta table...
- 66 Views
- 0 replies
- 0 kudos
- 35 Views
- 0 replies
- 0 kudos
Quest 5 Completed – Databricks AppQuest
I've completed Quest 5 of the Databricks AppQuest challenge.For this project, I built an AI-powered knowledge library using Databricks Apps, AppKit, Lakebase, and Databricks Model Serving.The application includes:Create, edit, and manage notes with M...
- 35 Views
- 0 replies
- 0 kudos
- 71 Views
- 0 replies
- 0 kudos
Announcement | Introducing Agent Memory on Databricks
Agent Memory is now available in Beta on Agent Bricks, Databricks’ agent developer platform. It gives agents durable sessions and long-term memory for facts, preferences, and workflows, so they can provide more personalized experiences while reducing...
- 71 Views
- 0 replies
- 0 kudos
- 46 Views
- 0 replies
- 0 kudos
Escalation Request: Ticket #01048934 | Name Correction Pending for Webassessor Certification Profile
Escalation Request: Ticket #01048934 | Name Correction Pending for Webassessor Certification Profile Databricks Training & Certification Team,I am writing to formally escalate support ticket #01048934, which was opened to correct the name on my Webas...
- 46 Views
- 0 replies
- 0 kudos
- 138 Views
- 4 replies
- 1 kudos
Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolves
From Azure Databricks serverless notebook compute (East US), uploading a file to a Unity Catalog volume with the Databricks SQL Connector PUT ... INTO '/Volumes/...' fails at TCP connect with [Errno 111] Connection refused. The warehouse accepts the ...
- 138 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
I tested this independently and was able to reproduce a very similar failure pattern on AWS us-east-2, using a Databricks serverless.Here are the results against the same UC Volume:Upload method ResultDirect /Volumes/.. writePASS (byte readback verif...
- 1 kudos
- 46 Views
- 0 replies
- 0 kudos
Avoid AWS SDK 1.x dependencies in cluster classpaths
Hi, As part of an AWS well architected review, we received a recommendation to stop using AWS SDK 1.x since this version has been deprecated since end of 2025. A cloud trail search showed that this lib is still in use in our data AWS accounts. In the...
- 46 Views
- 0 replies
- 0 kudos
- 145 Views
- 3 replies
- 0 kudos
Suspended Databricks Certified Professional Exam Without Any Reason
I am writing to formally appeal the automatic suspension of my recent Databricks Certified Professional exam session. I have been preparing hard for this exam from last 2-3 months as it is required on client side and urgent.There is other issues from...
- 145 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Anshul47b ,i am sorry to hear about your experience, especially after spending months preparing for the exam.Since the initial system checks passed but the webcam feed reportedly dropped during the session, I'm curious whether the support team ca...
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1504 | |
| 1255 | |
| 1189 | |
| 961 |
Latest from our Blog
End-to-End Cyber Lakehouse Streaming via Zerobus Ingest, Spark Declarative Pipelines, and OCSF
Streaming raw events into a cyber lakehouse and normalizing them to a standard schema is a common pattern for security analytics. This blog walks through an end-to-end architecture that uses Databrick...
AI Functions for Synthetic Test Data: Generate Realistic Datasets in Pure SQL
The views expressed in this post are my own and do not represent those of my employer. Not long ago, generating realistic synthetic data meant building custom pipelines with GANs, Transformers, or RN...
Databricks maintenance and cleanup: Visualize, Clean, and Log with Declarative Automation Bundles
Introduction Managing Databricks environments can quickly become overwhelming as jobs, AI Search indexes, and pipeline artifacts accumulate, making manual cleanup, logging, and compliance difficult. T...
Locate, Measure, Judge: A Pattern for Verifiable Visual AI on Databricks (Part 2)
Part 1 introduced the pattern for verifiable visual AI. Let the model find the evidence, use deterministic code to measure it, apply governed rules to produce a pass, review, or fail verdict, and send...
