Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, October 29! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and ...
- 648 Views
- 0 replies
- 2 kudos
2 weeks ago
Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, November 5! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and ...
- 241 Views
- 0 replies
- 2 kudos
Thursday
NOVEMBER · THE TOUR IS ON THE MOVE Nine stops. Three regions.One epic November. THIS MONTH'S ROUTE Munich → Atlanta → Amsterdam → London → Toronto → Los Angeles (Anaheim) → New York City → Milan → Tokyo A free, one-day event on how organizations...
- 161 Views
- 1 replies
- 0 kudos
yesterday
DATA + AI LIVE Time: AMER Nov 11 / 9 AM PT / 12 PM ET APAC Nov 12 / 9:30 AM IST / 12 PM SGT / 1 PM JST-KST / 3 PM AEDT EMEA Nov 17 / 9 AM GMT / 10 AM CET Register for the Workshop Note: Marking RSVP on this Community event does not register you fo...
- 610 Views
- 0 replies
- 1 kudos
Wednesday
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 3070 Views
- 0 replies
- 4 kudos
09-01-2026
Community Champion • September 2026 Meet Venugopal Dabbara Every month, our Community Champion Program recognizes a member who makes the Databricks Community better for everyone, through their knowledge, their willingness to help, and their commitmen...
- 880 Views
- 3 replies
- 7 kudos
3 weeks ago
Responsible gaming initiatives are becoming increasingly important as regulation, gameplay experiences and the betting and gaming industry continue to expand. The Responsible Gaming Solution Accelerator, developed in partnership with AWS, shows how t...
- 198 Views
- 1 replies
- 3 kudos
Thursday
Recap • September 14 – October 11 Community Pulse Four weeks of articles, learning content, and conversations – the highlights, all in one read. Community Stars The members who showed up and made a real difference : @ivanvyd @data_pulse @Tho...
- 526 Views
- 0 replies
- 4 kudos
2 weeks ago
Community Activity
- 92 Views
- 2 replies
- 0 kudos
Avoid AWS SDK 1.x dependencies in cluster classpaths
Hi, As part of an AWS well architected review, we received a recommendation to stop using AWS SDK 1.x since this version has been deprecated since end of 2025. A cloud trail search showed that this lib is still in use in our data AWS accounts. In the...
- 92 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Please refer to the website for details on AWS SDK v1 dependencies in Databricks Runtime 18.2. For the migration timeline and recommended DBR version, we should confirm the latest guidance with Databricks Support.
- 0 kudos
- 320 Views
- 4 replies
- 0 kudos
Resolved! Tools to parse Informatica PowerCenter XML to Databrics PySpark ?
Hi all,We're planning a move from legacy PowerCenter to Databricks.Creating Delta Tables in simple enough, but manually converting dozens of PowerCenter mappings, nested mapplets, & expression logic into PySpark is eating up way too much dev time.Bef...
- 320 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
We hit this exact same issue moving PowerCenter logic over to Databricks. Hand translating all those nested mapplets and expression transformations into PySpark is a huge drain on time.We used a tool named Pulse Convert to read our .xml exports and g...
- 0 kudos
- 95 Views
- 2 replies
- 0 kudos
Cluster environment variables not visible to Java Dataset.foreachPartition / mapPartitions
We run a Spark JAR task (Java) on a standard access mode cluster. Credentials come from cluster environmentvariables set as secret references in the cluster's Environment variables field, e.g.MY_PASSWORD={{secrets/<scope>/<key>}}.The variables are vi...
- 95 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @capoorhimanshu On standard access mode this is expected. Scala/Java closures (foreachPartition, mapPartitions, UDFs) don't run in the executor JVM where the cluster env vars are set. They run in an isolated per-user sandbox process with a fixe...
- 0 kudos
- 80 Views
- 1 replies
- 0 kudos
Escalation Request: Ticket #01048934 | Name Correction Pending for Webassessor Certification Profile
Escalation Request: Ticket #01048934 | Name Correction Pending for Webassessor Certification Profile Databricks Training & Certification Team,I am writing to formally escalate support ticket #01048934, which was opened to correct the name on my Webas...
- 80 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello,I’m sorry you’ve had to wait so long for this issue to be resolved. I hope the Databricks Certification Team escalates your ticket to the appropriate team and corrects your name as soon as possible. It’s important to have your certification pro...
- 0 kudos
- 161 Views
- 1 replies
- 0 kudos
✈️ 9 Cities, 1 Epic November: The Data + AI World Tour is Here!
NOVEMBER · THE TOUR IS ON THE MOVE Nine stops. Three regions.One epic November. THIS MONTH'S ROUTE Munich → Atlanta → Amsterdam → London → Toronto → Los Angeles (Anaheim) → New York City → Milan → Tokyo A free, one-day event on how organizations...
- 161 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
https://www.caplight.com/investor/databricks-ventures How could you mentioned my name as co founder then were is my team
- 0 kudos
- 41 Views
- 0 replies
- 0 kudos
Co founder mentioned in Caplight
Regarding Co Founder mentioned without my permissionPlease let me know us statusI am waiting for your responsecaplight/investor/databricks-ventures
- 41 Views
- 0 replies
- 0 kudos
- 1345 Views
- 17 replies
- 1 kudos
Databricks DiscountVoucher
When does Databricks send email Discount Voucher? Because I'm waiting for today and not get it?Thank you.
- 1345 Views
- 17 replies
- 1 kudos
Latest Reply
- 1 kudos
@Advika @Jim_Anderson I haven't received the voucher after I finished all the modules. Could you check, please? Thank you.
- 1 kudos
- 85 Views
- 1 replies
- 1 kudos
Resolved! Query on Databricks apps idle time out
Question:As of Today, Databricks Apps do not appear to have a generally available (GA) setting for automatic stopping or idle timeout. Although `compute_min_instances` and `compute_max_instances` are available in the CLI, these settings are still in ...
- 85 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @A_Motia Correct, there's no GA idle-timeout or auto-stop for Apps today. An app is billed for its compute size the whole time it's in the Running state, whether anyone uses it or not. The supported workaround is to stop it yourself on a schedu...
- 1 kudos
- 340 Views
- 5 replies
- 0 kudos
Feature Request: Cross-job dependencies with business-date awareness
SummaryWorkflows can chain tasks inside one job, but there is no native way to say: “Run job X once per day, after jobs A, B, C, D and E have each completed successfully for the same business date, even though they are independent jobs with their own...
- 340 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Nice design. One option for the gate: instead of paging through the Jobs API, query the jobs system tables. It's one SQL query and works across workspaces:SELECT job_id FROM system.lakeflow.job_run_timeline WHERE job_id IN (<dimjob1..5 ids>) AND resu...
- 0 kudos
- 422 Views
- 13 replies
- 3 kudos
MS Fabric Direct Lake, works with delta tables only?
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
- 422 Views
- 13 replies
- 3 kudos
Latest Reply
- 3 kudos
On the open question about an existing MV: the docs only describe setting those two properties when the MV is created. So I'd treat it as a create-time setting and not count on ALTER working on an existing MV. What I'd do:1. Add the properties to the...
- 3 kudos
- 171 Views
- 4 replies
- 0 kudos
Databricks Unity Catalog Foreign Catalog – Timezone Mismatch with AWS RDS PostgreSQL Affecting Query
Hi Databricks Community,I'm facing a timezone-related issue with a Unity Catalog Foreign Catalog connected to AWS RDS PostgreSQL and would appreciate some guidance on the best way to handle it centrally.Current Setup:Source: AWS RDS PostgreSQL (confi...
- 171 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Agree there's no timezone option on the connection or foreign catalog. Rather than casting in every query, I'd fix it once in a governed view:CREATE VIEW main.gold.orders_v ASSELECTorder_id,to_utc_timestamp(created_at, 'America/Chicago') AS created_a...
- 0 kudos
- 404 Views
- 1 replies
- 0 kudos
[FGAC_ON_DEDICATED_COMPUTE_FAILED] Fine-grained access control (FGAC) on dedicated compute failed
Hi everyone, Starting this thread to check if anyone else has seen a similar issue, and sharing the solution because in our case, it was a detail we hadn't noticed it in the documentation. https://docs.databricks.com/aws/en/compute/single-user-fgac P...
- 404 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Your point about the delegation hop is a critical detail. It means that even if you want to keep workloads on dedicated classic compute, you still have to enable the serverless path workspace-wide just to handle policy evaluation.We didn’t hit this o...
- 0 kudos
- 183 Views
- 1 replies
- 2 kudos
Beyond Bronze, Silver, and Gold: When Does a Separate Stage Make Sense?
I recently published my first technical blog! I was a little nervous about sharing it, but I’m happy to take this first step. I hope to keep sharing what I learn and improve my writing along the way.For this article, I chose a question I’ve encounter...
- 183 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Congrats on the first post, Juan. Treating logic, storage, and publishing as three separate decisions is the most useful part, because most layer debates go wrong when those get collapsed into one.My default is simplicity. Start with bronze, silver a...
- 2 kudos
- 109 Views
- 2 replies
- 0 kudos
Best pattern for ingesting large (multi-GB) zip files from an external SFTP into Databricks
What I'm trying to doPull one zip file per day from an external vendor's SFTP server into Databricks. Most are a few hundred MB, but some are several GB. The zips contain .txt files. Ultimately I just need the zip landed somewhere Databricks can read...
- 109 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
@ASH1243434 For this workload, I'd recommend a scheduled Databricks Job using Paramiko → UC Volume → ZIP extraction → Auto Loader.Although the native Lakeflow SFTP connector supports binaryFile, it doesn't provide a direct raw ZIP-to-Volume transfer ...
- 0 kudos
- 57 Views
- 0 replies
- 0 kudos
Unable to see trainings for Azure Databricks Platform Architect accreditation
Hello, I am wanting to get the Azure Platform Architect Badge as mentioned here: https://www.databricks.com/learn/certification/databricks-on-azure. However, the self-paced training it says to take for this accreditation (Azure Databricks Data + AI P...
- 57 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1504 | |
| 1255 | |
| 1189 | |
| 961 |
Latest from our Blog
End-to-End Cyber Lakehouse Streaming via Zerobus Ingest, Spark Declarative Pipelines, and OCSF
Streaming raw events into a cyber lakehouse and normalizing them to a standard schema is a common pattern for security analytics. This blog walks through an end-to-end architecture that uses Databrick...
AI Functions for Synthetic Test Data: Generate Realistic Datasets in Pure SQL
The views expressed in this post are my own and do not represent those of my employer. Not long ago, generating realistic synthetic data meant building custom pipelines with GANs, Transformers, or RN...
Databricks maintenance and cleanup: Visualize, Clean, and Log with Declarative Automation Bundles
Introduction Managing Databricks environments can quickly become overwhelming as jobs, AI Search indexes, and pipeline artifacts accumulate, making manual cleanup, logging, and compliance difficult. T...
Locate, Measure, Judge: A Pattern for Verifiable Visual AI on Databricks (Part 2)
Part 1 introduced the pattern for verifiable visual AI. Let the model find the evidence, use deterministic code to measure it, apply governed rules to produce a pass, review, or fail verdict, and send...
