Grow Your Skills and Earn Rewards! Mark your calendar: January 09 – January 30, 2026 Join us for a three-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. ...
- 40425 Views
- 110 replies
- 59 kudos
a month ago
June 15–18, 2026 Are you building the future with data and AI? Then this is your moment. The Call for Proposals for Data + AI Summit 2026 is officially open, and we want to hear from builders, practitioners, and innovators across the data and AI com...
- 4392 Views
- 4 replies
- 6 kudos
3 weeks ago
How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 3694 Views
- 2 replies
- 9 kudos
10-02-2025
Strengthening cybersecurity isn’t a one-time project — it’s an ongoing process of adapting to new and complex threats. The focus is on modernizing your data infrastructure and analytics capabilities to make security operations smarter, faster, and mo...
- 328 Views
- 1 replies
- 2 kudos
Tuesday
2025 wasn’t just another year on the Databricks Community.It was a year of showing up, helping each other, and building trust. Watch the 2025 Community Wrap-Up This video is a small reflection of that journey.It highlights the key contributors of 2...
- 442 Views
- 12 replies
- 17 kudos
yesterday
The calendar changed. The conversations didn’t. The year turned somewhere between a solved thread and a new question. No pause. No reset – just learning, continuing right where it left off. Here’s how it unfolded Voices That Moved the Conversatio...
- 132 Views
- 2 replies
- 4 kudos
Tuesday
Community Activity
- 17 Views
- 0 replies
- 0 kudos
Runtime 18 / Spark 4.1 improvements
Runtime 18 / Spark 4.1 brings Literal string coalescing everywhere, thanks to what you can make your code more readable. Useful, for example, for table comments #databricks Latest updates: read: https://databrickster.medium.com/databricks-news-week-1...
- 17 Views
- 0 replies
- 0 kudos
- 984 Views
- 1 replies
- 1 kudos
Resources to practice
Hello Team,I am looking for couple of useful resources to upskill myself on delta late and pyspark performance optimization on databricks. Can anyone suggest or recommend where I can get those resources?Thanks,Kailash
- 984 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Sign up for the Databricks Free Edition. https://www.databricks.com/learn/free-edition Lots of learning material available.
- 1 kudos
- 139 Views
- 2 replies
- 5 kudos
Community Fellows: December Shout-Out to our Top Bricksters in Community!
Our Community Fellows are Bricksters who consistently provide quick, accurate customer help and high quality content. This internal program recognizes those who go above their daily work to share knowledge, improve our forums, and support the Databri...
- 139 Views
- 2 replies
- 5 kudos
Latest Reply
- 5 kudos
Woohoo! Amazing to see the Databricks Community voices growing stronger – only getting louder from here .
- 5 kudos
- 153 Views
- 1 replies
- 0 kudos
Databricks Governance Dashboards
I am looking for any prebuilt governance dashboards. I see in some demo videos a governance portal. Is that or similar available to load to our environments. I am aware of the data quality and profiling, but a single view of key indicators would be a...
- 153 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Governance Hub is in Private Preview. You can contact your account team to enroll your account for it. My 2 cents, it's pretty cool and very useful.
- 0 kudos
- 17 Views
- 1 replies
- 1 kudos
Changing of name
Dear team,My name is B Ravishankar in Aadhar and in Databricks Account it is Ravishankar B will it affect the exam I need to take exam tommorow If affect help me to change my name
- 17 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Please open a ticket to the Training support team. They can advise best. https://help.databricks.com/s/contact-us?ReqType=training. But as far as I know this should not be a problem.
- 1 kudos
- 46 Views
- 5 replies
- 3 kudos
Resolved! Warehouse ID specified in job yaml file for sql tasks
My goal is to trigger an alert I have through a job, and it seems I have to specify the warehouse id within the job yaml file itself. We have different environments with different warehouse ids, and the issue is that if I specify the warehouse id in ...
- 46 Views
- 5 replies
- 3 kudos
Latest Reply
- 3 kudos
Thank you! looks like the alert_id also needs to be parametised, and I was wondering if its possible to use a job parameter to do so? If I can use the alert name then that would be great but I believe it has to be the alert id, which will be differen...
- 3 kudos
- 47 Views
- 2 replies
- 0 kudos
databricks all-purpose cluster
getting below error-Failure starting repl. Try detaching and re-attaching the notebook. while executing notebook and can see cluster have all installed lib.
- 47 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Ved88 , Usually this error occurs in case of library issues or conflict with versions. If you remove the libraries, do you still see the issue. Additionally, review the driver logs to see if you see metastore related logs, which can give insight...
- 0 kudos
- 35 Views
- 2 replies
- 0 kudos
Query does finish on serverless but will not on classic
Dear community,We are running nightly businessvaults. Last year it stopped to finish on classic wh and after testing until completion when switching to serverless wh it stayed that way but costs have increased a lot. I have been testing numerous spar...
- 35 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey @Ties , When a query reaches 100% completion but fails to commit, it usually points to a metadata bottleneck, file system contention, or a massive data skew. While the job is hanging at 100%, go to the Spark UI -> Executors tab -> Driver -> Threa...
- 0 kudos
- 140 Views
- 6 replies
- 0 kudos
Sql server setup for lakeflow sql server connector to create ingestion
When i am executing below command change instance is getting created but without lakeflow as prefix, i read the documentation and it mentioned that to track schema evolution we need to have prefix, can I please get some assistance.Command Used:EXEC d...
- 140 Views
- 6 replies
- 0 kudos
Latest Reply
- 0 kudos
and when i altered the table got below warning : WARNING: Table [dbo].[test_table] has a pre-existing capture instance named 'dbo_test_table' that was not created by lakeflow. Lakeflow will preserve this instance and create its own instance alongside...
- 0 kudos
- 174 Views
- 1 replies
- 0 kudos
Mongo Db connector - Connection timeout when trying to connect to AWS Document DB
I am on Databricks Run Time LTE 14.3 Spark 3.5.0 Scala 2.12 and mongodb-spark-connector_2.12:10.2.0. Trying to connect to Document DB using the connector and all I get is a connection timeout. I tried using PyMongo, which works as expected and I can ...
- 174 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
If PyMongo works but the Spark connector times out, the issue is almost always JVM TLS configuration or executor-level network access, not credentials or the database itself. TLS handling (most common cause):The MongoDB Spark connector runs on the JV...
- 0 kudos
- 51 Views
- 6 replies
- 3 kudos
Resolved! Result Difference Between View and Manually Run View Query
I am experiencing an issue where a view does not display the correct results, but running the view query manually in either a new notebook or the SQL Editor displays different, correct results. I have tried switching the compute resource in the noteb...
- 51 Views
- 6 replies
- 3 kudos
Latest Reply
- 3 kudos
There are 2 fixes that I can think off Option A: Make first_value deterministic first_value(Customer_ID, true) OVER ( PARTITION BY customer_name ORDER BY submitted ASC, event_id ASC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ) U...
- 3 kudos
- 161 Views
- 1 replies
- 1 kudos
Declarative Pipelines meets foreachBatch: Custom Streaming for Advanced Pipelines
If this is the first time you’re hearing about Sinks in Lakeflow Declarative Pipelines (LDP), we highly recommend reading Introducing the SDP Sink API, which explores the recently launched Sinks API, offering the ability to seamlessly sink data to ex...
- 161 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Great writeup @harshapasala , The @dp.foreach_batch_sink decorator is a great example of what I love about modern Databricks pipelines: declarative on the surface, but with real imperative muscle underneath. One sink, three destinations, and suddenl...
- 1 kudos
- 25 Views
- 1 replies
- 0 kudos
Salesforce Connector SCD2 - Get new record with isDeleted = true on deletion
Hi all,I'm using the Databricks Salesforce connector to ingest tables with history tracking enabled (SCD Type 2).When records are deleted in Salesforce, the connector closes the existing record by setting the end date. The isDeleted flag remains fals...
- 25 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Greetings @AcrobaticMonkey , I put on my researcher hat and dug into our internal docs. Here is what I found: Short answer: this isn’t configurable today. The connector’s SCD Type 2 behavior “closes” a record by setting __END_AT and does not emit a ...
- 0 kudos
- 30 Views
- 1 replies
- 0 kudos
Structured streaming for iceberg tables
According to this https://iceberg.apache.org/docs/latest/spark-structured-streaming/ , we can stream from iceberg tables. I have ensured that my source table is Iceberg version 3, but no matter what I do, I get Iceberg does not streaming reads. Looki...
- 30 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Greetings @SaugatMukherjee , I did some research and this is what I found. You’re running into a real (and documented) Databricks limitation here: managed Iceberg tables cannot be used as a streaming source today. That’s true even though upstream Ap...
- 0 kudos
- 51 Views
- 1 replies
- 1 kudos
github repository for the databricks provided generative AI pathway
Team,Is there anyway i can find the github repository of the trainer who provided the training for the Generative AI engineering pathway.I am looking for the code and the slides provided in the course pathway.Thank you
- 51 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @viky3110! Currently, the slides used in the courses aren’t available for download. If you’re enrolled in the free self-paced course, please note that the hands-on labs used by trainers are not included in those offerings.You can access the lab...
- 1 kudos
Announcements
| User | Count |
|---|---|
| 2230 | |
| 1221 | |
| 1068 | |
| 950 | |
| 939 |
Latest from our Blog
Declarative Pipelines meets foreachBatch: Custom Streaming for Advanced Pipelines
If this is the first time you’re hearing about Sinks in Lakeflow Declarative Pipelines (LDP), we highly recommend reading Introducing the SDP Sink API, which explores the recently launched Sinks API, ...
[PARTNER BLOG] Building an Agent-Native Data Quality Manager with Databricks Apps and DQX
Introduction: The Data Quality ParadoxWhat We BuiltThe Technology StackArchitecture Deep DiveThe Four-Layer ArchitectureLayer 1: UsersLayer 2: Databricks Apps PlatformLayer 3: Compute LayerLayer 4: Da...
[PARTNER BLOG] Zerobus Ingest on Databricks
Introduction TL;DR ZeroBus Ingest is a serverless, Kafka-free ingestion service in Databricks that allows applications and IoT devices to stream data directly into Delta Lake with low latency and mini...
[PARTNER BLOG] Beyond ADLS Limits: File Arrival Triggers for File Updates
Beyond ADLS Limitations: Making File Arrival Triggers Work for Existing File Updates Using a Flag File MechanismThe Flag File MechanismThe Root Problem: Triggers Only Work on “Create”, Not “Modify” Ev...
