Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, October 29! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and ...
- 487 Views
- 0 replies
- 1 kudos
a week ago
DATA + AI LIVE Time: AMER Nov 11 / 9 AM PT / 12 PM ET APAC Nov 12 / 9:30 AM IST / 12 PM SGT / 1 PM JST-KST / 3 PM AEDT EMEA Nov 17 / 9 AM GMT / 10 AM CET Register for the Workshop Note: Marking RSVP on this Community event does not register you fo...
- 74 Views
- 0 replies
- 0 kudos
10 hours ago
DATABRICKS Asia-Pacific Learning Festival Free, virtual, hands-on Databricks training 21 & 22 October 2026 Virtual, labs included Free Ready to level up your Databricks skills? The Databricks Asia-Pacific Learning Festival is back, and ...
- 2637 Views
- 1 replies
- 1 kudos
2 weeks ago
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 2732 Views
- 0 replies
- 3 kudos
09-01-2026
Free · Hands-on · Instructor-led AI/BI + Genie Launchpad From dashboards your team can trust to answers in plain language with Genie. A three-day arc of live, hands-on sessions, each built for a different role. Join only the day or days that fit you...
- 1089 Views
- 0 replies
- 0 kudos
2 weeks ago
Community Champion • September 2026 Meet Venugopal Dabbara Every month, our Community Champion Program recognizes a member who makes the Databricks Community better for everyone, through their knowledge, their willingness to help, and their commitmen...
- 732 Views
- 3 replies
- 6 kudos
2 weeks ago
Genome-wide association studies help identify genetic variations associated with a disease or trait. The Genome-Wide Association Studies Solution Accelerator builds on Glow, an open source project for genomic analysis, to provide a fast, scalable and...
- 285 Views
- 0 replies
- 1 kudos
a week ago
Recap • September 14 – October 11 Community Pulse Four weeks of articles, learning content, and conversations – the highlights, all in one read. Community Stars The members who showed up and made a real difference : @ivanvyd @data_pulse @Tho...
- 357 Views
- 0 replies
- 3 kudos
Thursday
Community Activity
- 38 Views
- 1 replies
- 0 kudos
False positive INGESTION_GATEWAY_SQL_SERVER_AGENT_NOT_RUNNING on Azure SQL Managed Instance
Issue: Pipeline failed with INGESTION_GATEWAY_SQL_SERVER_AGENT_NOT_RUNNING, but the SQL Server Agent on Azure SQL Managed Instance is always running and cannot be stopped. All CDC prerequisites are confirmed valid:sys.databases: is_cdc_enabled = 1 fo...
- 38 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @vsrekha213 This is most likely a permissions problem rather than the Agent itself. The gateway checks Agent/capture-job status while logged in as the connector's database user. If that login can't see server-level state or the msdb job tables, th...
- 0 kudos
- 56 Views
- 1 replies
- 0 kudos
Clarification on use_describe_as_json_for_relation_metadata and system.information_schema dependency
Hi Databricks Support Team,We are evaluating whether dbt can execute successfully without access to system.information_schema in our environment, due to metadata access restrictions.Our current setup uses:dbt-core: 1.12.3dbt-databricks: 1.12.5Configu...
- 56 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Aju_Antony Interesting question!!! Short answer: no flag removes it. In dbt-databricks 1.12.x, relation discovery for Unity Catalog is hardcoded to system.information_schema.tables (the get_uc_tables macro), and use_describe_as_json_for_relation_...
- 0 kudos
- 17 Views
- 0 replies
- 0 kudos
How can I get access to Lakehouse//RT Beta in Databricks Free Edition?
Hi Databricks Community,Now I'd like to test Storage-level Transcoding + Lakehouse//RTIs Lakehouse//RT available in Free Edition or Free Trial? If not, how can I request Beta access for hands-on learning?Thanks for your help!
- 17 Views
- 0 replies
- 0 kudos
- 1375 Views
- 17 replies
- 3 kudos
Change data feed from a materialized view
Hi everyone,Trying the CDF on materialized view (declarative pipeline) feature that is currently in Beta.- Configuration:** Verified the MVs (materialized veiws) have: 'delta.enableRowTracking'SHOW TBLPROPERTIES my_mv ('delta.enableRowTracking'); re...
- 1375 Views
- 17 replies
- 3 kudos
Latest Reply
- 3 kudos
I tried reproducing a similar scenario using serverless compute in Databricks Free Edition.I tested enabling pipelines.externalMetadata.enabled=true at the pipeline level, explicitly setting it in the materialized view's TBLPROPERTIES, and also creat...
- 3 kudos
- 498 Views
- 9 replies
- 11 kudos
Datavault implemention on Databricks
Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...
- 498 Views
- 9 replies
- 11 kudos
Latest Reply
- 11 kudos
how to handle duplicates(arriving at different time and not in the same batch) when data is ingested from stage/bronze to hubs? LDP/DLT does not support merge operations and what if i have multiple source table ingesting into a single hub. Any though...
- 11 kudos
- 63 Views
- 3 replies
- 0 kudos
Best Practices required in building the projects
Hi People,I am working on an important project, can anyone guide me with the best practices which can be implemented in the projects with databricks, sql, python integrated projects. with regards,Manasa BRData Analyst
- 63 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
@ItsTechieeGirl When setting up an integrated Python and SQL project on Databricks, you need to first establish clean code structure and modularity by treating notebooks as thin orchestration layers. Place the Python functions into importable .py mod...
- 0 kudos
- 75 Views
- 3 replies
- 1 kudos
Request correting name in webassesor portal
Hello Support Team,I am writing to request an urgent escalation for Support Case 01048934.Registered Email ID: snehaumapathy424@gmail.comExam Name: Databricks Certified Data Engineer ProfessionalScheduled on : 25 October 2026Date Contact Form Submitt...
- 75 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello ,Thank you for filing a ticket with our support team, Support team will respond shortly. Please note that we cannot provide support or handle exam suspensions via community (it is not advised that you post your email address here in community)....
- 1 kudos
- 23 Views
- 0 replies
- 0 kudos
What Is – an ontology? (How Genie learns your business like a new joiner)
At Big Data London this year, I spent two days on the Unity Catalog booth and the main topic of the booth was 'Databricks Ontology. So "what is an ontology?" - here's the plain-English answer. The short answer: Think of a new joiner On day one, a new...
- 23 Views
- 0 replies
- 0 kudos
- 30 Views
- 0 replies
- 0 kudos
How do data roles overlap in your Databricks projects?
The other day, my brother asked me a simple question:“What does a Data Engineer actually do?”I started talking about pipelines, databases, and cloud platforms. His face told me that none of it was helping.So I tried a simpler explanation using a libr...
- 30 Views
- 0 replies
- 0 kudos
- 166 Views
- 5 replies
- 2 kudos
Genie One Guardrails
HiWe are opening up the Genie AI to the business users. The Genie One seems to be a good fit, but the problem I noticed is the Guardrails and influence over the single orchestration. All the questions have been nicely interrupted by Genie one and wor...
- 166 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
Short answer: instructions alone are guidance, not hard routing, so the reliable fix is to make the right answer the easiest one for Genie to find.What worked for me:1. Build a benchmark first: 20–30 real questions, 2–4 phrasings each, one correct SQ...
- 2 kudos
- 18 Views
- 0 replies
- 0 kudos
Stop Wasting Time: Use AI Dev Kit on Mac with Cursor to Automate Your Data Work
Presented by Christina Song
- 18 Views
- 0 replies
- 0 kudos
- 120 Views
- 2 replies
- 0 kudos
Best Architecture for Building a Custom Agent in Databricks?
Hi Everyone,I'm building a Databricks cost optimization agent and would appreciate some guidance on the recommended architecture.My current design consists of the following functions:Retrieve the last N job runs along with cost, DBU usage, runtime, c...
- 120 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Recommendation: use a custom code agent, deployed on Databricks Apps/Agent Runtime. Suggested architecture Precompute the data Parse Spark event logs asynchronously from the UC Volume.Store normalized run-level and stage-level metrics in Delta table...
- 0 kudos
- 88 Views
- 2 replies
- 0 kudos
Genie One Answers Likes and Dislikes Button
Genie one answers shows an option to like or dislike answers. Where do they get stored? Is there a way for me to identify how many dislikes of the Q&A pair, and perform an analysis over that pattern and come up with proper instructions, benchmark tes...
- 88 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Genie One Feedback is recorded in the workspace audit system table: system.access.audit, under service_name = 'genieChat' and action_name = 'updateGenieChatConversationFeedback'. It is associated with the conversation_id and rating.You can count disl...
- 0 kudos
- 63 Views
- 0 replies
- 0 kudos
Announcement | Transactions are Generally Available, advancing warehousing on the Lakehouse
Multi-statement transactions are now Generally Available for Unity Catalog managed Delta Lake and Apache Iceberg tables, helping teams run complex, multi-step warehouse workloads directly on governed lakehouse data. Changes across statements and tabl...
- 63 Views
- 0 replies
- 0 kudos
- 31 Views
- 0 replies
- 0 kudos
Looking for Honest Feedback on My Digital Marketing Portfolio Website
Hi everyone,I’m a fresher who recently started learning digital marketing, and I’ve built my own website as part of my learning journey:sulfathdigitalmarketer.comI’m still learning and improving, so I’d genuinely appreciate feedback from people who h...
- 31 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1504 | |
| 1255 | |
| 1188 | |
| 961 |
Latest from our Blog
Databricks maintenance and cleanup: Visualize, Clean, and Log with Declarative Automation Bundles
Introduction Managing Databricks environments can quickly become overwhelming as jobs, AI Search indexes, and pipeline artifacts accumulate, making manual cleanup, logging, and compliance difficult. T...
Locate, Measure, Judge: A Pattern for Verifiable Visual AI on Databricks (Part 2)
Part 1 introduced the pattern for verifiable visual AI. Let the model find the evidence, use deterministic code to measure it, apply governed rules to produce a pass, review, or fail verdict, and send...
Locate, Measure, Judge: A Pattern for Verifiable Visual AI on Databricks (Part 1)
AI adoption has moved much faster than measurable business results. McKinsey reports that 88% of organizations now use AI in at least one business function, yet only 39% see a bottom-line effect. A co...
Minimize disruptions for continuous pipelines with Maintenance Windows
Minimize disruptions for continuous pipelines with Maintenance Windows In an agentic world, it is more important than ever to make decisions using the latest available data. To achieve this, data engi...
