Grow Your Skills and Earn Rewards! Mark your calendar: January 09 – January 30, 2026 Join us for a three-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. ...
- 101349 Views
- 320 replies
- 123 kudos
12-09-2025
June 15–18, 2026 Are you building the future with data and AI? Then this is your moment. The Call for Proposals for Data + AI Summit 2026 is officially open, and we want to hear from builders, practitioners, and innovators across the data and AI com...
- 6492 Views
- 4 replies
- 6 kudos
a month ago
How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 3886 Views
- 2 replies
- 9 kudos
10-02-2025
Strengthening cybersecurity isn’t a one-time project — it’s an ongoing process of adapting to new and complex threats. The focus is on modernizing your data infrastructure and analytics capabilities to make security operations smarter, faster, and mo...
- 697 Views
- 1 replies
- 2 kudos
a week ago
2025 wasn’t just another year on the Databricks Community.It was a year of showing up, helping each other, and building trust. Watch the 2025 Community Wrap-Up This video is a small reflection of that journey.It highlights the key contributors of 2...
- 1102 Views
- 13 replies
- 20 kudos
a week ago
Another strong week of engagement, learning, and problem-solving across the community.Let’s take a look at the highlights Standout Contributors These members stepped in with sharp insights, thoughtful follow-ups, and the kind of curiosity that ke...
- 87 Views
- 0 replies
- 0 kudos
yesterday
Community Activity
- 397 Views
- 1 replies
- 3 kudos
Combining Agent Bricks and ai_parse_document for Smarter Document Workflows
Building a Research Paper Curator for Knowledge Assistants Document-heavy workflows such as research analysis, contract review, and support ticket processing, are a natural fit for AI agents. Building them from scratch means orchestrating document pa...
- 397 Views
- 1 replies
- 3 kudos
Latest Reply
- 3 kudos
this is fire. thanks for sharing
- 3 kudos
- 37 Views
- 2 replies
- 1 kudos
Spark Declarative Pipelines use in All-purpose compute?
Hi there, I know Spark Declarative Pipelines (previously DLT) has undergone some changes since last year and is even now open source (announcement). For a long while, I know that SDP/DLT was locked to only working with job compute. I'm wondering, wit...
- 37 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Not today and likely not in future as well. 1. If you create a pipeline yourself, the pipeline creates it's own compute. This can be classic (ie you choose the nodes/node-types) or serverless (recommended) 2. If you define a Streaming Table or Materi...
- 1 kudos
- 21 Views
- 0 replies
- 0 kudos
Hiring: Senior Data Engineer (Azure Databricks) – Hybrid Irving
I’m looking to add a Senior Data Engineer to my team to help drive our migration from a legacy data warehouse to Azure Databricks. This is a hands-on role focused on building modern, scalable data pipelines and analytics solutions.What you’ll be work...
- 21 Views
- 0 replies
- 0 kudos
- 39 Views
- 3 replies
- 0 kudos
Date filter not working with Oracle SQL query federation
I created an external connection to an Oracle Database 12c Standard Edition Release 12.1.0.2.0.The connection is working, I executed successfully a lot of queries with different filters and joins. But I couldn't find a way to make any date filter pus...
- 39 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Can you spin up a DBR 18+ all-purpose compute cluster with this spark config set to true spark.databricks.connector.oracle.properlyCompileJava8Values true and then re-run your tests? If this works then I think it might be a known issue and we can fin...
- 0 kudos
- 298 Views
- 1 replies
- 1 kudos
PostgreSQL to Databricks made simpler with Lakeflow Connect(Public Preview)
PostgreSQL to Databricks made simpler with Lakeflow Connect (Public Preview).Databricks has introduced a PostgreSQL connector in Lakeflow Connect (Public Preview), enabling ingestion of PostgreSQL data into the Lakehouse using logical replication.Ins...
- 298 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
@Yogesh_Verma_ do you have timeline for when this will be Generally Available?
- 1 kudos
- 106 Views
- 2 replies
- 3 kudos
Why ENABLE_MLFLOW_TRACING does not work for serving endpoint?
I would like to ask you if you have experienced similar issue like me recently. I trained sklearn model. Logged this model with fe.log_model for automatic feature lookup. Online feature tables where published with currently recommended approach, whi...
- 106 Views
- 2 replies
- 3 kudos
Latest Reply
- 3 kudos
Hello @d_szepietowska , I did some research on my end and found a few helpful hints/tips to help you troubleshoot. Let’s walk through what should be happening, and then I’ll call out the most common reasons the feature lookup DataFrame doesn’t show ...
- 3 kudos
- 102 Views
- 4 replies
- 0 kudos
How to use/install a driver in Spark Declarative Pipelines (ETL)?
Salutations,I'm using SDP for an ETL that extracts data from HANA and put it in the Unity Catalog. I defined a Policy with the needed driver:But I get this error:An error occurred while calling o1013.load. : java.lang.ClassNotFoundException: com.sap....
- 102 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello,It is recommended that you upload libraries to source locations that support installation onto compute with standard access mode (formerly shared access mode), as this is the recommended mode for all workloads. Please refer the documentation ...
- 0 kudos
- 56 Views
- 2 replies
- 4 kudos
OCR Made Simple with Databricks AI Functions
Databricks MVP Conner Schiissler explores the new ai_parse_document() function, which replaces complex OCR integrations with a simple SQL command to extract structured markdown from unstructured files. Read here.
- 56 Views
- 2 replies
- 4 kudos
- 47 Views
- 2 replies
- 2 kudos
Resolved! Serverless Compute Spark Version Flexibility?
Hi there, I'm wondering what determines the Serverless Compute spark version? Is it based on the current DBR LTS? And is there a way to modify the spark version for serverless compute?For example, when I check the spark version for our serverless com...
- 47 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Serverless compute always run on the latest runtime version. You cannot choose it like in standard compute.
- 2 kudos
- 21601 Views
- 16 replies
- 7 kudos
Resolved! Require guidance to get free certification voucher
Hi team,I wanted to know , what are the steps to get the free certification voucher for the data engineer associate exam.Could anyone please help me with the same.Thanks!!
- 21601 Views
- 16 replies
- 7 kudos
Latest Reply
- 7 kudos
Generally you can get the voucher from College, Company you work for. If you are an active IEEE member they do have some budget for scholarships
- 7 kudos
- 451 Views
- 4 replies
- 3 kudos
Data Engineer Associate Certification in 2026
Hi Community,I’m planning to take the Data Engineer Associate Certification in 2026 to advance my skills and career. Are there any ongoing promotions or opportunities to get a 50% discount voucher for the exam? It would be incredibly helpful if you c...
- 451 Views
- 4 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi Community,I'm planning to take certification for Databricks data engineer associate.If possible could you please provide 50% voucher Thank you so much for your time and assistance! #Certfication, #Certification Vouchers, #Certified Data Engineer A...
- 3 kudos
- 52 Views
- 1 replies
- 0 kudos
Parity between Spark.sql and SQL Warehouse for Metric Views & Model Visualization
Hi everyone,I’m exploring the new Databricks Metric Views (Semantic Layer) and have two questions regarding programmatic management and UI visualization.1. Parser Disparity: spark.sql vs. SQL WarehouseI'm noticing that CREATE OR REPLACE VIEW ... WITH...
- 52 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
To add to #1 aboveThe following```sqlCREATE OR REPLACE VIEW catalog.schema.customer_sales_metric_view-------``` **Works**: SQL Warehouse/Editor **Fails**: PySpark spark.sql() **Fails**: Notebook %sql MagicBusiness ImpactThis limitation significantly ...
- 0 kudos
- 171 Views
- 0 replies
- 1 kudos
From Traces to Evaluation: An Autonomous MLflow Agent Harness for AI AgentOps
The Gap in Agent Development Databricks has made it straightforward to deploy AI agents—Model Serving endpoints, automatic MLflow tracing, Unity Catalog integration. But there's a gap between "deployed" and "production-ready": You have: Agent code ru...
- 171 Views
- 0 replies
- 1 kudos
- 80 Views
- 3 replies
- 0 kudos
Resolved! PERMISSION_DENIED: Request for user delegation key is not authorized.
I am attempting to copy files from an Azure Storage container using an Azure Databricks Volume. When attempting to list files using dbutils.fs.ls('/Volumes/myCatalog/mySchema/myVolume' I get the following error:ExecutionError: (com.databricks.sql.man...
- 80 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
@szymon_dybczak Can you provide a link to the documentation you noted? I confirmed this with my own testing, that the Storage Blob Delegator role must be at the ADLS account-level, and Storage Blob Data Reader can then be applied at the container-lev...
- 0 kudos
- 90 Views
- 4 replies
- 7 kudos
Resolved! Testing Spark Declarative Pipeline in Docker Container > PySparkRuntimeError
Hi there, I see via an announcement last year that Spark Declarative Pipeline (previously DLT) was getting open sourced into Apache Spark, and I see that this recently is true as of Apache 4.1:Spark Declarative Pipelines Programming Guide I'm trying ...
- 90 Views
- 4 replies
- 7 kudos
Latest Reply
- 7 kudos
Hi @ChristianRRL ,In addition to @osingh 's answers, check out this old but good blog post about how to structure the pipelines's code to enable dev and test cycle: https://www.databricks.com/blog/applying-software-development-devops-best-practices-d...
- 7 kudos
Announcements
| User | Count |
|---|---|
| 2243 | |
| 1226 | |
| 1075 | |
| 1018 | |
| 950 |
Latest from our Blog
From Traces to Evaluation: An Autonomous MLflow Agent Harness for AI AgentOps
The Gap in Agent Development Databricks has made it straightforward to deploy AI agents—Model Serving endpoints, automatic MLflow tracing, Unity Catalog integration. But there's a gap between "deploye...
Combining Agent Bricks and ai_parse_document for Smarter Document Workflows
Building a Research Paper Curator for Knowledge Assistants Document-heavy workflows such as research analysis, contract review, and support ticket processing, are a natural fit for AI agents. Building...
Declarative Pipelines meets foreachBatch: Custom Streaming for Advanced Pipelines
If this is the first time you’re hearing about Sinks in Lakeflow Declarative Pipelines (LDP), we highly recommend reading Introducing the SDP Sink API, which explores the recently launched Sinks API, ...
[PARTNER BLOG] Building an Agent-Native Data Quality Manager with Databricks Apps and DQX
Introduction: The Data Quality ParadoxWhat We BuiltThe Technology StackArchitecture Deep DiveThe Four-Layer ArchitectureLayer 1: UsersLayer 2: Databricks Apps PlatformLayer 3: Compute LayerLayer 4: Da...
