- 1002 Views
- 2 replies
- 3 kudos
Databricks Advent Calendar 2025 #11
Real-time mode is a breakthrough that lets Spark utilize all available CPUs to process records with single-millisecond latency, while decoupling checkpointing from per-record processing.
- 1002 Views
- 2 replies
- 3 kudos
- 633 Views
- 0 replies
- 0 kudos
Databricks Advent Calendar 2025 #16
For many data engineers who love PySpark, the most significant improvement of 2025 was the addition of merge to the dataframe API, so no more Delta library or SQL is needed to perform MERGE. p.s. I still prefer SQL MERGE inside spark.sql()
- 633 Views
- 0 replies
- 0 kudos
- 659 Views
- 0 replies
- 2 kudos
Databricks Advent Calendar 2025 #15
New Lakakebase experience is a game-changer for transactional databases. That functionality is fantastic. Autoscaling to zero makes it really cost-effective. Do you need to deploy to prod? Just branch the production database to the release branch, an...
- 659 Views
- 0 replies
- 2 kudos
- 7222 Views
- 1 replies
- 3 kudos
Unity Catalog Lineage: Lineage That Just Works
I've been working with Unity Catalog's lineage capabilities for a while now, and I have to say—this is what lineage should have always been. Not a separate tool to configure. Not a manual process to maintain. Just automatic, real-time visibility into...
- 7222 Views
- 1 replies
- 3 kudos
- 3 kudos
I have been using and implementing UC in various workspaces across industry, BYOL is the one I am really looking forward to implement next.Thanks @AbhaySingh for consolidating it here.
- 3 kudos
- 611 Views
- 0 replies
- 0 kudos
Databricks Advent Calendar 2025 #14
Ingestion from SharePoint is now available directly in PySpark. Just define a connection and use spark-read or, even better, spark-readStream with an autoloader. Just specify the file type and options for that file (pdf, csv, Excel, etc.)
- 611 Views
- 0 replies
- 0 kudos
- 1255 Views
- 0 replies
- 1 kudos
Databricks News: Week 50: 8 December 2025 to 14 December 2025
Excel The big news this week is the possibility of native importing Excel files. Write operations are also possible. There is a possibility of choosing a data range. It also works with the streaming autoloader, currently in beta. GPT 5.2 The same day...
- 1255 Views
- 0 replies
- 1 kudos
- 655 Views
- 0 replies
- 0 kudos
Databricks Advent Calendar 2025 #13
ZeroBus changes the game: you can now push event data directly into Databricks, even from on-prem. No extra event layer needed. Every Unity Catalog table can act as an endpoint.
- 655 Views
- 0 replies
- 0 kudos
- 675 Views
- 0 replies
- 1 kudos
Databricks Advent Calendar 2025 #12
All leading LLMs are available natively in Databricks: - ChatGPT 5.2 from the day of the premiere! - System catalog with AI schema in Unity Catalog has multiple LLMs ready to serve! - OpenAI, Gemini, and Anthropic are available side by side!
- 675 Views
- 0 replies
- 1 kudos
- 640 Views
- 0 replies
- 2 kudos
Databricks Advent Calendar 2025 #10
Databricks goes native on Excel. You can now ingest + query .xls/.xlsx directly in Databricks (SQL + PySpark, batch and streaming), with auto schema/type inference, sheet + cell-range targeting, and evaluated formulas, no extra libraries anymore.
- 640 Views
- 0 replies
- 2 kudos
- 556 Views
- 0 replies
- 2 kudos
Databricks Advent Calendar 2025 #9
Tags, whether manually assigned or automatically assigned by the “data classification” service, can be protected using policies. Column masking can automatically mask columns with a given tag for all except some with elevated access.
- 556 Views
- 0 replies
- 2 kudos
- 1857 Views
- 1 replies
- 1 kudos
Building an AgenticLakehouse: Interacting with Databricks Workspace via LangGraph and MCP
This project, AgenticLakehouse, explores the cutting edge of "Agentic Data Analytics." I didn't just want a chatbot; I wanted a "living" interface for the Lakehouse. The result is a Multi-Agent System that intelligently orchestrates tasks, from query...
- 1857 Views
- 1 replies
- 1 kudos
- 1 kudos
Looks great, solid LangGraph + MCP setup on Databricks Apps. Thanks for sharing, @vinaygazula!
- 1 kudos
- 6961 Views
- 0 replies
- 4 kudos
Semantic Bridge to Unity Catalog
How Ontos bridges the gap between technical metadata and business meaning Here's a scenario that might sound familiar. You've got Unity Catalog humming along—tables are registered, lineage is tracked, access controls are in place. Technically, every...
- 6961 Views
- 0 replies
- 4 kudos
- 571 Views
- 0 replies
- 1 kudos
Databricks Advent Calendar 2025 #8
Data classification automatically tags Unity Catalog tables and is now available in system tables as well.
- 571 Views
- 0 replies
- 1 kudos
- 2893 Views
- 4 replies
- 5 kudos
Spark Jobs View on a serverless cluster!! Naah.....
Have you ever noticed (and wondered) that the wonderful Spark Job UI is no longer available in the databricks notebook if the cell is executed using 'serverless' cluster?Tradionally, whenever we run the spark code (action command), we used to see the...
- 2893 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi RamanThank you for the amazing insights! I am trying to understand more about SQL Warehouses - is it managed by Unity Catalog? From what I could gather, SQL Warehouse is a compute layer, not a data layer and therefore not managed by Unity Catalog....
- 5 kudos
- 626 Views
- 0 replies
- 2 kudos
Databricks Advent Calendar 2025 #7
Imagine all a data engineer or analyst needs to do to read from a REST API is use spark.read(), no direct request calls, no manual JSON parsing - just spark .read. That’s the power of a custom Spark Data Source. Soon, we will see a surge of open-sour...
- 626 Views
- 0 replies
- 2 kudos
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
