Ingestion from SharePoint is now available directly in PySpark. Just define a connection and use spark-read or, even better, spark-readStream with an autoloader. Just specify the file type and options for that file (pdf, csv, Excel, etc.)
Excel The big news this week is the possibility of native importing Excel files. Write operations are also possible. There is a possibility of choosing a data range. It also works with the streaming autoloader, currently in beta. GPT 5.2 The same day...
ZeroBus changes the game: you can now push event data directly into Databricks, even from on-prem. No extra event layer needed. Every Unity Catalog table can act as an endpoint.
All leading LLMs are available natively in Databricks: - ChatGPT 5.2 from the day of the premiere! - System catalog with AI schema in Unity Catalog has multiple LLMs ready to serve! - OpenAI, Gemini, and Anthropic are available side by side!
Databricks goes native on Excel. You can now ingest + query .xls/.xlsx directly in Databricks (SQL + PySpark, batch and streaming), with auto schema/type inference, sheet + cell-range targeting, and evaluated formulas, no extra libraries anymore.
Tags, whether manually assigned or automatically assigned by the “data classification” service, can be protected using policies. Column masking can automatically mask columns with a given tag for all except some with elevated access.
This project, AgenticLakehouse, explores the cutting edge of "Agentic Data Analytics." I didn't just want a chatbot; I wanted a "living" interface for the Lakehouse. The result is a Multi-Agent System that intelligently orchestrates tasks, from query...
Looks great, solid LangGraph + MCP setup on Databricks Apps. Thanks for sharing, @vinaygazula!
How Ontos bridges the gap between technical metadata and business meaning Here's a scenario that might sound familiar. You've got Unity Catalog humming along—tables are registered, lineage is tracked, access controls are in place. Technically, every...
Data classification automatically tags Unity Catalog tables and is now available in system tables as well.
Have you ever noticed (and wondered) that the wonderful Spark Job UI is no longer available in the databricks notebook if the cell is executed using 'serverless' cluster?Tradionally, whenever we run the spark code (action command), we used to see the...
Hi RamanThank you for the amazing insights! I am trying to understand more about SQL Warehouses - is it managed by Unity Catalog? From what I could gather, SQL Warehouse is a compute layer, not a data layer and therefore not managed by Unity Catalog....
Imagine all a data engineer or analyst needs to do to read from a REST API is use spark.read(), no direct request calls, no manual JSON parsing - just spark .read. That’s the power of a custom Spark Data Source. Soon, we will see a surge of open-sour...
DBX is one of the most crucial projects of dblabs this year, and we can expect that more and more great checks from it will be supported natively in databricks. More about dbx on https://databrickslabs.github.io/dqx/
When something goes wrong, and your pattern is doing MERGEs per day in your jobs, backfill jobs will help you to reload many days in one shot.
Full link of the actual blog for reference - https://www.databricks.com/blog/announcing-backfill-runs-lakeflow-jobs-higher-quality-downstream-data
With the first day of December comes the first window of our Databricks Advent Calendar. It’s a perfect time to look back at this year’s biggest achievements and surprises — and to dream about the new “presents” the platform may bring us next year. ...
Fantastic kickoff to the Databricks Advent Calendar 2025 , appreciate you steering the series, @Hubert-Dudek!
With the new ALTER SET, it is really easy to migrate (copy/move) tables. Quite awesome also when you need to make an initial load and have an old system under Lakehouse Federation (foreign tables).
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
