New blog articles in Databricks Community

Databricks AI Model Serving in production: scaling, cost, and latency lessons

srikantdas — Thu, 02 Jul 2026 14:03:27 GMT

A practical guide to running Databricks Model Serving in production across three levers, scale, cost, and latency, with a pre-launch checklist you can use.

One Tenant, Multiple Subsidiaries: Account and Tenant Architecture for Azure Databricks

JessieLi — Tue, 30 Jun 2026 09:20:23 GMT

On Azure, one Databricks Account maps 1:1 to an Azure Tenant — so subsidiaries sharing a Tenant share a Databricks Account, Unity Catalog Metastore, and System Tables. A practical guide to the architecture
options, design considerations, and mitigations.

Databricks Community Champion - June 2026 - Amira Bedhiafi

Rishabh_Tiwari — Fri, 26 Jun 2026 09:08:17 GMT

Our Community Champion Program celebrates members who go above and beyond to share knowledge, support fellow practitioners, and help make the Databricks Community a valuable place to learn and grow. Each month, we recognize individuals whose expertise, generosity, and passion for data create a meaningful impact on others.

This month, we're excited to spotlight our Community Champion for June 2026 — Amira Bedhiafi.

With a career spanning business intelligence, data engineering, analytics, and modern cloud data platforms, Amira has built a reputation for turning complex data challenges into practical, business-focused solutions. Beyond her technical expertise, she is an active contributor across multiple data communities and a strong advocate for knowledge sharing, continuous learning, and helping others succeed.

Here's more about Amira in her own words —

Name: Amira Bedhiafi
Community Nickname: @amirabedhiafi
Pronouns: she/her
Company: self-employed
Job Title: Senior BI/Data Engineer

Can you provide a brief overview of your career journey leading up to your current role?

My career has been focused on Business Intelligence, data engineering and analytics. I started by working with the Microsoft BI stack, including SQL Server, SSIS, SSAS, SSRS, and Power BI, before expanding into Azure, Databricks, Spark, and modern data platform architectures.

Over the years, I have worked on projects involving data warehousing, reporting, semantic models, performance optimization, data migration, and cloud-based analytics solutions. These experiences helped me grow from a BI developer into a senior BI/Data Engineer, with a strong focus on building reliable, scalable, and business-oriented data solutions.

Today, as a self-employed Senior BI/Data Engineer, I help organizations transform raw data into trusted insights and support teams in designing efficient data platforms and reporting solutions.

What do you enjoy most about your current job or role?

What I enjoy most is solving complex data problems and turning them into clear, useful solutions for business users. I like the combination of technical work and business impact: understanding the real need, designing the right data model or pipeline and seeing the final result help people make better decisions.

I also enjoy learning continuously. The data ecosystem changes very quickly, and working with tools like Databricks allows me to keep improving my skills and exploring better ways to build modern analytics solutions.

If you had to describe yourself using three words, what would they be? How do you think your coworkers would describe you?

I would describe myself as curious, persistent, and collaborative.

I think my coworkers would describe me as someone who is solution oriented, committed and always willing to help. I enjoy supporting others, sharing knowledge and working together to find practical answers to technical challenges.

Have you had any mentors or significant influences in your professional life? If so, could you tell us about them?

Yes, I have been influenced by several people throughout my career: colleagues, managers, community leaders and technical experts who encouraged me to keep learning and to share knowledge with others.

One of the biggest influences for me has been the different data communities themselves as I am part of the Microsoft Azure Community Champions Program and Fabric community also.

Seeing people openly share solutions, best practices and lessons learned motivated me to become more active in communities and to contribute back whenever I can.

When and why did you first start using Databricks?

I first started using Databricks while working on modern data platform and data engineering projects that required scalable processing, data transformation, and lakehouse architecture.

The main reason I started using it was the need to process and transform large volumes of data more efficiently, while also supporting collaboration between data engineers, analysts, and business teams. Databricks provided a strong environment for working with Spark, Delta Lake, notebooks, and structured data pipelines.

Are there any Databricks features that you particularly enjoy or find indispensable in your work?

Yes. I particularly enjoy working with Delta Lake, notebooks, Unity Catalog, and Databricks workflows.

Delta Lake is especially valuable because it brings reliability, versioning, and better data management to lakehouse projects. I also find notebooks very useful for development, debugging, documentation, and collaboration. Unity Catalog is important for governance, access control, and managing data assets in a more structured way.

Is there a Databricks feature you wish existed or would like to see in future updates?

I would like to see even more built-in guidance and automation around governance, data quality, and observability. For example, easier ways to monitor data pipelines, identify data quality issues, track lineage, and receive proactive recommendations would be very helpful.

I also think that simplifying some administrative and governance tasks would make Databricks even more accessible for teams that are growing their lakehouse maturity.

When did you join the Databricks Community, and what motivated you to do so?

I joined the Databricks Community recently to learn from others, share knowledge, and stay connected with people working on similar data engineering and analytics challenges.

My motivation was both professional and personal. I wanted to improve my Databricks skills, but I also wanted to contribute to a space where people help each other solve real-world problems.

Community platforms are very powerful because they allow us to learn from practical experiences, not only from documentation.

What aspects of the Databricks Community do you find most valuable or enjoyable?

The most valuable part of the Databricks Community is the knowledge sharing. I enjoy seeing real questions from users, practical solutions, discussions around best practices, and different perspectives from people working in various industries.

I also appreciate the collaborative spirit. The community makes it easier to learn, ask questions, discover new features, and feel connected to other data professionals.

Outside of work, what is your favourite hobby or pastime?

Outside of work, I enjoy travelling and reading. I usually set a goal for the books I want to read on Goodreads. Last year I read 30 books. I hope I will do the same number this year.

Where do you envision yourself professionally in the next three years?

In the next three years, I see myself intto writing technical books.

I would also like to continue contributing to the data community, mentoring others, and helping organizations build scalable and trusted data solutions. My goal is to combine deep technical expertise with leadership, community contribution, and business impact.

If you'd like to connect with Amira, you can find her on LinkedIn:

The Community Team would like to thank Amira for her contributions, knowledge sharing, and commitment to helping others learn and grow within the Databricks Community.

From sharing practical insights and best practices to actively participating in technical communities, Amira embodies the collaborative spirit that makes our community stronger. We're excited to see her continued impact as she pursues her goals of mentoring others, writing technical books, and helping organizations build trusted and scalable data solutions.

Thank you, @amirabedhiafi for being an inspiring member of our community and our Community Champion for June 2026! 🚀

Features in Motion: Three Patterns for Real-Time ML in Databricks Lakebase

uday_satapathy — Wed, 24 Jun 2026 08:44:05 GMT

A guide to choosing the right Lakebase pattern for computing and serving ML features based on how fresh they need to be.

Incremental REPLACE WHERE Flows Brings Targeted Refreshes to SDP and DBSQL

fran_martin — Tue, 16 Jun 2026 16:44:13 GMT

SDP has always given you two ways to keep a table fresh: a streaming flow for incremental ingestion, or a materialized view that recomputes an exact result. But most batch ETL teams live in a third world entirely: "refresh just the last 7 days, leave everything else alone." Incremental REPLACE WHERE flows finally make that a first-class pattern, with Enzyme doing the incremental math, so refreshes run over 3.4x faster and 2.5x cheaper than a naive partition overwrite.

One question, two data realities: supervisor agents across Kafka and the lakehouse

Tarzi-Simon — Mon, 29 Jun 2026 16:13:13 GMT

See how supervisor agents combine live Kafka data with governed lakehouse context to detect risky orders, answer faster, and act safely in production.

Triggering Databricks Jobs from Celonis Action Flows — A Step-by-Step Guide

Tarzi-Simon — Fri, 12 Jun 2026 14:10:40 GMT

Learn how to trigger Databricks Jobs directly from Celonis Action Flows using a simple HTTP call. This step-by-step guide walks through setup, dynamic parameters, error handling, polling, and security best practices to turn process insights into automated action.

Petabyte scale with Zerobus ingest: Download the Code and Ingest the Milky Way

Victoria_Bukta — Thu, 11 Jun 2026 20:27:22 GMT

Ingest 1 petabyte in 24 hours without infrastructure management. Meet Zerobus Ingest: Databricks’ serverless streaming service. Learn how dynamic partitioning, zero-copy parsing, and Delta Kernel Rust enable 12 GB/s per table throughput. Read our deep dive and run the benchmark yourself.

Unify Your Engineering Data: Jira Ingestion Best Practices

sonia_bendre — Thu, 11 Jun 2026 16:30:32 GMT

A best practices guide to setting up the Jira connector in Lakeflow Connect, plus how to extend it with Confluence for a full Atlassian pipeline in Databricks.

[PARTNER BLOG] Building a Fully Automated Power BI Refresh Pipeline with Databricks

Kavya21 — Thu, 11 Jun 2026 13:29:11 GMT

Still relying on scheduled refreshes or external APIs to update your Power BI semantic models? There’s now a much cleaner approach.

In this blog, I explore how Databricks Workflows can directly orchestrate Power BI semantic model refreshes using the native Power BI task and creating a more unified analytics pipeline. From Unity Catalog integration and secure authentication setup to end-to-end workflow automation, this walkthrough covers how data engineering and BI operations can finally work together seamlessly.

Automatic Git Deployments for Databricks Apps from Github

atreya — Wed, 10 Jun 2026 09:00:02 GMT

Databricks Apps now supports automatic Git deployments (Beta): connect a branch, and every new commit redeploys your app—no manual Deploy dialog, no external CI/CD tooling. Databricks registers a GitHub webhook that fires on each push to your watched branch, making Git the single source of truth for what's running in production.

Recording | BrickTalk: Using AI to Navigate Unfamiliar Business Data

Tushar_Parekar — Fri, 05 Jun 2026 15:43:52 GMT

Discover how AI helps teams confidently navigate unfamiliar business data with @AbhaySingh

Apache Spark’s Real-Time Mode Use Case Deep Dive: Gaming Sessionization

MuraliTalluri — Thu, 04 Jun 2026 08:42:10 GMT

Explore how Apache Spark™ Real-Time Mode enables real-time gaming sessionization for millions of active device sessions
Learn how transformWithState timers power proactive, timer-driven heartbeats — generating output on a schedule, independent of incoming data
See how Real-Time Mode paired with transformWithState replaces custom in-house applications and external streaming engines — delivering sub-second precision for both input processing and timer driven output.

[CUSTOMER BLOG] Enabling Seamless Inbound Data Sharing at Magnite

abhay-jalisatgi — Tue, 02 Jun 2026 13:45:53 GMT

We explore how Magnite and Databricks built a inbound data-sharing platform using Marketplace, Delta Sharing, and CDF to automate publishing, validation, and ingestion at scale.

Building a foundation for secure and automated data publishing
A joint Magnite–Databricks Professional Services effort that delivers a standards-compliant publishing framework using Delta Sharing, CDF, and a customer-installable Python wheel.
Automating data consumption for Magnite
An API-driven, Lakeflow-powered ingestion architecture that detects new customer shares, validates schemas, applies transformations, and operationalizes data with built-in observability.
Enabling adoption at scale through the Databricks Marketplace
A productized onboarding experience with schema requirements, guided runbooks, and packaged utilities.

Multi-Agent Supervisor for Hybrid Retrieval with Agent Bricks and MLflow

Daniel-Liden — Mon, 01 Jun 2026 21:15:19 GMT

A Databricks Supervisor Agent answers honeybee colony health questions across USDA SQL tables and guidance PDFs by coordinating Genie and Knowledge Assistant.

FinOps at Scale: Building a Repeatable Architecture for Cross-Account Cost Visibility in Databricks

mreiling_data — Fri, 29 May 2026 13:29:29 GMT

Manage Agent and Tool Sprawl with Unity AI Gateway in Databricks

alexandergenser — Fri, 29 May 2026 09:59:51 GMT

What’s new in the Lakeflow Pipelines Editor

theresahammer — Thu, 28 May 2026 15:44:15 GMT

The Lakeflow Pipelines Editor is now GA. This includes a redesigned layout for AI first development with the native Genie Code integration. You can also run and preview SQL without materializing anything, get insights on why your materialized views are not incrementalizing and write unit tests using the new testing framework (in Beta).

Databricks Serverless Migration: A Practical Production Playbook

matta — Tue, 26 May 2026 08:46:03 GMT

A single knowledge resource bridging platform limits, real PoC lessons, and automated ways of refactoring workflows

Databricks Serverless drives operational efficiency and slashes maintenance costs by replacing manual infrastructure management with instant, microsecond-level scaling.

Eliminating idle cluster waste allows data teams to focus entirely on delivering products rather than managing configurations.

Introduction

Beyond a Configuration Switch: Moving to Databricks Serverless is a comprehensive architectural audit, not a simple cluster setting toggle
The Legacy Gap: The promise of "minimal or no code changes" often overlooks the deep legacy dependencies embedded in mature production pipelines
Platform-Managed Optimization: Success on serverless relies on letting go of manual, low-level tuning (e.g., shuffle partitions) in favor of the platform’s native, dynamic scaling and layout mechanisms
Practical Blueprint & Automated Remediation: This runbook serves as a definitive experience guide to programmatically detect, analyze, and remediate incompatibilities at scale using the Databricks SDK and Genie Code
Pre-Flight Review: Engineers must review the official best practices and limitation guides referenced here before migrating to prevent non-negotiable production blockers

From Surprise Full Refreshes to Predictable Bills: REFRESH POLICY for MVs

Pravas007 — Tue, 26 May 2026 19:21:49 GMT