New board topics in Databricks Community

See Databricks costs right where you work

protmaks — Sat, 17 Oct 2026 08:33:25 GMT

What if you could see the cost of your Databricks resources without switching to a dashboard or digging through billing tables?

We built FinOpsWay, a free browser extension that brings cost information directly into the Databricks interface.

You can see costs next to clusters, individual runs, notebooks, and Databricks Apps — and even explore Lakebase costs while setting up databases and branches.

The idea is simple: make cloud costs easier to understand at the moment you're working with a resource, without adding another tool or dashboard to your daily workflow.

In this article, I share how the extension came to life, what it can do today, and why we decided to make it free.

Read the article: Databricks costs directly within the interface — FinOpsWay extension

How track of all Databricks badges and certifications

protmaks — Sat, 17 Oct 2026 08:30:18 GMT

There are now 292 Databricks badges and certifications. I wanted to see which ones I’ve already earned and which ones are still available, but navigating the full list was becoming a bit of a pain.

So I built a Credentials section on Bricksters to make this easier.

You can paste a link to your public Databricks credentials wallet, and it will:

Automatically find your existing badges and certifications
Mark the credentials you’ve already earned
Show you which ones you can still earn
Browse all credentials by category
Export your badges into a single image you can share on LinkedIn

I’ve also recorded a short video showing how it works.

You can try it here: https://bricksters.org/credentials/

PIM for Groups with Databricks: users appear in group without activating, nothing shows in PIM to ac

Charansai — Fri, 16 Oct 2026 20:09:46 GMT

Hi everyone,

We’re setting up just-in-time read access to a Unity Catalog catalog for a small audit team using Microsoft Entra PIM for Groups, and I’d like to check whether our setup is right.

Setup

Azure Databricks with account-level SCIM provisioning from Entra ID.
An Entra security group created through our IaC pipeline with no members. The group is synced to the Databricks account and holds the workspace assignment (User), USE CATALOG, USE SCHEMA and SELECT on the catalog, and CAN USE on a SQL warehouse.
The intent is that users are eligible members in PIM for Groups, activate for a few hours with a justification, get added to the group, and lose access when the activation expires.

Issue

In the Databricks Account console, the group shows 5 users as members, even though nobody has activated anything.
In PIM > My roles > Groups > Eligible assignments, there are no results to activate.

Questions

Does this mean the users were added as direct or Active members instead of Eligible, and that they currently have access without activating?
What’s the recommended way to make sure membership only comes from PIM activation and can’t be added directly (e.g. role-assignable groups, restricting group owners)?
With SCIM, membership changes take about 20 to 40 minutes to reach Databricks. Is automatic identity management a better fit for PIM-based access, and does it pick up activation and expiry faster?
Has anyone run this pattern in production, and are there any gotchas with sessions or tokens continuing after the activation expires?

Thanks in advance!

About Azure SQL Managed Connector

srikanthp24 — Fri, 16 Oct 2026 18:02:01 GMT

Hi Community,

We are trying to configure a Databricks Lakeflow Connect managed SQL Server connector to ingest data from Azure SQL Database into Databricks.

We are using a contained SQL database user that has access only to the specific source database, with no access to the "master" database.

However, during connection validation, we are encountering the following error:

Authentication failure while obtaining the connection to the source database: Error while creating connection pool for catalog master. Login failed for user '<contained_db_user>'.

Our understanding:

It appears that the Lakeflow Connect managed SQL Server connector attempts to establish a connection to the master database during connection validation, even though the intended source is a specific Azure SQL database.

Since we are using a contained database user, authentication to "master" fails.

Our security requirement:

We need to restrict the service account to the source database only, without granting access to "master".

Questions for the community:

Has anyone successfully configured the Lakeflow Connect managed SQL Server connector using a contained database user with access restricted to a single Azure SQL database?
Is there a supported configuration to bypass master database authentication and connect directly to the specified source database?
Is access to "master" mandatory for this connector, even when ingesting from only one database?
Are there any recommended workarounds that preserve database-level security isolation while still using the managed Lakeflow Connect connector?

We would prefer to use the managed Lakeflow Connect connector rather than building a custom JDBC ingestion framework.

If you have any better approach suggest?
Any insights, experiences, or recommendations would be greatly appreciated.

Thanks!

Jev as Judge for Genie Agents

sharath007 — Fri, 16 Oct 2026 17:41:28 GMT

Can we configure the Jev to be as judge for Genie Agents ? I know we can log traces of Genie Agents into ML Flow on top of it can we bring in Jev and score the outputs generated ?

Learn Databricks Performance Optimization | Master how to measure & speed up Databricks queries

Tushar_Parekar — Fri, 16 Oct 2026 17:29:36 GMT

Databricks Training & Certifications

Learn Databricks Performance Optimization 🚀

Measure first, then make your Spark and Delta Lake workloads faster and cheaper.

🔍 Spark UI & Query Profile 📦 Liquid clustering & data skipping ⚡ Photon & caching

Slow queries and oversized clusters cost time and money. Performance optimization on Databricks starts with measuring: find the real bottleneck, let the platform's automatic optimizations do the heavy lifting, and tune by hand only where it pays off.

The catalog offers one Professional-level course, part of the Advanced Data Engineering with Databricks series, in four formats. Start free and self-paced, or go hands-on with a lab or an instructor. 👇

✓ The course · Professional · Data Engineer

Professional · 2H to 4H

Databricks Performance Optimization

Learn how to optimize workloads and physical data layout with Spark and Delta Lake, and use the Spark UI to assess performance and debug applications. Topics include streaming, liquid clustering, data skipping, caching, Photon, and more.

Free · Self-paced · 2H → Paid / Subscription · Lab · 3H → Paid · Instructor-led Lab · 4H → Paid · Mandarin (中文) · 4H →

Instructor-led also available in English, 日本語, Português BR, and 한국어.

What you'll learn

01 Measure. Read the Spark UI and Query Profile to find where time is really spent.

02 Lay out your data. Use liquid clustering and data skipping so queries read less.

03 Speed up compute. Put caching and Photon to work, and tune streaming workloads.

04 Debug. Diagnose slow applications and confirm each fix made a difference.

Browse Performance Optimization in the catalog →

Measure. Optimize. Verify. Make every query count with official Databricks training.

External location Manage permission

orlandovallejos — Fri, 16 Oct 2026 16:34:10 GMT

Hi team, I'm trying to update the owner of an external location and I see the following message:

Requires permission MANAGE on external location 'location_name'

The owner of the resource, the user that created it doesn't have that permission? How is that possible?

ok, let's say the owner doesn't have it by default, how I add the MANAGE permission to this resource?

Thanks

Zerobus Variant Data not Processed as Expected (json data destined for Variant column).

DB1To3 — Fri, 16 Oct 2026 14:10:05 GMT

Lets say I have a table receiving zerobus data (json) over HTTP/rest.

That last VARIANT is "ValueJson".

If I want to submit data to it, I would expect to be able to use normal json formatting:

[ { "a": "E", "b": 123, "c": "xyz", "ValueJson": { "x" : "y" } } ]

Unfortunately it doesn't want that part of the record to be ACTUAL json. The error is like so

Record decoder/encoder error: invalid type: map, expected a string at line 13 column 24. Error Code: 4044, Error State: 3.

Unfortunately the fix is to send json-encoded json (escaped json in a string, since the error says it expected a string).

[ { "a": "E", "b": 123, "c": "xyz", "ValueJson": "{ \"x\" : \"y\" }" } ]

Does this seem reasonable behavior, or is it a bug? I think it is a bit unnatural to have to encode the json VARIANT instead of just nesting it inside the record as normal json. Ideally the use of this VARIANT wouldn't force us towards using json-encoded-json. It isn't the end of the world, but it seems like a very unfortunate way of sending data from a remote client - one that shouldn't need to have a deep understanding of the eccentricities of the VARIANT-parsing rules for zerobus.

Disable auto-scroll on Databricks Genie Code

humphriesdt — Fri, 16 Oct 2026 13:58:10 GMT

Hello,

I frequently use Genie Code when developing, particularly to ask Genie questions about certain lines or sections of code. The responses are great, but because they are often long, the chat auto-scrolls while text is being generated. Is there a way to disable this? I would like to be able to read the generated text while more is being created, and then scroll down when I'm ready to read, instead of Genie auto-scrolling away parts of text I'm not finished reading.

Thank you!!

Zerobus rescue (VARIANT) not working. No Preview Available.

DB1To3 — Fri, 16 Oct 2026 13:45:42 GMT

Has anyone tried using the rescue feature for json in zerobus-ingest? It seems like a very significant feature, assuming the schema incoming json is even 1% outside the control of team that may be ingesting on the databricks side. There needs to be a way to ingest a json record that is slightly non-conforming. Else you get unhelpful errors.

Here is the doc:

... but I'm not able to get this working, and there is no "preview" switches for this in my DBX account settings.

The table is configured with the NULLable, tagged column as required:

Yet whenever I submit json row data (using currently HTTP/rest), I just get an error that says zerobus doesn't approve:

Record decoder/encoder error: unrecognized field name 'whatever' at line 2 column 24. Error Code: 4044, Error State: 3.

I looked for a preview in my account settings and there is none listed (supposed to be "Beta" according to docs).
Any advice would be appreciated. Let me know if I'm misunderstanding the intended behavior of this rescue column.

Pattern: Sharing Databricks output with an AKS app through an external volume and managed identity

Amar_Naik — Fri, 16 Oct 2026 13:43:35 GMT

Scenario
A transaction app runs on AKS. It is the only component with a connection to MongoDB Atlas. Databricks processes the data, and the app has to pick up the results and write them back to Atlas. The requirements are no PATs, no CLI, no Databricks API calls from the pod, and no Databricks access to the production database.

Pattern

Databricks writes to an external volume.
- An Access Connector (managed identity) has Storage Blob Data Contributor on a dedicated ADLS Gen2 export container.
- A Unity Catalog storage credential, external location, and external volume sit on top of it.
- Jobs write Parquet or Delta to /Volumes/<cat>/<schema>/<vol>/... and write a manifest file last.
AKS mounts the same container as blob storage.
- The Azure Blob CSI driver mounts the export container into the pod as a read-only volume.
- Authentication uses a managed identity with Storage Blob Data Reader on that container only. No account keys or SAS tokens are stored in the cluster.
- The app reads the files as a normal local path, for example /mnt/exports/<batch_id>/.... No Azure SDK is needed in the app code.
The app job upserts into Atlas.

Why this works well

Only the app touches Atlas.
No secrets in either direction.
Databricks and the app are decoupled, and every batch can be replayed from storage.

Trade-offs

Unity Catalog does not govern the AKS reads. Azure RBAC does. A dedicated export container, least-privilege roles, and storage diagnostic logs cover the audit side.
The pattern suits batch and micro-batch. Low-latency needs call for Event Hubs or Kafka.
Many consumers across teams call for Delta Sharing instead.

Open points

How others handle the governance gap on reads outside Unity Catalog.

What Is – a system table? (Your Databricks bank statement)

Valeria_Koz_DBX — Fri, 16 Oct 2026 13:01:32 GMT

Clients ask me the same three questions all the time:

What are system tables?
What are they about?
And what can I actually do with them?

I recently came across a Databricks blog that answers the last one brilliantly: Top 5 System Table Queries for Understanding Your Databricks Costs. So here's the simple version first. Then go and read it.

The short answer

Your Databricks invoice is the total on your card bill. System tables are the statement behind it: every line, every date, every shop.

They're read-only tables that Databricks fills in for you, in a catalog called system. You query them like any other table, with SQL, a dashboard or Genie.

What's in them

They cover more than cost. Think of them as a building's logbooks:

Billing: what you used, and what it costs
Access: who opened which door (the audit log)
Compute and jobs: which machines ran, what ran on them, and whether it worked

For cost, two tables do the heavy lifting:

system.billing.usage is the statement: what was used, how much, and when.
system.billing.list_prices is the price tag. Usage is counted in DBUs, not pounds, so you need both.

What you can do with them

The blog gives ready-made SQL for five questions:

Where's the money going? Spend by product, day by day.
Which warehouse is the big spender? And is it growing?
Which team spent it? Like name labels in a shared fridge, this only works if people label things.
Was yesterday weird? Like a smart meter telling you the heating was left on.
Where will we land this quarter? A forecast built on your own history.

I won't repeat the SQL. That's what the blog is for.

Four gotchas

Usage isn't money. Multiplying units by the price list gives you the list price, which may not be your negotiated rate.
It only covers Databricks. On classic compute, the bill for the cloud provider's machines and storage sits somewhere else.
No label, no owner. Untagged spend shows up as "untagged", and that number is the size of your gap.
Free to read, not free to run. The tables cost nothing, but the warehouse that queries them does. You may also need an admin to give you access.

Why it matters

One finance team I worked with had all its costs in one unmarked pile. Every budget meeting started with an argument. Once they could see the statement, the meeting started with "here's the line, who owns it?"

Now go and read the blog: Top 5 System Table Queries for Understanding Your Databricks Costs

What's the first question you'd ask your Databricks statement?

auto.arima in R keeps running without end

arkobose — Fri, 16 Oct 2026 10:41:59 GMT

I have a time-series that I am trying to model using auto.arima in R. Upon execution on a cluster runtime with 96GB memory and 48 cores, I observed that the code kept running without any end in sight. It ran for over a day without finishing.

Curious, I ran the same code on the same time-series on my modest laptop (32GB RAM, 10 cores), the code finished execution in under 5 minutes.

I am at a complete loss to understand what is taking the code I am running on my Databricks cluster runtime take so long!

For completeness, the code I am trying to execute is simply this:

baselines_daily <- auto.arima(daily_network_efficiency,

xreg = F,

seasonal = FALSE,

stepwise = FALSE,

parallel = TRUE,

approximation = FALSE)

where daily_network_efficiency is my time-series.

Any help in shedding light on this mystery will be greatly appreciated.

Thank you!

Announcement | Your data, your storage, your rules: A 2026 guide to Unity Catalog managed tables

Tushar_Parekar — Fri, 16 Oct 2026 09:26:53 GMT

Unity Catalog managed tables let teams automate table management while keeping data in cloud storage they own. You control where new managed tables and volumes are written, while Unity Catalog manages governance, lifecycle, and optimization across Delta Lake and Apache Iceberg.

Key Highlights

Keep data in storage you control: Managed table data can live in your own S3 bucket, ADLS container, or GCS bucket, at a managed storage location defined for your metastore, catalog, or schema.
Set storage placement by domain: Configure a managed location at the metastore, catalog, or schema level. The most specific location takes precedence, making it possible to separate data for compliance, residency, administration, or cost attribution.
Change where new data lands: Use ALTER CATALOG or ALTER SCHEMA ... SET MANAGED LOCATION when requirements change. Existing tables remain in their current locations; new tables and volumes use the updated path.
Convert external tables deliberately: Before converting an external table to managed, set the target managed location. During conversion, Databricks copies the table data and transaction log into that location.
Stay open and interoperable: Managed tables remain accessible through open APIs, including the Unity REST API and Iceberg REST Catalog, with credential vending and governance controls for supported external engines.
Automate optimization and maintenance: Unity Catalog managed tables support capabilities such as predictive optimization, metadata management, and automatic lifecycle operations to help reduce manual administration.

👉 Read the full post here

Possible bug in Lakeflow Connect SQL Server ingestion with GUID primary keys – Missing records

volodymyr_b — Fri, 16 Oct 2026 08:43:43 GMT

Hi everyone,

We've encountered a strange issue with Databricks Lakeflow Connect query-based ingestion from SQL Server, and I'm wondering if anyone else has experienced something similar.

We have a SQL Server table with 1,448 records and a uniqueidentifier primary key. When performing a full refresh, Databricks ingests only 364 records, but the pipeline completes successfully without any errors.

After digging into the cluster logs, I suspect the problem is related to how Databricks handles GUID ordering during snapshot partitioning.

The connector correctly identifies 1,448 distinct IDs:

-- logs
getSnapshotSplitsByNonNumericKey:
splitCount=1, totalDistinct=1448

However, the generated query applies the following filter:

-- logs
Pushed Filters: Id IS NOT NULL, (RPAD(Id, 36, ' ')) <= '38C5A9C6-D111-4182-8A7F-FFE14F608876'

The snapshot then completes with:

-- logs
numInputRows: 364
dropped: 0

What I think is happening

SQL Server compares uniqueidentifier values differently from their string representations.

My theory is that Databricks:

Retrieves the GUIDs using SQL Server's native GUID ordering.
Determines the upper boundary for the snapshot partition.
Applies that boundary using a string comparison (RPAD(Id, 36, ' ') <= ...).
Accidentally excludes records because string ordering and native GUID ordering are different.

For example, consider these two GUIDs:

38DE50FE-2AC8-46E8-91AB-EAB0A9E944B0
38C5A9C6-D111-4182-8A7F-FFE14F608876

You can reproduce the difference directly in SQL Server. Try this query:

DECLARE @guid_a uniqueidentifier = '38DE50FE-2AC8-46E8-91AB-EAB0A9E944B0';
DECLARE @guid_b uniqueidentifier = '38C5A9C6-D111-4182-8A7F-FFE14F608876';

SELECT CASE WHEN @guid_a < @guid_b THEN 'TRUE' ELSE 'FALSE' END, 'GUID'
UNION
SELECT CASE WHEN CONVERT(char(36), @guid_a) < CONVERT(char(36), @guid_b) THEN 'TRUE' ELSE 'FALSE' END, 'STR';

The result is:

TRUE     GUID
FALSE    STR

So the same two GUIDs produce opposite comparison results depending on whether they are compared as native uniqueidentifier values or strings.

My questions

Has anyone encountered this issue with Lakeflow Connect query-based ingestion?
Is there a way to override the automatically selected snapshot partitioning column in the pipeline configuration?
Are there any known workarounds that don't require modifying the source SQL Server tables?

What concerns me most is that the pipeline reports a successful full refresh despite missing a significant number of records. This could easily go unnoticed and lead to silently incomplete datasets.

Any insights would be appreciated. Thanks!

Datavault implemention on Databricks

ankit2604 — Fri, 16 Oct 2026 06:01:25 GMT

Hi everyone,

I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.

While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on complex Business Vault objects.

Would love to hear from anyone who has built a native framework from the ground up!

Thanks

Genie can now live in your favorite IDE!

LingeshK — Fri, 16 Oct 2026 00:06:03 GMT

You can now open a terminal in your project folder and type genie 🎉

You now have a coding agent that can read your local files, run commands, and query your Unity Catalog tables in one conversation. 🚀

That is Genie Code CLI, now available in Beta.

Most data work lives in two places. Your scripts, sample files, and prototypes sit on your laptop. Your governed tables, pipelines, serving endpoints, and apps sit in your Databricks workspace.

Until now, you carried the work across that line yourself. You read a file, checked a table, wrote the code, ran it, checked the result, and deployed. Each step is easy. The hand-offs between the steps are where the time goes.

Genie Code CLI works on both sides of the line. You describe the task. The agent reads and edits your local files, runs commands, and reaches into Databricks on your behalf.

What is Genie Code CLI?

Genie Code CLI is a coding agent that runs locally in your terminal. Databricks tuned it for data and AI work.

"Locally" refers to where the agent runs and which files it operates on. The model does not run on your machine. Databricks provides and governs model access through Unity Gateway, so you do not bring your own model subscription.

Genie Code CLI is also a separate tool from the Databricks CLI. You install the Databricks CLI yourself, and Genie Code CLI uses it.

To find your data and answer questions about it, Genie Code CLI calls the Databricks CLI, including databricks genie ask. The agent reaches your workspace through the same command-line tool you already signed in with. Genie Code CLI runs under your own identity. It can access only the data and perform only the operations that your Databricks permissions allow.

Fig 1: One conversation across your laptop and your workspace.

How do you get started?

You have to adhere to the following three requirements first:

A Databricks workspace in a Unity Gateway-supported region.
Unity Catalog is enabled for that workspace.
The Databricks CLI, version 1.0.0 or above, is installed separately.

Then set it up in four steps:

Install: on macOS or Linux, run curl -fsSL https://github.com/databricks/genie-code-cli/releases/latest/download/install.sh | bash. On Windows, use the PowerShell command in the docs.
Check: run genie --version.
Launch: go to your project with cd my-project, then run genie.
Connect: select a Databricks CLI profile when prompted, or authenticate if you have none. Then confirm that you trust the contents of the directory.

The agent will read and edit files and run commands in that project. Pick the profile on purpose, especially if your machine has both a development and a production profile.

What can you ask it to do?

The docs give five example prompts. Each one crosses the line between your laptop and your workspace:

"Clean up these monthly sales CSVs, load them into Unity Catalog, and show me revenue by region over time."
"Which of my catalogs has customer churn data, and what were the top drivers last quarter?"
"Add a silver refunds table to my orders pipeline and backfill the last 90 days."
"Run this training script on Databricks and put the model behind a serving endpoint."
"Build a local app prototype for my customer's table and deploy it as a Databricks App."

Take the first prompt. It holds three jobs: clean the files, load them into a governed table, and answer a business question. Most of us would do those in three different tools.

A short prompt still needs exact terms. Name the catalog and schema to write to. Say if the agent may replace an existing table. Say what counts as revenue, for example, whether refunds are in or out. The agent can connect the steps. It cannot guess your business rules.

The backfill prompt needs the same care. A backfill processes historical data, so it is more than a code edit. Name the environment, and ask the agent to show you its plan before it runs anything.

Is this the same Genie Code you use in the workspace?

Genie Code CLI is a separate experience, built for local, terminal-based work.

It has its own tools, skills, and instructions, and at this time, it does not share them with Genie Code in the workspace. If you set up personal skills or instructions in the workspace, Genie Code CLI will not see them today.

It also does not yet include all the capabilities of the workspace UI. Use Genie Code CLI when you work locally with files, scripts, and prototypes. Stay with Genie Code in the workspace for the richer, fully featured experience.

Fig 2: Genie Code CLI and Genie Code in the workspace, side by side.

Which budget controls the spend?

This is the point I would raise with workspace admins first. The product carries the Genie name, and the billing runs through Unity Gateway. Genie Code CLI usage is billed the same as using the model directly through Unity Gateway, and it appears in your Unity Gateway usage.

It is not billed under Genie pricing, and Genie budgets do not apply to it. To control spending, set budgets for Unity Gateway. Databricks also manages the model choice. During Beta, Genie Code CLI uses GPT 5.6, and you cannot choose a specific model.

My 2 cents....

Beta means change. The docs say the capabilities, commands, and interfaces change frequently. Check the docs before you build automation on top of a command.
Start small. Open Genie Code CLI in a non-production project with a clean git working tree. Ask it to explain the project first. Then ask for one small change, and read the diff before you keep it.
Keep two boundaries apart. In the workspace, your Databricks permissions limit what the agent can do. On your laptop, the agent runs in your terminal session, so treat any command it runs as one you ran yourself.

Set up well, I would expect to see:

One Databricks CLI profile per environment, named clearly, so you do not point a prototype at production by mistake.
A Unity Gateway budget in place before a wider team rollout.
Prompts that name the catalog, the schema, and the environment.
Changes are reviewed as a diff before they land.

Thanks for reading this far. I would love to hear how you plan to use it, so here are three questions to start:

Which of your workflows crosses the line between your laptop and your workspace most often?
How would you split the work between Genie Code CLI and Genie Code in the workspace for your team?
For the admins here, what would you want to see in Unity Gateway usage before you roll Genie Code CLI out to more people?

For more good discussion about Genie, I am always up for it. 🧞👇🏻

Feel free to connect with me directly or drop a DM: linkedin.com/in/lingeshwarankanniappan

Sources

Announcement | Unlocking Data Portability with REGISTER and UNREGISTER APIs

Tushar_Parekar — Thu, 15 Oct 2026 15:38:02 GMT

Open table formats make data accessible across engines, but true portability also requires the ability to move catalog management safely. New REGISTER and UNREGISTER endpoints provide a standard way to transfer an Iceberg table between catalogs without copying, rewriting, or moving the underlying data.

Key Highlights

Attach existing tables with REGISTER: Register an existing table by providing its current metadata location. The catalog creates a reference to the table without copying or modifying data files.
Relinquish ownership with UNREGISTER: Remove a table from its current catalog without deleting its data or metadata. The response returns the latest metadata location needed by the next catalog.
Prevent split-brain scenarios: REGISTER alone can leave two catalogs believing they manage the same table. UNREGISTER provides the explicit handoff step needed to ensure only one catalog coordinate commits.
Build on an open standard: UNREGISTER has been contributed to the Apache Iceberg REST catalog specification, adding a standard mechanism for clean catalog handoffs.
Keep migrations operationally safe: A production move still requires stopping writers, unregistering from the original catalog, registering with the new catalog, and repointing jobs before writing resume.

👉 Read the full post here

Databricks Apps gives you the app you wanted, without the React plumbing or the Power Apps limits

Amar_Naik — Thu, 15 Oct 2026 15:06:58 GMT

Most teams think they have to pick one. Build it in React and own the API, auth, hosting, and audit trail. Or
build it in Power Apps and hit the limits when the app gets complex. With Databricks Apps, you don't have to
choose. You keep your React UI and drop the infrastructure behind it.

The real choice is not React vs Databricks
You don't have to pick between React and Databricks Apps. You can run your React frontend, with your design system, on Databricks Apps.

The real question is who owns everything behind the UI:

Standalone React SPA: Your team builds and runs the API server, authentication, hosting, database access, BI connectivity, and the audit trail.
React on Databricks Apps: Your team builds the UI and the business logic. The platform provides hosting, SSO, secrets, networking, and direct access to Delta tables .

Where Power Apps runs out

Power Apps is quick to start and strong on reporting. It gets harder as the app grows. Complex conditional forms,
custom business rules, and pixel-accurate UI are where low-code limits show up. Databricks Apps gives you full
code control and keeps the platform benefits Power Apps is known for.

Five reasons Databricks Apps wins

Five reasons Databricks Apps wins

Your data is already there. The app queries Delta tables directly, with no separate pipeline for it.
The audit trail is built in. Delta Lake Change Data Feed and time travel give immutable history and
point-in-time reconstruction with no custom code [2].
Governance is declared once. Workspace SSO and Unity Catalog enforce access in the app, dashboards, Power BI, and future AI tooling [3].
There is little infrastructure to approve. We deployed a working PoC the same day with zero net-new cloud
resources: no new networks, app registrations, or key vaults.
It's ready for more than the app. Delta Sharing gives outside parties read-only access in place of file
exports [4]. Genie gives natural-language access to the same data [5]

Serverless, scale-to-zero compute also replaces always-on servers and per-user licensing.

Your turn

What matters most when you choose a platform for internal Databricks apps? What would you add to this list?

#DatabricksApps #Delta Lake #Unity Catalog #Delta Sharing #Genie

References

Databricks Apps overview: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/
Change data feed on Azure Databricks:https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed
Unity Catalog: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/
Delta Sharing: https://www.databricks.com/product/delta-sharing
Genie: https://docs.databricks.com/aws/en/genie/

🌟 Community Pulse: Your Community Roundup! September 14 – October 11, 2026

Advika — Thu, 15 Oct 2026 09:29:40 GMT

Recap • September 14 – October 11

Community Pulse

Four weeks of articles, learning content, and conversations – the highlights, all in one read. 🗞💬

🏅 Community Stars

The members who showed up and made a real difference 😎:

@ivanvyd

@data_pulse

@ThomazNeto

+ Top Brickster contributor:

@Louis_Frolio

@anuj_lathi

@Ashwin_DSA

Note: Names are listed in no particular order.

📖 Knowledge Hub

📃 MVP Articles

🗒 Lakebase Articles

✍️ Community Articles

🏗 Data Engineering & Architecture

SAP Medallion Architecture Patterns: What Bronze, Silver and Gold Have to Do Differently ↗

A Metadata-Driven Backdated Extraction Pattern for IoT/Sensor Pipelines Using Zerobus and Databricks ↗

Preventing Duplicate Records When Reprocessing Data in Databricks ↗

Lakehouse vs data warehouse vs data lake, drawn step by step: the one-line interview answer ↗

Real-Time SAP Accounts Receivable on Databricks Lakeflow Declarative Pipelines ↗

Architecting a Medallion Lakehouse ↗

⚡ Performance, SQL & Cost

Databricks SQL 2026: From SQL Analytics to an AI-Ready Data Platform ↗

How to force Databricks Serverless to process 1 TB (40 Billion Rows) in 14 Minutes ↗

Spark Internals on Databricks ↗

How to set budgets and alerts on SQL warehouses so self-service never surprises Finance ↗

🤖 AI & Agents

Enterprise AI Has a Serious Context Problem ↗

Building Custom Agents on Databricks:LangGraph, Atlan-Grounded Routing, and Per-Run Cost with MLflow ↗

From Data Quality to Context Quality ↗

Beyond Chat History: Designing Portable Memory and Decision Boundaries for Enterprise AI Agents ↗

🛡 Data Quality & Governance

Row-level security when the filter column is not on your table ↗

Databricks Data Quality - Agent Proposes, DQX Disposes ↗

Declaring a Primary Key RELY turned my dbt unique Test Green on Duplicate Data ↗

USE CONNECTION bypasses your MCP Service's Tool Selection and Policies ↗

🏢 Industry Use Cases

Unmasking the Connected Consumer ↗

Insurers Don’t Need Faster APIs. They Need Fewer Calls: Geospatial Scoring with H3 and Spatial SQL ↗

Learning Corner

Build something, go deeper, or learn live, pick your path:

⚙️

Solution Accelerators

Ready-made, industry-focused solutions you can build on

Customer Entity Resolution Start learning →

Entity Resolution for Public Sector Start learning →

Incident Investigation Using Graphistry Start learning →

IOC Matching and Multicloud Query Federation Start learning →

📚

Learning Series

Structured courses to take your skills deeper

Learn Databricks Apache Spark Declarative Pipelines | From Associate to Professional Start learning →

Learn Databricks AI Agents | Build, Deploy & Evaluate on Databricks Platform Start learning →

🎥

Live Training

Free, hands-on sessions with Databricks experts

Level Up Your Databricks Skills: Learning Paths & Free Live Training Join the session →

AI/BI + Genie Launchpad (Oct 20–22) | Free Hands-On Training Join the session →

📌 Technical Blogs

Technical Blog 5 Ways to Enhance Databricks AI/BI Dashboard Tables with HTML Read article →	Technical Blog Beyond the Dashboard: How Transferz Built a Truly Data-Driven Company Culture With AI/BI Genie Read article →	Technical Blog Tutorial: Databricks Genie for Data Engineers and Data Scientists Read article →
Technical Blog Multi-region model serving on Databricks with OpenSharing Read article →	Technical Blog Step-by-Step: Building a Vacation Rental Operations App with AppKit Read article →	Technical Blog How to choose the right AI agent path on Databricks Read article →
Technical Blog Build Your First Databricks App with Codex: A Practical Setup Guide Read article →	Technical Blog TokenMaxing to ValueMaxing with Databricks Read article →	Technical Blog Rebuilding an 800-Line PySpark Pipeline in 200 Lines of Databricks SQL Read article →
Technical Blog Unlocking High-Frequency Workloads with Nanosecond-Precision Timestamps Read article →	Technical Blog From Batch to Milliseconds: Real-Time Feature Views for Recommenders Read article →	Technical Blog Introducing session restore for serverless jobs Read article →