Data Engineering topics

AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=`

phanvy5403 — Fri, 17 Jul 2026 04:54:21 GMT

Hi all,

We are experiencing an AnalysisException when migrating our Databricks cluster runtime from 13.3 LTS to 16.4 LTS (Scala 2.13), and would appreciate your help in identifying the root cause.

**Error:**
```
AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=` on search path
[`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].
SQLSTATE: 42883
```

**Where it fails:**
The exception is thrown inside a Scala `object` class when calling `.where()` on a DataFrame — specifically at:
```
org.apache.spark.sql.Dataset.filter(Dataset.scala:1993)
org.apache.spark.sql.Dataset.where(Dataset.scala:2021)
```
The filter uses the Column DSL (e.g. `col("job_id") === lit(jobId)`), not raw SQL strings.

**What we have tried:**
- Replacing `.where("enabled = 1")` string predicates with Column DSL equivalents (`===`, `=!=`) — same error
- Confirming `SHOW FUNCTIONS LIKE '='` returns the function as expected
- Verifying `rawDF.printSchema()` succeeds — the DataFrame itself is healthy

**Questions:**
1. Is this a known regression or breaking change in the HybridAnalyzer between DBR 13.3 LTS and 16.4 LTS?
2. Is there a recommended way to ensure built-in operators are resolved correctly when the active catalog is not `spark_catalog`?
3. Is setting `USE CATALOG spark_catalog` at session start the intended workaround, or is there a runtime/cluster config that should handle this?

Thank you

Apache Spark 4.2 is officially here! Key architectural updates for AI-Native & Governed Platforms

GabFernandes — Thu, 16 Jul 2026 18:39:01 GMT

Hi community!

Matei Zaharia and the Databricks team just announced the release of Apache Spark 4.2. As a Data Architect, seeing how this engine is evolving to bridge the gap between traditional data engineering, governance, and the AI era is incredibly exciting.

Spark 4.2 is moving away from being just a computational engine to becoming a governed, incremental, and AI-native platform.

Here are the 4 major updates that will impact how we design modern Data Lakehouses:

Metric Views (The Semantic Layer We Needed)

One of the biggest historical pain points in data mesh/lakehouse architectures is business logic drift (e.g., an AI agent calculating revenue differently from a BI dashboard).

What's new: Spark 4.2 introduces governed Metric Views. You can now define business metrics once as a first-class semantic layer. This guarantees consistent catalog resolution, permission application, and identical analytical results across SQL, BI tools, and LLMs.

Native AI Primitives (Vector Search in Spark Engine)

Integrating Generative AI, Retrieval-Augmented Generation (RAG), and recommendation pipelines usually required moving data out of Spark into specialized vector stores.

What's new: Spark 4.2 brings vector primitives directly into the engine planner! This includes vector distance, similarity functions, vector normalization, and NEAREST BY—a top-K ranking join optimized for distance-based matching. This enables vector retrieval, candidate generation, and entity resolution at Lakehouse scale.

Catalog-Managed Flows & Incremental Processing

Unifying batch and streaming workloads has always been the holy grail of lakehouse engineering.

What's new: With Catalog-Managed Flows, incremental and streaming queries now become lifecycle-aware catalog objects. This simplifies the orchestration and governance of real-time streaming pipelines directly under the metadata catalog.

Data Source V2 & Python Profiling

The transition of connectors to the modern DSv2 standard takes another leap forward.

What's new: If your team writes custom Python Data Sources to fetch data, you no longer have to treat them as a performance black box. Spark 4.2 adds built-in profiling for these connectors, making them much easier to tune, debug, and monitor in production.

Let's Discuss!

This release is a huge step forward for performance, developer experience, and semantic consistency.

Which of these features are you most excited to test in your production workloads?
Personally, I see Metric Views as a game-changer for standardizing enterprise KPIs across data and AI applications.

Share your thoughts below! 👇

Are enterprises moving from "Data Lakehouse" to "Agentic Lakehouse"?

Bitrip007 — Thu, 16 Jul 2026 15:39:59 GMT

Introduction several capabilities that seem to shift Databricks from being primarily a data platform to becoming an AI operating system—Genie One, Agent Bricks, AI Gateway, Lakebase Search, and business semantics.

Do you see the future enterprise architecture evolving from:

Data Sources → Lakehouse → BI

Data Sources → Lakehouse → Agents → Business Users

If you're already building production AI solutions on Databricks, what architectural changes are you making after these announcements?

I'd love to hear perspectives from architects, platform engineers, and the Databricks product team.

Serverless egress - public internet access issues

MPD2 — Thu, 16 Jul 2026 09:17:04 GMT

I've just setup a new Databricks account on AWS with a workspace with "Serverless compute and default storage" and I am unable to access the public internet from notebooks and jobs for all but a handful of domains.

A subset works fine, I've identified it can connect to pypi.org, github.com, docker.com, databricks.com

response = requests.get("https://www.databricks.com/robots.txt") display({"status_code": response.status_code, "ok": response.ok}) Returns... {'status_code': 200, 'ok': True}

But others fail...

response = requests.get("https://www.google.com/robots.txt") display({"status_code": response.status_code, "ok": response.ok}) Responds... (": Failed to resolve 'www.google.com' ([Errno -3] Temporary failure in name resolution)"))

What I've verified so far...

Network policy: Serverless egress control is on the default policy, set to Full access (unrestricted outbound).
NCC: No Network Connectivity Configuration is attached to the workspace.
There are no entries in the system.access.outbound_network table.

Bypassing DNS entirely does not help, connecting directly to a resolved IP address fails with "Network is unreachable" so this isn't a resolver problem, I think it's a routing/egress problem at the network layer.

I have tried defining a custom Network policy in the account testing both Serverless egress control with "Allow access to all destinations" and "Restricted access to specific destinations" with a subset of domains but that has not had any impact (I have been terminating and restarting the serverless compute between each change and executing the notebook).

I have also tried setting up new workspaces in different AWS regions but the same problem persists.

My understanding of Serverless compute is that it is running within a Databricks managed network environment and not my AWS account is that correct? i.e. it's not running within a VPC where I have control over ingress/egress

Any help or suggestions appreciated.

My Apps get stopped automatically

Abarghodrat — Thu, 16 Jul 2026 08:03:39 GMT

Hi team,

I have 2 applications in my workspace which they will get stopped sometimes without me knowing the reason. I’m on premium account and I have to run them when I can’t have access to my app.
I need help.

Lakeflow Connect SharePoint - 403 Error

ChristianRRL — Wed, 15 Jul 2026 19:24:21 GMT

Hi, I'd like some help clarifying how lakeflow connect pipelines work (or should work). I went through the steps to (1) set up a connection, and (2) setup the ingestion, but I keep running into the following error:

Operation failed: "This request is not authorized to perform this operation.", 403

On checking the Ingestion Pipeline, I noticed that it's spinning up what I suspect are Job Clusters, but I'm not sure if the issue we're seeing here is due to either (A) the job cluster not having proper access due to some kind of network whitelisting issue for example, or (B) some issue with the access of the unity catalog/schema itself, or (C) something else entirely I may be missing.

Would appreciate some assistance here! Below are some related screenshots:

How to Deploy Lakehouse Federation Using DABs from the Dev Environment to the Stage Environment

shan-databricks — Wed, 15 Jul 2026 08:12:43 GMT

We are using Lakehouse Federation to connect to a SQL Server database and are currently looking for a CI/CD deployment solution using DABs. Could you please advise on the available options for deploying from the Dev environment to the Stage environment?

SCIM API shows only databricks-sql-access,but Workspace Access is still displayed (grayed out) in UI

Sainath368 — Tue, 14 Jul 2026 13:33:29 GMT

Hi Team,

I'm testing Databricks entitlement automation using the Workspace Groups SCIM API.

I have an Entra ID group that is synchronized to the Databricks Account via SCIM. I'm using a Service Principal authentication to manage workspace-level entitlements through the SCIM Groups API.

I patched the group with the following entitlement:

When I retrieve the group through the SCIM API, I get the following response:

Based on the API response, it appears that the group has only databricks-sql-access assigned directly.

However, in the Databricks workspace UI under:

Settings → Identity and Access → Groups → Group01

I see:

Workspace Access = Enabled (grayed out)
Databricks SQL Access = Enabled

Since Workspace Access is grayed out, it looks like it may be inherited. However, the SCIM API response shows "groups": [], which suggests the group is not a member of another parent group.

My questions are:

Does the SCIM API return only directly assigned entitlements while the UI shows both direct and inherited entitlements?
If Workspace Access is inherited, how can I determine the source of inheritance when the group's groups attribute is empty?
What is considered the source of truth for entitlement automation—the SCIM API response or the workspace UI?

I'm trying to build automation to onboard groups to workspaces and manage entitlements through JSON-driven configuration, so I want to ensure I correctly understand the entitlement model before proceeding.

Thanks in advance for any guidance.

ALTER Not Working in Databricks for external Table.

Taj2019 — Tue, 14 Jul 2026 06:55:49 GMT

Hi Team,

i am getting error alter not suported .

col1 DOUBLE NOT CONVERTING TO col1 DECIMAL(31,5) VIA ALTER STATEMENT.
ERROR-
liquibase.exception.DatabaseException: [Databricks][JDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: 0A000, Query:
ALTER TABL***, Error message from Server: org.apache.hive.service.cli.HiveSQLException: Error running query: [NOT_SUPPORTED_CHANGE_COLUMN]
org.apache.spark.sql.catalyst.ExtendedAnalysisException: [NOT_SUPPORTED_CHANGE_COLUMN] ALTER TABLE ALTER/CHANGE COLUMN is not supported for.. Any suggestion , would be appreciate.

Genie Space and App Error

emsmarcela — Mon, 13 Jul 2026 20:42:59 GMT

I built an App with a chatbot which generates queries and accesses data through a Genie Space. When making questions in the chatbot now, I keep getting the following error in the Genie monitoring page:

An error happened. Please try again later.

We're sorry, but an unexpected internal error has occurred. Please try again in a few minutes. If the issue persists, please contact your Databricks account team.

Details: INTERNAL_ERROR: AI service did not respond with a valid answer.

I have not made changes to my app code. How can I fix this?

What timezone is the “timestamp” value on a table's history?

yit337 — Mon, 13 Jul 2026 13:13:15 GMT

Is it UTC, or the local zone of whoever accesses it?

Trunked-based development model for Databricks question

theanhdo — Mon, 13 Jul 2026 07:18:48 GMT

Hi there,

My name is William Do (a senior engineer at Hub24, Australia). I am writing to seek for your advice on a Trunk-based development model that we are hoping to implement for our Databricks project.

Problem statement

To give you some context, I am providing you the problem that we are trying to solve here. In our DBX project, currently we have adopted the Git Flow development model with the following environment alignment

Branch	Environment
develop	DEV
integration	TEST
master	DEMO
master	PRODUCTION

Developers create a feature branch from develop branch, write their code, create a pull request, merge the feature branch back to develop branch, and deploy develop branch to DEV environment
Two weeks before the Production release, we do a code cutoff, create a pull request to merge develop branch to integration branch, and deploy integration branch to TEST environment
One day before the Production release, we do another code cutoff, create a pull request to merge integration branch to master branch, and deploy master branch to DEMO environment
On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

A major limitation of this model is that we can’t iterate quickly. Code changes would have to wait for almost one month before being released to PRODUCTION.

Next step

We have been exploring the Trunk-based development model to quickly release changes to Production. The environment alignment is given below.

Branch	Environment
master	DEV
master	TEST
master	DEMO
master	PRODUCTION

Developers create a feature branch from master branch, write their code, create a pull request, and merge the feature branch back to master branch
Then we deploy master branch to these three environments DEV, TEST and DEMO
On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

Although this model appears simple and allows for rapid development iteration, it has limitations

If we have a long-running feature branch which would require two weeks of testing, then at the time we merge this feature branch to master branch and deploy to DEV, TEST and DEMO, effectively we block master branch and these three environments for 2 weeks. Specifically, if we have other short-running feature branches which would require 1 or 2 days of testing, then if we merge to master branch, these changes can’t be deployed PRODUCTION after 2 weeks. In the worse-case scenario, if we have an urgent hotfix, we can’t simply merge it to master, test it independently and then deploy to PRODUCTION.
If we have multiple long-running feature branches, then this situation is getting worse because all the testing needs to be completed before deployment.

Our practical experience shows that

In financial business, our Databricks project and our code is highly complex to support complicated business rules and calculations
Code changes would require sufficient time to test and to be stable before releasing to Production
The large, increasing, and complex amount of data makes it hard to ensure high-quality test coverage
Feature flags approach is not always possible especially when dealing with schema and query changes

Proposed approach

To mitigate the limitations mentioned, we “cooked” the Trunk-based development model to fit our project.

Branch	Environment
develop	DEV
master	TEST
master	DEMO
master	PRODUCTION

For long-running feature branches, developers would create a feature branch from develop branch, write their code, create a pull request, merge the feature branch back to develop branch, and deploy develop branch to DEV environment
Testing for long-running features can be performed in DEV environment. When testing is complete, then we merge develop branch to master branch
For short-running branches, developers would create a feature branch from master branch, write their code, create a pull request, merge the feature branch back to master branch
We deploy master branch to these two environments TEST and DEMO for testing. For long-running feature branches, the testing in TEST and DEMO would be minimal as it has been done in DEV environment. For short-running features, the testing would be minimal due to the small changes
On the day of the Production release, the code in master branch is deployed to PRODUCTION environment

Seeking for your advice

We are uncertain whether the proposed method is suitable for implementing a trunk-based development model in Databricks, and we would greatly appreciate your advice to help validate this approach and any experience you or your team may have that would assist us

Given your expertise and experience working with numerous clients, you likely already have a solution to our problem; therefore, we would greatly appreciate your advice on the optimal approach for implementing a trunk-based development model on Databricks

We look forward to hearing from you. Thanks in advance

Kind regards

William

Clarification on Auto Loader Managed File Events with Unity Catalog Managed Volumes

APJESK — Mon, 13 Jul 2026 03:37:05 GMT

Hi Databricks Team,

I'm trying to understand how Auto Loader Managed File Events work with Unity Catalog Managed Volumes, and I'm looking for some clarification.

My understanding

I create an External Location:

CREATE EXTERNAL LOCATION app_ext_loc
URL 's3://my-bucket/catalog/cat1'
WITH (STORAGE CREDENTIAL my_storage_credential);

I enable File Events on this External Location.
I create a Managed Volume or Managed Table that uses managed storage under this External Location.

The physical storage becomes something like:

s3://my-bucket/catalog/cat1/__unitystorage/catalogs/<catalog-id>/...

My confusion

The documentation says that enabling File Events on an External Location configures S3 Event Notifications for the bucket/prefix and allows Auto Loader to use:

.option("cloudFiles.useManagedFileEvents", "true")

However, Managed Volumes and Managed Tables are stored under the internally managed __unitystorage path.

My questions are:

Does Databricks configure S3 Event Notifications for the entire External Location prefix (for example, catalog/cat1/), so events are also generated for objects created under __unitystorage?
If yes, can Auto Loader with cloudFiles.useManagedFileEvents=true be used to ingest data from a Managed Volume, or is this feature officially supported only for External Volumes/External Locations?
If Managed Volumes are not intended to use Managed File Events, what is the purpose of enabling File Events on an External Location that is also used as the managed storage location for Unity Catalog?
Is the recommended architecture:

External Volume + Managed File Events for ingestion from cloud storage
Managed Volume + Directory Listing for Databricks-managed storage?

I'm looking for clarification on the intended architecture because the documentation on External Location File Events and Unity Catalog managed storage seems to overlap, and I'd like to understand the supported design pattern.

Thank you.

50% discount voucher not received despite completing all 4 courses

Avinash_Turpu — Wed, 15 Jul 2026 13:44:21 GMT

Hi team,

I completed all 4 required courses for the recent Learning Festival campaign (June 15 – July 6, 2026):

- DevOps Essentials for Data Engineering

- Build Data Pipelines with Lakeflow Spark Declarative Pipelines

- Deploy Workloads with Lakeflow Jobs

- Data Ingestion with Lakeflow Connect

All four show "Completed" in My Activities, though the 4th course

(Data Ingestion with Lakeflow Connect) shows "Course is under

maintenance" — not sure if that's affecting eligibility.

I have not received the 50% discount voucher email. I've checked

spam/junk folders with no luck.

Could you please check my eligibility and resend the voucher?

Thank you!

Declarative Asset Bundle Service Principals best practices

DazzaiDe — Thu, 09 Jul 2026 08:21:57 GMT

I have 2 service principals for deployment and runtime like databricks suggested, since declarative asset bundles dont include delta tables creation what should i just let my runtime to create table on the first run of a specific job?

I want the community opinion and best practices out there.

Thanks!

Permissions on job with SQL task

stravi001 — Wed, 08 Jul 2026 18:52:15 GMT

Hello,

I am authenticated as GCP service account. I have to create query, run job as service principle and get result.

I'm both cases, I set permissions CAN_MANAGE for group "users" for both query and job. When I create the query with RunAsMode.OWNER, it gets limited permissions of invoker SP, but I can then access output of the task. When I create query with RunAsMode.VIEWER it gets required permissions of RunAs SP, but I am not able to access output of the task. I get error: insufficient privileges to access object queryExecutionSessions/*. As I understand from the roles description, it should have permissions with CAN_MANAGE.

Thank you for help.

select table_changes() return unexpected results

hdu — Wed, 08 Jul 2026 15:39:25 GMT

Hi Folks,

My understanding is

SELECT *
FROM table_changes('mytable', x)

should return all changes from version x through the latest available version, regardless of when I run it. However what happen to me is:

When immediately after a streaming table update it only returns changes for version x, this is as expected.
but after some times, it returns all versions 1, 2, ..., x-1, x, x+1, ..., this is not as expected. It should only return versions x, x+1, ....

in my example below, I expect the second query only return one row: 2026-04-08 8676 from version 25 only, but it return other versions as well. Any thoughts why?

Advanced Learning Festival (15 June - 06 July 2026) - Voucher Eligibility Confirmation

rjcrystal786 — Wed, 08 Jul 2026 14:53:29 GMT

Hello @Jim ,

I completed all four modules of Learning Pathway 1 (Associate Data Engineering) for the Advanced Learning Festival on July 6, 2026, the published end date of the event. My Academy account email is [datasci.patel.rupesh@gmail.com]. Screenshots of my completed courses from My Activities are attached. Could you please confirm my eligibility for the 50% certification voucher?

Thank you.

Getting start with Data Brick machine creation.

Shaily_DBA — Wed, 08 Jul 2026 08:31:17 GMT

I am a beginner for Data Bricks. Want to learn from the scratch.

Is your AI strategy driving real impact or just more pilots?

Inument — Wed, 08 Jul 2026 07:55:02 GMT

AI only matters when it improves workflows, speeds up decisions, and creates measurable business value.

So, what would real AI impact look like inside your company?