Data Engineering topics

About Azure SQL Managed Connector

srikanthp24 — Fri, 16 Oct 2026 18:02:01 GMT

Hi Community,

We are trying to configure a Databricks Lakeflow Connect managed SQL Server connector to ingest data from Azure SQL Database into Databricks.

We are using a contained SQL database user that has access only to the specific source database, with no access to the "master" database.

However, during connection validation, we are encountering the following error:

Authentication failure while obtaining the connection to the source database: Error while creating connection pool for catalog master. Login failed for user '<contained_db_user>'.

Our understanding:

It appears that the Lakeflow Connect managed SQL Server connector attempts to establish a connection to the master database during connection validation, even though the intended source is a specific Azure SQL database.

Since we are using a contained database user, authentication to "master" fails.

Our security requirement:

We need to restrict the service account to the source database only, without granting access to "master".

Questions for the community:

Has anyone successfully configured the Lakeflow Connect managed SQL Server connector using a contained database user with access restricted to a single Azure SQL database?
Is there a supported configuration to bypass master database authentication and connect directly to the specified source database?
Is access to "master" mandatory for this connector, even when ingesting from only one database?
Are there any recommended workarounds that preserve database-level security isolation while still using the managed Lakeflow Connect connector?

We would prefer to use the managed Lakeflow Connect connector rather than building a custom JDBC ingestion framework.

If you have any better approach suggest?
Any insights, experiences, or recommendations would be greatly appreciated.

Thanks!

External location Manage permission

orlandovallejos — Fri, 16 Oct 2026 16:34:10 GMT

Hi team, I'm trying to update the owner of an external location and I see the following message:

Requires permission MANAGE on external location 'location_name'

The owner of the resource, the user that created it doesn't have that permission? How is that possible?

ok, let's say the owner doesn't have it by default, how I add the MANAGE permission to this resource?

Thanks

Zerobus Variant Data not Processed as Expected (json data destined for Variant column).

DB1To3 — Fri, 16 Oct 2026 14:10:05 GMT

Lets say I have a table receiving zerobus data (json) over HTTP/rest.

That last VARIANT is "ValueJson".

If I want to submit data to it, I would expect to be able to use normal json formatting:

[ { "a": "E", "b": 123, "c": "xyz", "ValueJson": { "x" : "y" } } ]

Unfortunately it doesn't want that part of the record to be ACTUAL json. The error is like so

Record decoder/encoder error: invalid type: map, expected a string at line 13 column 24. Error Code: 4044, Error State: 3.

Unfortunately the fix is to send json-encoded json (escaped json in a string, since the error says it expected a string).

[ { "a": "E", "b": 123, "c": "xyz", "ValueJson": "{ \"x\" : \"y\" }" } ]

Does this seem reasonable behavior, or is it a bug? I think it is a bit unnatural to have to encode the json VARIANT instead of just nesting it inside the record as normal json. Ideally the use of this VARIANT wouldn't force us towards using json-encoded-json. It isn't the end of the world, but it seems like a very unfortunate way of sending data from a remote client - one that shouldn't need to have a deep understanding of the eccentricities of the VARIANT-parsing rules for zerobus.

Zerobus rescue (VARIANT) not working. No Preview Available.

DB1To3 — Fri, 16 Oct 2026 13:45:42 GMT

Has anyone tried using the rescue feature for json in zerobus-ingest? It seems like a very significant feature, assuming the schema incoming json is even 1% outside the control of team that may be ingesting on the databricks side. There needs to be a way to ingest a json record that is slightly non-conforming. Else you get unhelpful errors.

Here is the doc:

... but I'm not able to get this working, and there is no "preview" switches for this in my DBX account settings.

The table is configured with the NULLable, tagged column as required:

Yet whenever I submit json row data (using currently HTTP/rest), I just get an error that says zerobus doesn't approve:

Record decoder/encoder error: unrecognized field name 'whatever' at line 2 column 24. Error Code: 4044, Error State: 3.

I looked for a preview in my account settings and there is none listed (supposed to be "Beta" according to docs).
Any advice would be appreciated. Let me know if I'm misunderstanding the intended behavior of this rescue column.

Datavault implemention on Databricks

ankit2604 — Fri, 16 Oct 2026 06:01:25 GMT

Hi everyone,

I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.

While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on complex Business Vault objects.

Would love to hear from anyone who has built a native framework from the ground up!

Thanks

Issue with query editor - auto suggest does not respect the casing of columns in CTE

Ram_Sundar — Thu, 15 Oct 2026 08:26:15 GMT

This behaviour (see image) of auto-suggest in the new editor makes the usage of auto-suggest pretty much useless to me.

I prefer PascalCase for column names in CTEs. Auto-suggest defaults to lowercase.

I need to type out everything. The older version of the editor supported this.

Is there a workaround?

Databricks Connect + AutoDependencies + UDF + Streaming breaks

i4ata — Wed, 14 Oct 2026 17:46:50 GMT

I have this toy python script:

CATALOG = 'mpt_usd_gbl_bronze_dev' # <-- replace
SCHEMA = 'mpt_gbl_tco_fof' # <-- replace

src=f'{CATALOG}.{SCHEMA}.toy_source'
DEST = f'{CATALOG}.{SCHEMA}.toy_dest'

from databricks.connect import DatabricksSession, DatabricksEnv
from pyspark.sql import functions as F, SparkSession, DataFrame

env = DatabricksEnv().withAutoDependencies(upload_local=True, use_index=True)
spark: SparkSession = DatabricksSession.builder.serverless().withEnvironment(env).getOrCreate()

spark.sql(f'CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}')
spark.sql(f'DROP TABLE IF EXISTS {SRC}')
spark.sql(f'DROP TABLE IF EXISTS {DEST}')

(
    spark.createDataFrame(
        [('Alice', 1), ('Bob', 2), ('Claire', 3),],
        schema='name string, value int',
    )
    .write.format('delta')
    .saveAsTable(SRC)
)


def test(batch_df: DataFrame, batch_id: int):
f = F.udf(lambda x: 'here')
(
    batch_df
    .withColumn('here', f(F.col('file_name')))
    .write.format('delta').mode('append')
    .saveAsTable(DEST)
)

(
    spark
    .readStream.format('delta').table(SRC)
    .writeStream.foreachBatch(test)
    .trigger(availableNow=True).start().awaitTermination()
)

It (1) initializes databricks-connect (2) loads dependencies (3) creates a UDF (4) runs a streaming query. I am getting this error:

Traceback (most recent call last):
File "/home/ivaylo-rusinov/tieto/magna/MPT_Cloud_Data_Analytics/MPT_UNS_Engineering_Pipeline_3S/PipelineConfigurations/MPT_GBL_Workspace/SourceAligned_GBL_TCO_FoF/tests3.py", line 40, in <module>
.trigger(availableNow=True).start().awaitTermination()
^^^^^^^
File "/home/ivaylo-rusinov/python_312_env/lib/python3.12/site-packages/pyspark/sql/connect/streaming/readwriter.py", line 793, in start
return self._start_internal(
^^^^^^^^^^^^^^^^^^^^^
File "/home/ivaylo-rusinov/python_312_env/lib/python3.12/site-packages/pyspark/sql/connect/streaming/readwriter.py", line 755, in _start_internal
_, properties, _ = self._session.client.execute_command(cmd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ivaylo-rusinov/python_312_env/lib/python3.12/site-packages/pyspark/sql/connect/client/core.py", line 1880, in execute_command
data, _, metrics, observed_metrics, properties = self._execute_and_fetch(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ivaylo-rusinov/python_312_env/lib/python3.12/site-packages/pyspark/sql/connect/client/core.py", line 2512, in _execute_and_fetch
for response in self._execute_and_fetch_as_iterator(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ivaylo-rusinov/python_312_env/lib/python3.12/site-packages/pyspark/sql/connect/client/core.py", line 2266, in _execute_and_fetch_as_iterator
req = hook.on_execute_plan(req)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ivaylo-rusinov/python_312_env/lib/python3.12/site-packages/databricks/connect/auto_dependencies/hook.py", line 110, in on_execute_plan
obj = obj[0]
~~~^^^
TypeError: 'function' object is not subscriptable

Is this intended behavior? I assume this is quite a common workflow. How can I fix this?

How do you preserve PostgreSQL table ownership when an app resource is recreated?

A_Motia — Wed, 14 Oct 2026 01:49:20 GMT

Hi Community members,

I'm looking into a potentially tricky deployment scenario involving Databricks Apps, Lakebase, and Databricks Asset Bundles (DABs).

Consider an application that uses Lakebase as its PostgreSQL backend. The app's service principal owns some application tables, while a separate service principal running a synchronization job owns other tables.

If a deployment or configuration change removes and recreates the app's Lakebase resource, what is the recommended way to preserve database roles, grants, and table ownership?

A few specific questions:

Is there a supported way to restore ownership of existing tables to the original service principal if its PostgreSQL role is removed and recreated?
What is the recommended pattern for managing application schema migrations with Alembic when different service principals own application tables and synchronized tables?
Should database roles, grants, and ownership be managed separately from the app's DAB resource lifecycle to prevent deployment changes from affecting existing data?
What recovery strategy would you recommend if the app reconnects successfully but subsequently loses the privileges needed to alter or drop existing tables?

I'm particularly interested in production-safe approaches that avoid manual database repair after deployments.

Thanks!

Production table maintenance

A_Motia — Wed, 14 Oct 2026 01:44:05 GMT

Hi Community members,

I'm interested in how people monitor the effectiveness of Liquid Clustering as a Delta table grows over time.

Consider below scenario:

A table initially performs well, but MERGE operations gradually become slower even though the clustering keys haven't changed.

Which metrics do you monitor to determine whether the issue is stale file layout, poor data skipping, or source-side processing?
How do you decide when to run OPTIMIZE versus reconsider the clustering keys?
Do you use Delta operation metrics from DESCRIBE HISTORY to track when performance starts degrading?

I'd appreciate examples of how you've diagnosed this in production system in an automated fashion

issues creating a foreign catalog

DaveCh — Tue, 13 Oct 2026 09:28:15 GMT

I've created a foreign connection to an on prem sql server...

CREATE CONNECTION IF NOT EXISTS `dagrbs-dr01`

TYPE SQLSERVER

OPTIONS (

host = secret('scope-kv-key-vault', 'ip-dagrbs-dr01'),

port = '1433',

user = secret('scope-kv-key-vault', 'username-dagrbs-dr01'),

password = secret('scope-kv-key-vault', 'password-dagrbs-dr01')

next step to create the foreign catalog returns...

summary: [DATA_SOURCE_OPTION_CONTAINS_INVALID_CHARACTERS] Option host must not be empty and should not contain invalid characters, query strings, or parameters. SQLSTATE: 42602

the schema and tables are compliant with databricks naming conventions, but the server name (not specified in the connection) does include a hyphen.

Is this the issue, or am I missing something else?

Are PATs (user or service principal) accepted by Databricks Apps?

databrickspooja — Tue, 13 Oct 2026 09:09:14 GMT

Is the 401 expected behavior, and is it documented anywhere?
Is there a supported way for an external system with only a static header to call a Databricks App?

I want to receive webhooks from an external SaaS product. The sender can only POST to a public HTTPS URL, and it has a beta feature that lets me add static custom header values. It can’t run an OAuth flow or refresh tokens.

what I've tested:

A service principal with Can USE on the app, and an OAuth M2M secret. Calling the app with the OAuth access token works. The request reaches my code.
A PAT for the service principal, sent to the app URL. This returns HTTP 401 with an empty body, so the app proxy rejects it before my code runs.

Best Practices for Managing Permissions in Databricks Workspaces

oliviaanne — Tue, 13 Oct 2026 06:21:34 GMT

Hi everyone,

I’m interested in learning how other teams manage user permissions and access control in Databricks workspaces, especially when multiple users and groups need different levels of access.

For example, when working with SQL Warehouses, Unity Catalog, and hared data resources, what approach do you find most effective for keeping permissions organized and secure?

Do you prefer managing permissions directly through the Databricks UI, using REST APIs, or automating everything with Terraform?

I’d love to hear about your experiences, any challenges you’ve faced, and the best practices you would recommend for making permission management easier as a team grows.

Thanks in advance for sharing your insights!

Genie spaces with DABs - how do you deal with the table ordering?

juanlozadab — Mon, 12 Oct 2026 21:03:50 GMT

Hi all,

I'm about to start managing Genie spaces as code with Asset Bundles and wanted
to ask before I build it.

From what I'm reading, the genie_spaces resource needs the tables in its
data_sources to already exist when you deploy. That's fine for an environment
that's already running, but it seems to break the first deploy to a new target,
where the same bundle is creating the schema and tables and then declaring the
Genie space on top of them.

I could split it into two deploys, but then I lose the single command deploy
that's half the point of using bundles.

For anyone already doing this:

Are you keeping the Genie space in the same bundle as the tables it reads, or
did you end up separating them?

And one I haven't found an answer to anywhere: on a redeploy, what happens to
the instructions and example queries people add to the space through the UI?
Does the bundle overwrite them?

Thanks

Lakeflow Connect SQL Server (integrated CDC pipeline): no way to Include/exclude columns

Oumeima — Mon, 12 Oct 2026 17:11:42 GMT

Setup:

- Lakeflow Connect, SQL Server source, `connector_type: CDC`, integrated pipeline (gateway-less — extraction and apply in one triggered pipeline on classic compute)
- Deployed with Declarative Automation Bundles, `databricks-bundles` 1.20.0, CLI current
- Migrated from the gateway + ingestion-pipeline pair, where `table_configuration.include_columns` was in use and working

Problem:

We don't need all the columns from the tables we're ingesting. So we need to include or exclude a selection.

When adding the parameter "include_columns" under "table_configuration" (the list that worked on the gateway-based pipeline) on the integrated pipeline, the bundle deployment fails with:
```
Column selection (include_columns / exclude_columns) is not supported for Some(SQLSERVER).
```

The "TableSpecificConfig" model in "databricks-bundles" documents both fields as "[Public Preview]" with no connector caveat, and the SQL Server connector docs show them in the example config, so the rejection looks like a service-side limitation specific to the integrated SQL Server connector rather than a client issue.

Questions:

- Is column selection (`include_columns` / `exclude_columns`) expected to work on the integrated CDC pipeline for SQL Server?

Happy to share more details about this!

Genie "Field" error: Table name or alias is required for column 'amount'

dkxxx-rc — Mon, 12 Oct 2026 16:24:44 GMT

The documentation at https://docs.databricks.com/aws/en/genie-agents/tune-quality#define-sql-expressions says that you can create a "Field" in a genie agent (in the Examples section of config) with the code

CASE WHEN amount < 10000 THEN 'Small' WHEN amount < 50000 THEN 'Medium' ELSE 'Large' END

However, entering this code yields an error

Table name or alias is required for column 'amount'

presumably because the field names aren't specified as belonging to a specific table.

So #1, that's an error in the documentation, right?

#2, what does "alias" mean in the error message? I get that if I were creating an example Query, I could create table aliases for the result of the query. But such aliases do not appear to be valid for using in the Field definition.

Is there a separate method for defining an "alias" in the context of a Genie Agent?

Supervisor Agent Deployments!

karuppusamy — Mon, 12 Oct 2026 12:14:28 GMT

Hi Team,

Could someone please help me deploy the Supervisor Agent using Azure DevOps? I would appreciate it if you could guide me through the deployment steps as well.

Thank you.

ServiceNow ingestion failed since October 1st.

scott_kao — Sat, 10 Oct 2026 21:09:18 GMT

I have a data ingestion pipeline ingesting ServiceNow data into databricks. It has been working for several months, but it stopped working since October 1st.

From what I see, the pipeline suffered connection time out.

However, from the connection, I can see the token being refreshed properly, and the application registry in servicenow shows the account is active.

I have tried everything I can, but I just can't seem to get pass this error. I am wondering if anyone else experienced this and how you fixed it?

Declarative Pipeline with editable wheel: PYTHON.MODULE_NOT_FOUND_ERROR

a_user12 — Sat, 10 Oct 2026 18:33:22 GMT

Hi!

I create a bundle with

databricks bundle init

and i let create a pipeline including a package.

The pipeline definition looks like the following

# The main pipeline for pipelinewithpackage resources: pipelines: pipelinewithpackage_etl: name: pipelinewithpackage_etl catalog: ${var.catalog} schema: ${var.schema} serverless: true root_path: "../src/pipelinewithpackage_etl" libraries: - glob: include: ../src/pipelinewithpackage_etl/transformations/** environment: dependencies: # We include every dependency defined by pyproject.toml by defining an editable environment # that points to the folder where pyproject.toml is deployed. - --editable ${workspace.file_path}

The package looks like the following

Now, I want to use the package in my pipeline. So I simply added

from pipelinewithpackage import taxis

before my @DP.table defintions. This is the only modificaiton which I did - the rest of the code is the generatedbundle code.

I deploy the bundle and run it: then i get the ModuleNotFoundError: No module named 'pipelinewithpackage' error.

In the cluster logs I can see, that the wheel is created:

Building wheels for collected packages: pipelinewithpackage Building editable for pipelinewithpackage (pyproject.toml): started Building editable for pipelinewithpackage (pyproject.toml): finished with status 'done' Created wheel for pipelinewithpackage: filename=pipelinewithpackage-0.0.1-py3-none-any.whl size=1433 sha256=492c44e0c2d9eb063b3f3c01a6fffe9d1699de5123a389a0e8b5708bbf61060c Successfully built pipelinewithpackage Installing collected packages: pipelinewithpackage Successfully installed pipelinewithpackage-0.0.1

Why isn't it finding the library?

Thank you!

Beyond Bronze, Silver, and Gold: When Does a Separate Stage Make Sense?

juanlozadab — Fri, 09 Oct 2026 16:44:26 GMT

I recently published my first technical blog! I was a little nervous about sharing it, but I’m happy to take this first step. I hope to keep sharing what I learn and improve my writing along the way.

For this article, I chose a question I’ve encountered in real projects: when does it make sense to split a Medallion layer into separate stages?

The article explores how to make that decision based on responsibilities, reuse, data quality, ownership, and consumer needs. It also separates three decisions that can easily become mixed together: organizing transformation logic, storing intermediate results, and making outputs available to consumers.

It includes a practical Silver example using a private materialized view, expectations, and a quarantine output, followed by a discussion of how logical layers relate to Unity Catalog schemas.

My main takeaway is that a layer should have a clear responsibility and a reason for its output to exist.

Read the article here:
https://www.sunnydata.ai/blog//blog/medallion-architecture-databricks-layers

I’d be interested to hear what you think and how you approach these decisions in your projects. Suggestions on what I could improve or explain more clearly would also be welcome.

Propagating SCD2 History from Silver to Gold with AUTO CDC FLOW and Handling Full Refresh

yit337 — Fri, 09 Oct 2026 14:05:50 GMT

I have a streaming SCD2 table in the Silver layer, and I need to propagate all historical and future SCD2 events into another SCD2 streaming table in the Gold layer.

The Gold table is maintained through a DLT pipeline using two AUTO CDC FLOWs:

Initial history load
- Uses once=True.
- Copies the entire Silver table into Gold.
- This is required because the source table's CDF does not contain the complete historical event history.
Incremental updates
- Continuously reads from the Silver table's Change Data Feed (CDF).
- Captures new changes as they occur.

This setup works fine during the initial deployment:

The Silver table already contains historical SCD2 records.
The CDF is empty
The once=True flow loads all existing records into Gold.
The second flow then keeps Gold synchronized using CDF going forward.

The challenge appears when performing a full refresh of the Gold table.

At that point:

The Silver table still contains the complete SCD2 history.
The Silver table's CDF now also contains some historical change events (not empty, but it does not contain full history because it has retention period).
If both flows are executed again:
- The once=True flow reloads the entire Silver table.
- The incremental CDC flow replays changes from CDF.
As a result, duplicate historical events are written into the Gold table.

Although full refreshes are expected to be rare, I'd like to have a robust solution before moving this design to production.

Is there a recommended Databricks pattern for handling this scenario? Have you implemented a similar architecture using AUTO CDC FLOW / Lakeflow Declarative Pipelines, and if so, what approach did you take?