All Data Engineering posts

Databricks Latest Features: Genie Code CLI, App Builder, Agent Memory and DABs Deployment History

rajkumar8k — Sun, 15 Nov 2026 14:30:31 GMT

Hi everyone,

Databricks has shipped a lot of new features recently. Here’s a quick summary of the most useful ones, grouped by area.

1. DABs Deployment History

What it is:
Databricks Asset Bundles (DABs) now keep a history of every deployment. Each record shows the deployment version, its status, and the resources it manages.

Why it matters:
Until now, working out what changed between two bundle deployments meant digging through Git history or comparing configurations by hand. Deployment history shows those changes directly. For example, if you raise a job’s timeout from 60 to 120 seconds and redeploy, the history records exactly that change.

Where you’d use it:

Auditing changes in production environments
Troubleshooting when a job starts behaving differently after a deployment
Getting better visibility in CI/CD pipelines built on DABs

2. Genie Code CLI

What it is:
Genie Code, the Databricks AI coding assistant, now runs from your terminal. It works with your local files and Databricks workflows.

Why it matters:
Many developers spend most of their time in a local IDE and terminal. With the CLI, you get AI help without switching to the workspace UI. You can ask it to compare two bundle files, upload files, or help with workflow-related tasks.

Good to know:
Model usage from the CLI is billed through Unity Gateway, so cost tracking and governance stay in one place.

Where you’d use it:

Reviewing changes between bundle configurations
Doing routine file and workflow tasks faster
Local-first development

3. Genie App Builder

What it is:
Describe the app you want in plain language, and Genie App Builder creates a Databricks App. You can then keep refining it through chat.

Why it matters:
Building even a simple internal app used to take frontend skills and setup time. Now you can go from an idea to a working prototype in minutes. The original article’s example is a news tracker with sample data that filters six cards by topic, built from a short description.

Governance:
Building, sharing and permissions are controlled through App Spaces, so teams can create apps quickly without losing control over who builds and who can access them.

Where you’d use it:

Quick internal tools and dashboards
Prototypes to show business stakeholders
Letting analysts and less technical users build simple apps

4. Materialized Views and Streaming Tables from Python

What it is:
You can now create standalone materialized views and streaming tables directly from a serverless notebook, using the @DP.materialized_view and @DP.table decorators.

Why it matters:
Previously these objects were usually defined inside a full declarative pipeline. Now, for simpler cases, you can define them right in a notebook in Python, which means less setup and faster development.

Where you’d use it:

Quick incremental aggregations
Lightweight streaming ingestion
Testing transformation logic before moving it into a full pipeline

5. JDBC Connections in Unity Catalog (Generally Available)

What it is:
You can save a JDBC driver, endpoint and credentials once as a Unity Catalog connection, then use it by name in any notebook.

Requirements:
Serverless compute and DBR 19 or above.

Why it matters:
Teams often copy connection strings and credentials across many notebooks, which is a maintenance headache and a security risk. With UC JDBC connections:

Credentials are stored once and governed centrally
Access is controlled through Unity Catalog permissions
Notebooks stay clean, with no hardcoded connection details

Where you’d use it:

Reading from external databases such as MySQL, PostgreSQL or SQL Server
Sharing approved connections across teams safely

6. Audio Transcription in SQL with ai_transcribe (Beta)

What it is:
A new AI function that turns audio into timestamped text with speaker labels, called directly from SQL.

Languages: English and Spanish (currently)

Why it matters:
Audio data used to need separate speech-to-text services and custom pipelines. Now you can transcribe audio with a simple SQL query, then analyze the text with other AI functions such as ai_summarize, ai_classify or ai_analyze_sentiment, all within Databricks.

Where you’d use it:

Call-center analytics (customer sentiment, complaint detection)
Meeting and interview transcription
Compliance monitoring of recorded calls

7. Agent Memory

What it is:
AI agents can now store useful facts and user preferences and use them in later conversations. The memory is stored in Lakebase.

Why it matters:
Without memory, every conversation with an agent starts from scratch. With Agent Memory, the agent remembers context. In the original article’s example, a user’s preferred news format is saved and used in a later session, without the earlier conversation being sent again.

Where you’d use it:

Personalized assistants and chatbots
Customer support agents that remember past interactions
Internal copilots that learn team preferences over time

Re: Issue with CDC ingestion pipeline

rajkumar8k — Sun, 15 Nov 2026 14:14:19 GMT

Hi Michael,

I’ve seen this error with Lakeflow Connect database connectors. The s3:/ with nothing after it suggests the pipeline couldn’t work out a storage location for its staging data.

The ingestion gateway first writes the change data to a Unity Catalog volume in the staging catalog and schema you chose during setup. If that catalog or schema has no managed storage location, or the location is set up wrong, the path comes out empty and you get this error.

Things worth checking:

Staging catalog/schema storage: Make sure the staging catalog (or schema) has a valid managed location backed by an external location, or that your metastore has a root storage location.
Permissions: The user or service principal running the pipeline needs rights on the external location and storage credential, plus CREATE VOLUME on the staging schema.
Default storage catalogs: If the staging catalog uses default storage, try a catalog with your own S3 managed location instead.
Event log: Look at the gateway pipeline’s event log for a more detailed error than the main pipeline shows.

If you point the gateway at a staging catalog/schema with a proper S3 managed location and run it again, it should get past this step.

Hope this helps!

Re: How do you test data quality in a Delta Lake pipeline?

rajkumar8k — Sun, 15 Nov 2026 14:10:05 GMT

In my experience a combination works best:

DLT / Lakeflow expectations check data quality while the pipeline runs. Use warn, drop or fail depending on how serious the issue is, and send bad records to a quarantine table instead of losing them.
PySpark unit tests (pytest) check your transformation logic in CI before you deploy.

For your specific points:

Schema changes: Use Auto Loader with schema evolution and the rescued data column, so unexpected fields get flagged instead of breaking the pipeline.
Duplicates: Deduplicate on business keys and use MERGE so reruns don’t create duplicates.
Nulls: Use expectations on key columns.
Incremental validation: Check row counts, data freshness and expectation metrics for each batch, using the pipeline event log.
Source vs target: Add a reconciliation step that compares row counts and key totals after each load.

Re: Missing spark structured streaming metadata blobs?

Yougasundar — Sun, 15 Nov 2026 04:34:37 GMT

Hi @DB1To3,

Thanks for accepting the answer. On the docs you asked for: the Structured Streaming checkpoints page says "Each query must have a different checkpoint location. Multiple queries should never share the same location", and the two error messages are in the Databricks error reference (search for CONCURRENT_QUERY and CONCURRENT_STREAM_LOG_UPDATE). I have not seen a page that explains how they are detected, so your test is a good idea: start the same availableNow query twice, on two compute resources, with the same checkpoint, and you should see one of the two errors. On your runs/submit question: the API reference says that runs submitted with that endpoint "are not saved as a job", so a job's "maximum concurrent runs" setting cannot protect them. The endpoint has an optional idempotency_token, and the same page says "Databricks guarantees that exactly one run is launched with that idempotency token". If your scheduler builds the token from the schedule slot (for example the query name plus the planned start time), a retry or a double submit for the same slot cannot start a second run. It does not stop the next slot from starting while the previous run is still going, so for that the checkpoint errors remain your guard, or you can use jobs/create plus jobs/run-now, which the same page suggests.

Docs: https://docs.databricks.com/api/workspace/jobs/submit

Re: Pushing a notebook with outputs to Github

Yougasundar — Sun, 15 Nov 2026 04:25:47 GMT

Hi @2nonto,

That message comes from the step that creates the .databricks/commit_outputs file, which controls whether the outputs of .ipynb notebooks are committed. Please check these points in order. 1) Committing outputs is off by default. A workspace admin must open Settings > Workspace admin > Development and, in the Repos section, set "Allow repos to export IPYNB output" to "Allow: IPYNB outputs can be toggled on". 2) The notebook must be in IPYNB format and inside a Git folder, because only IPYNB notebooks can carry outputs. You can convert it with File > Notebook format > Jupyter (.ipynb). 3) Then create the file again from the Git dialog with "Create commit_outputs config file", or with File > Commit notebook outputs > Include Outputs. 4) The file is created inside your Git folder, so also check that you can edit that folder. If it still fails after these checks, please share the full error text and tell us if the admin setting is on.

Docs: https://docs.databricks.com/aws/en/notebooks/notebook-format

Re: Is Databricks Losing Its Charm in the Age of AI?

data_pulse — Sat, 14 Nov 2026 22:26:15 GMT

@thekushalvishwa

AI can write the pipeline in few seconds. It might not explain the DBU bill to finance. A timely post, and one I suspect many of us relate to.

My take is that AI has made writing the code cheap, so our value has moved to the parts it can’t see. Databricks Assistant can write a decent MERGE, but it doesn’t know your table is skewed on one customer key, that Auto Loader is picking up late-arriving files, or that the job has to fit a monthly DBU budget. Choosing between liquid clustering and partitioning, reading a query plan to see why AQE didn’t fix a bad join, deciding whether Photon is worth the cost for a given workload, setting up UC permissions and lineage properly, these are judgment calls that depend on context the AI doesn’t have.

On clients arriving with AI-generated solutions and timelines, I treat it as a starting point and ask what the AI didn’t. What happens at 10x volume? What does this cost per run according to the system tables? How do we reprocess when the logic changes? Who owns it when the Workflow fails at 2am? The estimate usually changes once those are on the table.

There’s also a point we don’t talk about enough. AI is a service, and services go down. If a region outage or a data centre incident takes the assistant offline in the middle of a production issue, the pipeline still has to be fixed, and the person who fixes it is the one who understands Spark, Delta Lake and the data flow without help. We tend to recognise the value of something only when it’s unavailable, and I think that applies to engineers as much as to the tools.

As for how I’m approaching it in practice:

Design first, AI second. I sketch the architecture and data flow myself (medallion layers, clustering strategy, incremental vs full load) and then use AI to speed up the build, not to decide the design.
Review AI output like a junior’s pull request. I check the query plan, the shuffle and file sizes, and the cost before anything reaches production.
Keep one manual habit. Every so often I debug a failed job or tune a slow query without the assistant, so the skill stays sharp.
Spend the saved time upstream. The hours AI gives back go into data quality expectations, governance in UC and talking to the business about what they need.

Keen to hear other views. Until then, I’ll be here explaining the DBU bill.

Is Databricks Losing Its Charm in the Age of AI?

thekushalvishwa — Sat, 14 Nov 2026 20:46:34 GMT

I've been working with Databricks for nearly eight years, and lately I've been reflecting on how our role as data engineers is evolving.

One thing I've noticed is that development is increasingly becoming about prompting AI, troubleshooting issues, fixing bugs, and making workflow adjustments rather than actually engineering solutions from the ground up.

Don't get me wrong, AI is a powerful tool, and there's no point in resisting it. But I sometimes feel that by becoming too dependent on AI, we're gradually losing some of our core engineering capabilities and, more importantly, our ability to challenge things independently.

Another concern is that clients are becoming increasingly familiar with AI-generated solutions. They come to us with suggested implementations, estimated development timelines, and even proposed approaches to fixing technical issues. Sometimes, they already have an idea of how long something should take because AI has given them an answer.

So, the real question is: How do we differentiate ourselves as experienced data engineers in this new AI-driven environment?

How do we move beyond simply executing AI-generated suggestions and demonstrate that we understand the underlying architecture, data flows, performance implications, scalability, cost optimisation, security, and long-term maintainability better than a generic AI-generated solution?

Ultimately, I believe our value shouldn't be measured by how quickly we can prompt AI or fix a bug. It should be measured by our ability to question assumptions, identify problems that aren't immediately obvious, make sound engineering decisions, and deliver solutions that are technically superior to what a client might initially propose.

I'd be interested in hearing how other experienced data engineers and architects are approaching this shift.
@thekushalvishwa

#Spar#Delta Lake#Workflows

Pushing a notebook with outputs to Github

2nonto — Sat, 14 Nov 2026 07:57:02 GMT

I am experiencing an error when trying to commit output and Databricks notebook to GitHub. The error message states, "Error creating or updating commit outputs config file."

Could you please assist me in resolving this issue?

Re: How do you test data quality in a Delta Lake pipeline?

balajij8 — Sat, 14 Nov 2026 07:22:08 GMT

The right pattern is to treat validation as multi layered instead of a single gate. You can start with handling schema changes and schema evolution at ingestion. You can configure Auto Loader with a dedicated schema at the ingestion point to prevent type mismatch challenges. It can route un parsable data to a rescued data columncan fail the stream when a new column appears so you can evolve the schema consciously. On the Delta table, enabling schema merging allows new columns to be added automatically without a table rewrite. For changes like column renames, drops and type widening, enabling column mapping and type widening on the table lets those updates go through without triggering full rewrites.

Handling duplicate records and null or missing values requires separating storage level guarantees from transformation logic. You can handle deduplication in the transformation logic - by using a merge operation with a key predicate to upsert on a last write win basis or by applying a row number window function partitioned by the key and ordered by timestamp to retain the latest record prior to writing. You can pair the deduplication step with a pipeline expectation - such as asserting that an order ID is not null under a unique keys rule to track violations in the event log while routing dropped duplicates to a quarantine table for replay. For null and missing values, You can enforce rules across two distinct layers using not null and check constraints. Inside the pipeline, expectations with explicit failure policies give granular control - use an expect or drop policy to discard malformed rows like missing values while still logging the violation count or an expect or fail policy for critical fields that must stop execution immediately. Use the warn policy strictly for metrics where you actively check the trend and otherwise default to dropping bad records into a quarantine table so nothing is lost.

For incremental data validation and cross-table consistency, your can shift toward observability and reconciliation. Enabling Change Data Feed in the table properties of upstream Delta tables allows downstream pipelines to read a queryable change log of inserts, updates, and deletes instead of re scanning the entire table on every run. You can then validate each batch by asserting expectations directly against the incoming change feed - checking for non-null keys, valid date ranges and expected event types before you merge records into the target. In streaming workloads - Auto Loader checkpoints combined with the rescued data column provide per micro batch visibility into rejected records and tracking expectation pass rates over time in the pipeline event log helps you catch data drift early. To isolate silent drift between source and target tables after each pipeline update, you can run an reconciliation step that compares row counts, sum or count aggregations on key numeric columns or a 64 bit hash across key columns on both sides and it executes faster than a row by row comparison. For upsert cases using merge operations, leveraging a when not matched by source clause with a conditional predicate lets detect and handle target side records that no longer exist in the source without a full rewrite. Embedding these reconciliation checks as downstream SQL assertion job tasks ensures the workflow fails automatically if counts diverge beyond defined tolerance while preserving rejected rows for investigation.

You can use PySpark tests, pipeline expectations and table constraints as a combination in real world projects because each mechanism protects a different moment in the lifecycle. Expectations within Lakeflow Declarative Pipelines handle inline, row level validation during processing using warn, drop, and fail policies while centralizing quality metrics in the event log. Delta not null and check constraints serve as the backstop at write time, preventing corrupt data from landing in the table even if the primary pipeline is bypassed by manual inserts or external jobs. PySpark assertion jobs cover post write validation and continuous integration testing - verifying that filtering for null IDs returns zero rows, comparing dataframe column sets for schema drift and running count assertions that fail the job on error. Every dataset capable of receiving bad data should define at least one expectation and you should deliberately choose between warn, drop and fail policies for each individual constraint.

Re: SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading

ifishbein — Sat, 14 Nov 2026 02:31:38 GMT

@Louis_Frolio I submitted a ticket to support. Apparently without a paid support contract they won't address the issue even though it appears to be a sudden breaking changing in platform behavior vs user error.

Re: Lakebase Data API

suranga — Fri, 13 Nov 2026 21:33:24 GMT

Hi @Jack17,

One small addition to the service principal advice above, because it makes testing a lot easier. Since 12 November, OAuth on-behalf-of for service principals is generally available, so you can test the Data API locally as the same service principal your job will run as, without creating an OAuth secret for it:

databricks auth login --host https://your-workspace-url
# in the browser, pick the service principal from the dropdown and authorize
databricks auth token --host https://your-workspace-url

You need the Service Principal Manager role on that service principal (you get it automatically if you created it). Use the token as the Bearer token for your test POSTs. If the inserts work locally with it, you know the Postgres role and grants for the service principal are right before you schedule anything, and the scheduled job then runs as that same service principal.

Cheers,
Suranga

Re: How do you test data quality in a Delta Lake pipeline?

suranga — Fri, 13 Nov 2026 21:32:15 GMT

Hi @AlexMorgan,

Mateus covered the checks really well, so I'll just add how I'd answer the PySpark tests vs expectations part, because the two do different jobs:

Tests check your code. Unit-test the transformation logic in CI on small, hand-made DataFrames. pyspark.testing has assertDataFrameEqual and assertSchemaEqual, so a test is just "this input should give this output", including the awkward rows: nulls, duplicates, late updates.
Expectations check your data. They run on every batch inside the pipeline, so they catch what tests never will, like a source that suddenly starts sending nulls or duplicates.

CONSTRAINT valid_id EXPECT (id IS NOT NULL) ON VIOLATION DROP ROW

In Python it's @DP.expect_or_drop or @DP.expect_or_fail. My rule of thumb: warn on soft rules, drop or quarantine on row-level problems, and fail when it means the whole batch is wrong.

For schema changes, if you ingest files with Auto Loader, cloudFiles.schemaEvolutionMode decides what happens. addNewColumns (the default) stops the stream so it can restart with the new columns, rescue keeps the schema and puts anything unexpected in _rescued_data, and failOnNewColumns simply fails. An expectation that _rescued_data is null is a cheap way to catch type drift early.

One newer option worth a look: Genie ZeroOps went into Beta on 12 November. It monitors jobs, pipelines and tables in the background, flags failures and data quality issues, and proposes a fix. It's only just out, so I'd treat it as a complement to the checks above rather than a replacement.

Cheers,
Suranga

Re: Issue with CDC ingestion pipeline

suranga — Fri, 13 Nov 2026 21:13:21 GMT

Hi @MichaelAnckaert,

The interesting part is the path in the error: s3:/ with nothing after it. That isn't coming from MySQL. It looks like the pipeline tried to resolve a storage location in Unity Catalog and got back an empty S3 path, and since the destination tables were already created by then, my first suspect is the staging location the connector writes to.

Three things I'd check:

1. Where the staging catalog and schema actually store data.

DESCRIBE CATALOG EXTENDED <staging_catalog>;
DESCRIBE SCHEMA EXTENDED <staging_catalog>.<staging_schema>;

Look at the storage root or location in the output. If neither the catalog nor the schema has one, and the metastore has no root storage either, there's nowhere for the staging volume to live.

2. Default storage. If the staging catalog was created on default storage (the option you get in a serverless workspace), the docs say it can only be accessed from serverless compute. The MySQL ingestion gateway runs on classic job compute, so it can't write there. Pointing staging at a catalog or schema with a managed location on your own S3 bucket (through an external location) gets around that.

3. Privileges on the staging schema. You need USE SCHEMA, CREATE TABLE and CREATE VOLUME there, and the troubleshooting page also notes that the staging catalog can't be a foreign catalog.

If you'd rather stay fully serverless, the newer integrated CDC pipeline for MySQL can run on serverless compute, as long as serverless can reach your MySQL instance over the network.

If none of that matches, the pipeline's event log should have the full error with a stack trace. Post it here and I'm happy to take another look.

Cheers,
Suranga

Re: SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading

ifishbein — Fri, 13 Nov 2026 19:39:26 GMT

Thanks Louis! I'll reach out to support to see if they can pinpoint a more specific root cause for this becoming a problem yesterday and will refactor our codebase.

Re: Lakebase Data API

Louis_Frolio — Fri, 13 Nov 2026 19:16:29 GMT

Hello Jack17, I took a look at both internal and external documentation and here is what I found.

Divya_Pathak's flow (pull with requests, flatten, write to Lakebase, schedule it as a job) is the right skeleton, and DoTA's cautions are the ones that bite: the OAuth token expires after an hour, there's no documented payload or rate limit on Data API POSTs, and a service principal is the identity to run under. One question decides the rest of the design, and the post doesn't say: where will the code that pulls from the external API run?

If it runs outside Databricks (a small service, a serverless function, or the source system pushing to you), the Data API fits. It's PostgREST-compatible, so POST a JSON array to /public/<table> and you get one row per element in a single request. Add the header Prefer: resolution=merge-duplicates (optionally with on_conflict=<key_column>) and re-pulling overlapping data becomes insert-or-update instead of a duplicate-key error. The API URL is on the API tab in the Lakebase App; the token comes from the OAuth flow on the authentication page, not from that tab. Setup gotchas from the docs: it needs an Autoscaling project; don't use the project owner identity, because authenticator can't assume it; create the role with SELECT databricks_create_role(...) in the SQL editor rather than the Roles & Databases UI, or GRANT "<identity>" TO authenticator fails with SQLSTATE 42501; and click Refresh schema cache after you create or alter the table.

If it runs as a Databricks notebook or job, skip the HTTP hop and use a Postgres driver. Mint the same token with the SDK, connect with psycopg over SSL, and use executemany or COPY. If the JSON is nested, store the raw payload in a jsonb column and flatten it with SQL later:

import psycopg, requests
from databricks.sdk import WorkspaceClient
from psycopg.types.json import Jsonb

records = requests.get("https://api.example.com/items", timeout=60).json()
cred = WorkspaceClient().postgres.generate_database_credential(
    endpoint="projects/<project>/branches/<branch>/endpoints/<endpoint>")

with psycopg.connect(host="<host>", dbname="databricks_postgres",
                     user="<identity>", password=cred.token, sslmode="require") as conn:
    conn.cursor().executemany(
        """INSERT INTO public.api_data (id, payload) VALUES (%s, %s)
           ON CONFLICT (id) DO UPDATE SET payload = EXCLUDED.payload""",
        [(r["id"], Jsonb(r)) for r in records])

Generate a fresh credential each time you open a connection, keep the service principal secret in a Databricks secret scope, and give that principal INSERT on the target table.

If the JSON also belongs in the lakehouse, there's a third route: write it to a Delta table first, then create a synced table from Unity Catalog into your Lakebase project. The Postgres copy is read-only, which is fine if the app only reads it, and you keep the Delta history.

Short version: Data API when the writer lives outside Databricks, a Postgres driver when it lives inside, synced tables when the data needs to exist in both places anyway.

References:

Lakebase Data API: https://docs.databricks.com/aws/en/oltp/projects/data-api
Lakebase authentication and token rotation: https://docs.databricks.com/aws/en/oltp/projects/authentication
Connect an external app with the SDK (service principal plus psycopg): https://docs.databricks.com/aws/en/oltp/projects/external-apps-connect
Synced tables for Autoscaling projects: https://docs.databricks.com/aws/en/oltp/projects/sync-tables
PostgREST API reference (bulk insert, upsert, Prefer headers): https://postgrest.org/en/stable/references/api.html

Regards,
Louis.

Re: SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading

Louis_Frolio — Fri, 13 Nov 2026 18:54:12 GMT

Hello @ifishbein, I dug in and here is what I found.

The serverless behavior is explainable, and your workaround is the right one. Serverless runs on Spark Connect, and in the Spark Connect flavor of PySpark SparkSession.getActiveSession() is backed by a plain Python threading.local(). That slot gets filled in the thread that created the session (your notebook's main thread) and nowhere else. A ThreadPoolExecutor worker starts with an empty slot, so getActiveSession() returns None and the first .read on it fails with the exact 'NoneType' object has no attribute 'read' you're seeing. SparkSession.builder.getOrCreate() checks the thread-local first and then falls back to the process-wide default session (your notebook's spark), which is why swapping the call fixed it. Classic compute reaches the session by a different route, which likely explains the different symptom there.

What I can't explain is why it worked before roughly 3PM EST on 11/12. I don't know what changed in the serverless environment that day and I'd rather not guess. Please open a support ticket with a few failing run IDs and timestamps for both the classic and serverless jobs; the classic failures clearing on their own around 6PM is a useful data point for them. Also check the history on status.databricks.com for your region that afternoon.

On the code side, I'd steer away from getActiveSession() inside worker threads entirely, on either compute type. The most portable fix is to pass the session in from the main thread:

from concurrent.futures import ThreadPoolExecutor

def test(foo, spark):
    print(spark)
    # spark.read.... goes here

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = {executor.submit(test, foo, spark): foo for foo in ["A", "B", "C", "D"]}

One SparkSession is safe to share across driver-side threads like this. If you'd rather look the session up inside the thread, SparkSession.active() (PySpark 3.5 and later) returns the active session if there is one and otherwise the default, which is the fallback you actually want. getOrCreate() does the same thing in practice.

One caution if you go looking at pyspark.InheritableThread or inheritable_thread_target as a fix. They were built for pinned thread mode on classic compute, where each Python thread gets its own JVM thread and Spark local properties don't carry over. On Spark Connect they require you to pass the session explicitly, and they propagate the client's thread-locals such as tags, not the active session, so getActiveSession() in the child thread still comes back None. Pass spark in instead.

Takeaway: treat the session as something you hand to a thread, not something a thread goes looking for. That works the same on classic and serverless and survives whatever happened on Wednesday.

References:

getActiveSession(): https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/getActiveSession
SparkSession.active(): https://docs.databricks.com/aws/en/pyspark/reference/classes/sparksession/active
Spark Connect session source (see _active_session and getActiveSession😞 https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py
pyspark.InheritableThread: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.InheritableThread.html
pyspark.inheritable_thread_target: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.inheritable_thread_target.html
Databricks Connect behavior in notebooks and jobs: https://docs.databricks.com/aws/en/dev-tools/databricks-connect/notebooks
Platform status: https://status.databricks.com

Regards,
Louis

Re: How to Connect Databricks to Snowflake & Query Live Data?

Louis_Frolio — Fri, 13 Nov 2026 17:54:49 GMT

@raj , if you are happy with the answer and it addresses your issue please mark "Accept as Solution" so that others can have confidence in the response. Cheers, Lou.

Re: Serverless declarative pipeline ingestion from ADLS is running x4 SLOWER randomly

Louis_Frolio — Fri, 13 Nov 2026 17:53:04 GMT

@saurabh18cs , That is classic compute. Serverless does not allow the configuration of max workers.

SparkSession.getActiveSession() is Suddenly Broken in Python Multi-threading

ifishbein — Fri, 13 Nov 2026 15:45:09 GMT

Hello everyone,

Around 3PM EST yesterday (2026-11-12), dozens of Lakeflow jobs started failing on both classic and serverless compute with distinct symptoms

1. On classic compute (DBR 15.4), reading a file from S3 would result in a PATH NOT FOUND error

2. On serverless compute (v2 and v5), reading a file from S3 would result in a "'NoneType' has not attribute 'read'" error indicating the SparkSession object was None/null

The issue appears to be resolved on classic compute as of 6PM EST 2026-11-12. The issue is ongoing on serverless compute. This is the simplest bit of code that demonstrates the issue.

from concurrent.futures import ThreadPoolExecutor from pyspark.sql import SparkSession def test(foo): print(SparkSession.getActiveSession()) with ThreadPoolExecutor(max_workers=4) as executor: future_to_batch = { executor.submit(test, foo): foo for foo in [“A”, “B”, “C”, “D”] }

Instead of a SparkSession object appearing in cell output, we see "None". Replacing SparkSession.getActiveSession() with SparkSession.builder.getOrCreate() alleviates the problem.

Is anyone else seeing this?

Issue with CDC ingestion pipeline

MichaelAnckaert — Fri, 13 Nov 2026 15:18:05 GMT

Hello everyone

I'm trying to setup an ingestion pipeline from a mysql server and keep running into an issue.

The setup was validated and the pipeline created. When I run the pipeline the destination tables are created in the schema, but then the following error is displayed:

Error class: INVALID_PARAMETER_VALUE
Message: Input URI is not a valid UC Path: s3:/

The pipeline stops after this without further information.

I would appreciate some information or pointers on how I can resolve this issue.