spark conf for serveless jobs

seefoods — Wed, 17 Dec 2025 15:39:19 GMT

Hello Guys,

I use serveless on databricks Azure, so i have build a decorator which instanciate a SparkSession. My job use autolaoder / kafka using mode availableNow. Someone Knows which spark conf is required beacause i want to add it ?

Thanx

import logging

from databricks.sdk import WorkspaceClient

from pyspark.sql import SparkSession
from pyspark.sql import DataFrame
from pyspark.sql.functions import col, udf

from pydantic import ValidationError

from typing import Callable
from functools import wraps




def get_spark_session():
    """

    :return:
    """
    return SparkSession.builder.getOrCreate()


def provide_spark_session(function):
    """"
    :param function:
    :return: """

    def wrapper(*args, **kwargs):
        return function(spark=get_spark_session(), *args, **kwargs)

    return wrapper


def get_workspace_session():
    """
    :return: WorkspaceClient instance
    """
    return WorkspaceClient()


def provide_workspace_session(function):
    """
    Decorator to provide a Databricks WorkspaceClient session
    :param function: Function to decorate
    :return: Decorated function
    """

    @wraps(function)
    def wrapper(*args, **kwargs):
        return function(workspace_session=get_workspace_session(), *args, **kwargs)

    return wrapper

Re: spark conf for serveless jobs

Saritha_S — Sat, 20 Dec 2025 15:31:25 GMT

Hi @seefoods

Please find below my findings for your case.

You don’t need (and can’t meaningfully add) any Spark conf to enable availableNow on Databricks Serverless.

Let me explain clearly, and then show what is safe to do in your decorator.

availableNow is not a Spark conf

For Auto Loader and Kafka, availableNow is a Structured Streaming trigger, not a Spark configuration.Correct usage (this is the only place it belongs):

(spark.readStream.format("cloudFiles") # or "kafka"

.option("cloudFiles.format", "json")

.option("cloudFiles.schemaLocation", schema_path)

.load(input_path)

.writeStream

.trigger(availableNow=True)

.option("checkpointLocation", checkpoint)

.start(output_path)

)

There is no spark.conf.set(...) required (or available) for availableNow.

Why Spark conf won’t work on Serverless

On Databricks Serverless :

SparkSession is already created
Executor / driver / streaming engine configs are locked
SparkSession.builder.getOrCreate():

does not create a new session
silently ignores most configs

So adding something like:

SparkSession.builder \

.config("spark.sql.streaming.availableNow", "true") # does not exist

.getOrCreate()

topic Re: spark conf for serveless jobs in Data Engineering

spark conf for serveless jobs

Re: spark conf for serveless jobs