topic Re: Serverless Capabilities Not Available In My Workspace in Data Engineering

Serverless Capabilities Not Available In My Workspace

jduran9987 — Mon, 21 Sep 2026 02:48:07 GMT

Hello,

My AWS Databricks account is paid Premium and has serverless compute enabled but the only types I see for SQL warehouses are Pro and Classic.

I created two workspaces each with “Use serverless compute with default storage,” and "Use your existing cloud account" but SQL Warehouses still only show Pro and Classic.

Creating a serverless SQL warehouse through the CLI returns:

"Workspace <workspace-id> is no longer eligible for Serverless Compute. Please reach out to your administrator."

I've followed the prerequisite checklist according to the docs - Premium account, serverless terms accepted, no s3 access policies, no external legacy hive metastore, AWS region us-east-1, and I'm using the admin user.

Please advise.

Best,

Jonathan Duran

Re: Serverless Capabilities Not Available In My Workspace

Satyasai — Mon, 21 Sep 2026 04:48:42 GMT

Hi @jduran9987 ,

have you tried these steps

Log into Databricks Account Console (accounts.cloud.databricks.com) as an Account Admin (not just Workspace Admin)

Go to Workspaces > Select your Workspace ID > Feature Enablement

Check if Serverless Compute or Serverless SQL Warehouses is toggled ON. If it is OFF, please toggle it ON and wait 10 minutes.

Check Settings > Security at the Account level to make sure that there is not a Compliance Security Profile (PCI-DSS/HIPAA) enabled on the account or workspace.

Re: Serverless Capabilities Not Available In My Workspace

masonreed11 — Mon, 21 Sep 2026 05:39:16 GMT

It looks like a workspace-level Serverless eligibility issue rather than a missing prerequisite. Since the CLI specifically says the workspace is no longer eligible, I’d check the Serverless Compute status in the Databricks account console. If all prerequisites are met, Databricks Support may need to verify or re-enable the workspace’s backend eligibility, as creating a new workspace doesn’t necessarily guarantee Serverless SQL access.

Re: Serverless Capabilities Not Available In My Workspace

data_pulse — Mon, 21 Sep 2026 09:08:11 GMT

One thing worth checking is whether the workspace meets all the documented Serverless SQL requirements. Databricks lists these under the Serverless enablement prerequisites, including Premium or above.

Also, if the account has a granted Serverless postponement, new workspaces inherit that status. So, creating another workspace would not necessarily change the behaviour as documented.

Re: Serverless Capabilities Not Available In My Workspace

Ashwin_DSA — Mon, 21 Sep 2026 10:17:58 GMT

Hi @jduran9987,

The "no longer eligible for Serverless Compute" error from the CLI, combined with only Pro and Classic showing in the UI, points to an account-level enablement issue rather than anything wrong with your workspace configuration.

On AWS, serverless is enabled by default in most accounts, but accounts created through certain channels (such as the AWS Marketplace free trial path) may end up in a state where serverless is never enabled at the account level, even after upgrading to a paid Premium plan. The Set up serverless SQL warehouses docs note that serverless requires the account to not be on a free trial, and that accounts with a "granted postponement" inherit that restriction across all workspaces. Without account-level enablement, no workspace under that account can offer the Serverless warehouse type, which is exactly the behaviour you are seeing.

Unfortunately, there's no self-service fix for this. You'll need to open a support ticket (or reach out to your Databricks account team) with your account ID and workspace IDs, and mention the CLI error message you received. The backend team can enable the feature for your account directly, and the resolution is typically quick once it reaches the right team.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Serverless Capabilities Not Available In My Workspace

aayush_410 — Mon, 21 Sep 2026 13:22:45 GMT

The core issue with your prototype is that the file discovery step is being duplicated: Auto Loader already does incremental, stateful file discovery internally (and can use cloud-native file notifications instead of directory listing), but your dbutils.fs.ls loop is a separate manual full listing that doesn't scale and has to be re-run to find new objecttypes. The fix is to stop doing objecttype discovery before the stream and instead do it inside a single stream.

Recommended pattern: one Auto Loader stream landing everything, fan-out in foreachBatch

One Auto Loader stream reads the whole directory, not per-objecttype:
python
raw = (spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.useNotifications", "true") # avoid repeated full listing as volume grows
.option("cloudFiles.schemaLocation", schema_loc)
.option("header", "true")
.schema(generic_string_schema) # or land everything as strings — see below
.load(source_path)
)

Since each objecttype has a different real schema, don't try to get Auto Loader to infer one unified schema across all of them — that's what was pushing you toward per-objecttype streams in the first place. Instead, either:

Land every column as string (schema inference will do this by default for CSV if you don't force it otherwise) plus _rescued_data, or
Land the whole row as a single VARIANT column — Databricks explicitly supports this pattern for exactly this kind of "many shapes landing in one place" ingestion, and it's schema-agnostic by construction.
Extract objecttype from the filename inside the stream, using the built-in _metadata column rather than a separate listing pass:
python
from pyspark.sql.functions import regexp_extract, col

raw = raw.withColumn(
"objecttype",
regexp_extract(col("_metadata.file_name"), r"^([^_]+)_", 1)
)

This means new objecttypes are picked up automatically the moment their files land — no code change, no stream restart, no separate scan.

Fan out to per-objecttype tables in foreachBatch, dynamically, based on whatever objecttypes actually appear in each micro-batch:
python
def route_batch(batch_df, batch_id):
for obj_type in [r.objecttype for r in batch_df.select("objecttype").distinct().collect()]:
subset = batch_df.filter(col("objecttype") == obj_type)
target_table = f"bronze.{obj_type}"
(subset.write.format("delta")
.mode("append")
.option("mergeSchema", "true")
.saveAsTable(target_table)) # creates the table on first appearance

raw.writeStream.foreachBatch(route_batch).option("checkpointLocation", checkpoint_loc).start()

Why this scales where your prototype doesn't:

One stream, one checkpoint — no per-objecttype glob patterns to maintain or restart as new types appear.
No separate full-directory listing to enumerate objecttypes — that work is now done incrementally by Auto Loader's own file-tracking state, and cheaply, per micro-batch, only on files that already got pulled in.
New objecttypes just work — the first time a new <objecttype>_*.csv.gz lands, foreachBatch sees a new distinct value and creates the table, no code deploy required.

One thing to decide up front: if downstream consumers need real typed columns (not everything as strings/VARIANT), you'll want a lightweight per-objecttype schema registry (even just a small control Delta table: objecttype -> expected_schema_json) that the foreachBatch function looks up to cast columns properly before writing — otherwise you're pushing the "what's the real schema" problem one layer downstream instead of solving it. That's a reasonable v2; landing as strings/VARIANT + rescue column first, then adding typed casting once you've stabilized the objecttype list, keeps the initial rollout simple.

Re: Serverless Capabilities Not Available In My Workspace

jduran9987 — Mon, 21 Sep 2026 13:35:13 GMT

Ashwin,

Thanks for your response. I tried opening a ticket but was told to try community support, as I don't have an active support plan.

What do you suggest I do in this situation? Is there a support portal for individual-use, pay-as-you-go accounts like myself?

Re: Serverless Capabilities Not Available In My Workspace

Ashwin_DSA — Mon, 21 Sep 2026 13:53:37 GMT

Hi @jduran9987,

Pay-as-you-go accounts don't include a formal support plan, so the ticketing portal isn't available to you. The community forum is one of the channels Databricks points PAYGO users to, so you are already in the right place.

That said, for account-level enablement issues like this one, the community forum can only go so far since the fix requires a backend change on Databricks' side. Here are a couple of options worth trying:

Email help@databricks.com directly. The Databricks getting started documentation mentions this as a channel for live help. Include your account ID, workspace ID(s), and the CLI error message. Explain that serverless compute is not available and that this appears to be an account-level enablement issue requiring backend intervention.

Contact your AWS Marketplace Databricks listing if you subscribed through AWS Marketplace. Some enablement issues for Marketplace-originated accounts can be escalated through that channel.

Hope this helps.