cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

12833 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

578 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

510 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

1057 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

743 Posts

Activity in Databricks Platform Discussions

ifra_inam
by > • New Contributor II
  • 501 Views
  • 4 replies
  • 2 kudos

Databricks Free Edition: AWS S3 Unity Catalog Storage Credential Fails AssumeRole Validation

I'm using Databricks Free Edition and trying to configure an AWS S3 external location through Unity Catalog.My storage credential validation consistently fails at Assume Role, while Self Assume Role and External ID Condition are skipped.Error: Failed...

  • 501 Views
  • 4 replies
  • 2 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 2 kudos

The error means Databricks' cross-account sts:AssumeRole call is being denied by AWS before any of the later checks run, which is why Self Assume and External ID show as skipped. Everything you've verified is about the IAM role's configuration, so t...

  • 2 kudos
3 More Replies
MikeGo
by > • Valued Contributor
  • 395 Views
  • 5 replies
  • 2 kudos

Zerobus to lakebase

Hi,Is there a plan to make lakebase as destination of Zerobus?Thanks.

  • 395 Views
  • 5 replies
  • 2 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 2 kudos

I can't speak to roadmap commitments in a public forum, and as far as I know Lakebase isn't a direct Zerobus Ingest target today. Your account team can tell you whether it's planned and under what NDA terms. What works today Zerobus Ingest is design...

  • 2 kudos
4 More Replies
Ignacio_Soler
by > • New Contributor
  • 371 Views
  • 3 replies
  • 2 kudos

Private network gateway (ALL_TRAFFIC): serverless can't reach on-prem by raw IPv4, only by hostname

Hi all,We're testing the private network gateway (Beta) on Azure Databricks so that serverless compute egresses through a subnet in our VNet and reaches on-prem resources (through a hub firewall + ExpressRoute/VPN) from a predictable IP range.Setup- ...

  • 371 Views
  • 3 replies
  • 2 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 2 kudos

Short version: your reading matches how I'd expect this to behave. The gateway path is driven by name resolution, so IP literals fall outside it. Treat that as a Beta limitation to confirm with your Databricks account team, not as a guarantee you ca...

  • 2 kudos
2 More Replies
vamsi_simbus
by > • Databricks Partner
  • 195 Views
  • 2 replies
  • 0 kudos

Anyone using Kinaxis Data Publish (Delta Sharing) from Databricks Marketplace?

Hi all,We're looking at Kinaxis Data Publish on the Databricks Marketplace. It shares Kinaxis Maestro planning data into Databricks through Delta Sharing. We requested access a couple of weeks ago and haven't heard back, so I'd like to hear from anyo...

  • 195 Views
  • 2 replies
  • 0 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 0 kudos

Short version: Kinaxis decides what goes into the share and who gets approved, and from the Databricks side the share is read-only. I can't give you first-hand Kinaxis-specific details, but the platform mechanics answer most of your questions. Data ...

  • 0 kudos
1 More Replies
Pravieen
by > • New Contributor
  • 295 Views
  • 3 replies
  • 0 kudos

Request for Guidance: Enabling PostgreSQL CDC in Azure Databricks Lakeflow Connect

Hello Databricks Community,I am using Azure Databricks Lakeflow Connect with a PostgreSQL source.The CDC option is disabled in the PostgreSQL ingestion wizard, and I see the following message:This mode is not available in this workspace for the selec...

  • 295 Views
  • 3 replies
  • 0 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 0 kudos

The message means that CDC mode for the PostgreSQL connector isn't turned on for your workspace. It isn't a problem with your connection or source configuration. Since you already have a ticket open, that's the right path, but there are a couple of ...

  • 0 kudos
2 More Replies
krishgarikipati
by > • Databricks Partner
  • 286 Views
  • 3 replies
  • 0 kudos

Delta Tables: Is there any possibility on Delta tables Data file formats other than Parquet

Hi Team,Since Parquet files are immutable and Delta tables framework writes data changes into new data files, Is there any option to specify any other file format and try to use same data file for minor data changes.(Below 10 Record transaction chang...

  • 286 Views
  • 3 replies
  • 0 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 0 kudos

Short answer: No, you can't swap Parquet for another file format in Delta. Delta Lake is Parquet data files plus a file-based transaction log [1]. What you're really after, avoiding a rewrite of huge files for a handful of changed rows, is handled b...

  • 0 kudos
2 More Replies
jumelik
by > • New Contributor
  • 162 Views
  • 2 replies
  • 1 kudos

Hourly streaming table refresh jobs keep running after the table was dropped (Free Edition)

I’m using Databricks Free Edition. I created the streaming table retail_analytics.sales_data.customers_latest with SCHEDULE EVERY 1 HOUR and later dropped the table. Two refresh jobs with the same name still run hourly and send failure emails.Job 516...

  • 162 Views
  • 2 replies
  • 1 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 1 kudos

The hourly runs come from orphaned scheduler jobs. Dropping the streaming table should have removed them, but it didn't. That's why DROP SCHEDULE fails: it's an ALTER on the table, and the table no longer exists. The fix has to go through the Jobs l...

  • 1 kudos
1 More Replies
YuriS
by > • New Contributor III
  • 233 Views
  • 4 replies
  • 0 kudos

DEEP CLONE fails with DELTA_CLONE_INCOMPLETE_FILE_COPY

Hi, Would appreciate some support on the DELTA_CLONE_INCOMPLETE_FILE_COPY issue with DEEP CLONE.Exact error is as follows: An error occurred while calling o459.sql.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 144 in stage...

  • 233 Views
  • 4 replies
  • 0 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 0 kudos

Short answer: I wouldn't set spark.databricks.delta.clone.checkWrite to false. That check is a safety net. DEEP CLONE compares the size of each file it wrote against the size recorded in the source table's transaction log. Expected 146660 and found ...

  • 0 kudos
3 More Replies
Coffee77
by > • Honored Contributor III
  • 284 Views
  • 5 replies
  • 0 kudos

External Access and iFrame Embedding of Dashboards with Private Link and Public Net Access Disabled

Hi Databricks Community,I am looking for recommendations and real-world experiences on how to provide external users with browser-based access to Azure Databricks AI/BI Dashboards, including embedding dashboards in external applications, while keepin...

  • 284 Views
  • 5 replies
  • 0 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 0 kudos

Short answer: I wouldn't build this with APIM, Entra Application Proxy, or a clientless ZTNA tool in front of the workspace UI. Proxying the Databricks UI or its dashboard embeds under a different public hostname is not a supported architecture. Wit...

  • 0 kudos
4 More Replies
2nonto
by > • New Contributor
  • 323 Views
  • 4 replies
  • 1 kudos

Pushing a notebook with outputs to Github

I am experiencing an error when trying to commit output and Databricks notebook to GitHub. The error message states, "Error creating or updating commit outputs config file."Could you please assist me in resolving this issue?

Screenshot 2026-11-11 214121.png
  • 323 Views
  • 4 replies
  • 1 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 1 kudos

This error comes from the step before anything is sent to GitHub. Databricks is trying to create or update the commit_outputs config file inside the Git folder's working tree and failing. That points to the workspace setting, your permissions on the...

  • 1 kudos
3 More Replies
Innovation1
by > • New Contributor II
  • 474 Views
  • 3 replies
  • 1 kudos

Serverless outbound internet still restricted after upgrading express setup trial to Premium

Hi all,I started a Databricks trial with express setup on AWS and upgraded to Premium pay-as-you-go on Oct 19, 2026. Serverless compute in my workspace can only reach a short list of domains. All other public sites are refused at DNS.Test run on serv...

  • 474 Views
  • 3 replies
  • 1 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 1 kudos

Short answer: No, the restricted egress is a trial-only guardrail and it shouldn't persist on a paid Premium workspace. What you're seeing is most likely the trial restriction not being cleared on your workspace after the upgrade. The DNS-level REFU...

  • 1 kudos
2 More Replies
julien_inquipus
by > • New Contributor
  • 329 Views
  • 3 replies
  • 2 kudos

Serverless egress blocked (DNS REFUSED) on a Premium AWS workspace created via Express – only pypi/g

Hello,We run our platform on Databricks on AWS (eu-central-1). The workspace was created through the Express sign-up in early November 2026, then upgraded to Premium, and a credit card has been added to the account (pay-as-you-go). Compute is serverl...

  • 329 Views
  • 3 replies
  • 2 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 2 kudos

Short answer: what you're describing doesn't look like a network policy problem. default-policy says FULL_ACCESS, but the traffic is being refused at DNS (rcode 5) and at the IP layer, so something outside the policy you can see is restricting the w...

  • 2 kudos
2 More Replies
itsme_gk
by > • New Contributor
  • 219 Views
  • 2 replies
  • 0 kudos

Free edition - serverless option stopped working

I am on free edition and suddently serverless option stopped working.  Is there any restriction on free edition or something is down today?

  • 219 Views
  • 2 replies
  • 0 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 0 kudos

Most likely you've hit the Free Edition daily usage quota. A platform outage is the second possibility, and it's easy to rule out. What's probably happening Free Edition runs only on serverless compute, and it comes with a fair-usage quota. If you u...

  • 0 kudos
1 More Replies
User10
by > • New Contributor II
  • 267 Views
  • 4 replies
  • 4 kudos

Resolved! Preserving Salesforce connector SCD2 history across Full Refreshes

We use the managed Lakeflow Connect Salesforce connector with SCD2 history tracking enabled in Bronze, with Silver materialized views serving downstream consumers.Salesforce schema changes are propagated across environments at different times. Removi...

  • 267 Views
  • 4 replies
  • 4 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 4 kudos

Short answer: Treat the Bronze SCD2 table as disposable, because a full refresh on a managed ingestion pipeline will always wipe it. Salesforce can't replay history, so the SCD2 history can only come from a copy you keep yourself. The pattern that w...

  • 4 kudos
3 More Replies
kzahiri
by > • New Contributor
  • 125 Views
  • 2 replies
  • 0 kudos

Can Databricks temporarily raise the GCP workspace creation limit?

Hi Databricks team, I’m setting up Databricks on Google Cloud for a university course. Fifteen of the 18 project workspaces are running, and three student teams are waiting for their workspaces so they can start their projects today.The latest creati...

  • 125 Views
  • 2 replies
  • 0 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 0 kudos

I can't raise the limit from the forum. Workspace-creation quotas are enforced on Databricks' side, and only internal staff can change them. Community members, myself included, have no way to approve that. For today, the fastest way to unblock the t...

  • 0 kudos
1 More Replies