Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the co...
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the co...
Engage in vibrant discussions covering diverse learning topics within the Databricks Community. Expl...
How do you debug OOM on long-running streaming jobs?Hi All,One of my Structured Streaming jobs has started dying with OutOfMemoryError. Runs fine for about approx 40 hours, then falls over, restarts, and does the same thing again. I had introduced fe...
Short version: your own charts say this isn't a Java heap leak. The live heap after full GC has no rising floor, and the state store is tiny. What they do show is a container that sits at about 9.5-10 GB of a roughly 10.8 GB limit from the first min...
I'm using Databricks Free Edition and trying to configure an AWS S3 external location through Unity Catalog.My storage credential validation consistently fails at Assume Role, while Self Assume Role and External ID Condition are skipped.Error: Failed...
The error means Databricks' cross-account sts:AssumeRole call is being denied by AWS before any of the later checks run, which is why Self Assume and External ID show as skipped. Everything you've verified is about the IAM role's configuration, so t...
Hi,Is there a plan to make lakebase as destination of Zerobus?Thanks.
I can't speak to roadmap commitments in a public forum, and as far as I know Lakebase isn't a direct Zerobus Ingest target today. Your account team can tell you whether it's planned and under what NDA terms. What works today Zerobus Ingest is design...
Hi all,We're testing the private network gateway (Beta) on Azure Databricks so that serverless compute egresses through a subnet in our VNet and reaches on-prem resources (through a hub firewall + ExpressRoute/VPN) from a predictable IP range.Setup- ...
Short version: your reading matches how I'd expect this to behave. The gateway path is driven by name resolution, so IP literals fall outside it. Treat that as a Beta limitation to confirm with your Databricks account team, not as a guarantee you ca...
Hello Databricks Team,I recently created a new Databricks Free Edition account using my Google account and also completed LinkedIn identity verification.I am learning Generative AI and prompt engineering, but I cannot access any foundation models in ...
Short version: system.ai and the old per-endpoint pages aren't where you get chat models in a current workspace. Pay-per-token models are surfaced through AI Gateway → Models, and that's what Playground reads from. Your errors come from opening lega...
Hi all,We're looking at Kinaxis Data Publish on the Databricks Marketplace. It shares Kinaxis Maestro planning data into Databricks through Delta Sharing. We requested access a couple of weeks ago and haven't heard back, so I'd like to hear from anyo...
Short version: Kinaxis decides what goes into the share and who gets approved, and from the Databricks side the share is read-only. I can't give you first-hand Kinaxis-specific details, but the platform mechanics answer most of your questions. Data ...
Hello Databricks Community,I am using Databricks Free Edition and have developed an Academy of Data Marketing Analytics application using React, Node.js, Express and Databricks SQL.My application was deploying successfully earlier. However, since Nov...
Short version: this is failing on the platform side, before your code ever runs, and the OAuth secret you generated isn't the credential the deployer uses. The most likely cause is a broken binding between the app and its service principal. The fast...
Hello Databricks Community,I am using Azure Databricks Lakeflow Connect with a PostgreSQL source.The CDC option is disabled in the PostgreSQL ingestion wizard, and I see the following message:This mode is not available in this workspace for the selec...
The message means that CDC mode for the PostgreSQL connector isn't turned on for your workspace. It isn't a problem with your connection or source configuration. Since you already have a ticket open, that's the right path, but there are a couple of ...
Hi Team,Since Parquet files are immutable and Delta tables framework writes data changes into new data files, Is there any option to specify any other file format and try to use same data file for minor data changes.(Below 10 Record transaction chang...
Short answer: No, you can't swap Parquet for another file format in Delta. Delta Lake is Parquet data files plus a file-based transaction log [1]. What you're really after, avoiding a rewrite of huge files for a handful of changed rows, is handled b...
Hey, I have recently started using Free Edition and since beginning, I am getting this issue.I have tried to run this warehouse but could not resolve it on my own. What can be done ?
Hi @Louis_Frolio , Thanks for the update. As mentioned earlier, As a free edition user, I am not able to delete a warehouse. I can still execute sql queries in notebook. However, there are some tasks which needs SQL Warehouse to run.So, After few ch...
I’m using Databricks Free Edition. I created the streaming table retail_analytics.sales_data.customers_latest with SCHEDULE EVERY 1 HOUR and later dropped the table. Two refresh jobs with the same name still run hourly and send failure emails.Job 516...
The hourly runs come from orphaned scheduler jobs. Dropping the streaming table should have removed them, but it didn't. That's why DROP SCHEDULE fails: it's an ALTER on the table, and the table no longer exists. The fix has to go through the Jobs l...
Hi, Would appreciate some support on the DELTA_CLONE_INCOMPLETE_FILE_COPY issue with DEEP CLONE.Exact error is as follows: An error occurred while calling o459.sql.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 144 in stage...
Short answer: I wouldn't set spark.databricks.delta.clone.checkWrite to false. That check is a safety net. DEEP CLONE compares the size of each file it wrote against the size recorded in the source table's transaction log. Expected 146660 and found ...
Hi Databricks Community,I am looking for recommendations and real-world experiences on how to provide external users with browser-based access to Azure Databricks AI/BI Dashboards, including embedding dashboards in external applications, while keepin...
Short answer: I wouldn't build this with APIM, Entra Application Proxy, or a clientless ZTNA tool in front of the workspace UI. Proxying the Databricks UI or its dashboard embeds under a different public hostname is not a supported architecture. Wit...
I am experiencing an error when trying to commit output and Databricks notebook to GitHub. The error message states, "Error creating or updating commit outputs config file."Could you please assist me in resolving this issue?
This error comes from the step before anything is sent to GitHub. Databricks is trying to create or update the commit_outputs config file inside the Git folder's working tree and failing. That points to the workspace setting, your permissions on the...
Hello,I am getting acquainted with Databricks, of which I have signed up for a free account. I have a Lambda function using Python 3.14 and the databricks-sql-connector library that runs on a schedule, to take data from a file in S3 and put the diffe...
Short version: both errors look like the same problem, the SQL warehouse being in a state where it can't accept the request (starting, stopping, or unavailable), and raise_for_status() is hiding the real message. A retry alone won't fix it. You need...
| User | Count |
|---|---|
| 1900 | |
| 1049 | |
| 931 | |
| 480 | |
| 411 |