- 6540 Views
- 3 replies
- 0 kudos
UnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same
My Code:-- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT Key_ID, Distributor_ID, Customer_ID, Customer_Name, ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source table...
- 6540 Views
- 3 replies
- 0 kudos
- 0 kudos
@Prashant777 In your scenario at update section, you are trying to update primary keys aswell, which Delta Table can't differentiate when you re-run the same batch/file and throws error as all duplicates, to run without error/fail, remove (Target.Dis...
- 0 kudos
- 5255 Views
- 4 replies
- 2 kudos
Error in SQL Warehouse: User is not part of org
I tried to start the Databricks SQL Warehouse cluster today, but received the following error message:Clusters are failing to launch. Cluster launch will be retired Request to create a cluster failed with an exception: PERMISSION_DENIED: User xxxx is...
- 5255 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi All,What you need to do is set a new owner. You can do this by clicking on permissions, then the setup icon, and choosing Assign new owner.It works for me.Thanks
- 2 kudos
- 7586 Views
- 10 replies
- 4 kudos
ModuleNotFoundError when using foreachBatch on runtime 14 with Unity
Recently we have run into an issue using foreachBatch after upgrading our Databricks cluster on Azure to a runtime version 14 with Spark 3.5 with Shared access mode and Unity catalogue.The issue was manifested by ModuleNotFoundError error being throw...
- 7586 Views
- 10 replies
- 4 kudos
- 4 kudos
I am having the same issue using serverless compute. I think the issue comes from this documentation limitations https://docs.databricks.com/aws/en/structured-streaming/foreach#behavior-changes-for-foreachbatch-in-databricks-runtime-140
- 4 kudos
- 1642 Views
- 2 replies
- 0 kudos
Notebooks owner shows 'Unknown'
Hi All,We are using CI/CD to deploy notebooks from GitHub and authenticating via Azure Service Principals (SPNs). Until last week, the notebook owner was correctly displayed as the SPN. However, over the past few days, the owner is now shown as "Unkn...
- 1642 Views
- 2 replies
- 0 kudos
- 0 kudos
Same here, but indeed it does not seem to have any impact at all.So I guess something changed in the databricks backend as suggested before.
- 0 kudos
- 3311 Views
- 3 replies
- 2 kudos
Resolved! Job run failing to import modules
I have several notebooks that run code to ingest data from various APIs into our Data Warehouse. I have several modules that I reuse in multiple notebooks, things like redshift functions, string cleaning functions and json cleaning functions. Out of ...
- 3311 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @marcio_oliveira Thanks for sharing the error and the context — this intermittent module import issue in Databricks Serverless jobs is a known behavior in some environments,and here’s what’s likely going wrong :Root Cause:A race condition or cold-...
- 2 kudos
- 1425 Views
- 1 replies
- 0 kudos
Need advice for a big source table DLT Pipeline
Hi all,I was hoping to get advice from someone with DLT Pipelines, I want to apologize in advance if this is a noob question, I'm really new into DLT, materialized views and streaming tablesI have the following scenario, my source is a big sales delt...
- 1425 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MauricioS Absolutely not a noob question — you're touching on a common and important challenge in DLT pipelines,especially when dealing with large shared Delta tables and incremental ingestion from Unity Catalog sources.Let’s break it down so it’...
- 0 kudos
- 9035 Views
- 6 replies
- 0 kudos
Error in SQL statement: UnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same
My code:- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT Key_ID, Distributor_ID, Customer_ID, Customer_Name, ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source tableM...
- 9035 Views
- 6 replies
- 0 kudos
- 0 kudos
This error occurs; when we try to update all the cells of target_data without a single updated record in source_data(updates_data) , to resolve this issue add a update_time column with unix timestamp (or) make changes in at least one cell of streamin...
- 0 kudos
- 2136 Views
- 2 replies
- 2 kudos
Resolved! CDF metadata columns are lost after importing dlt
Hi Databricks Community, I attempted to read the Change Feed from a CDF-enabled table. Initially, the correct table schema, including the metadata columns (_change_type, _commit_version, and _commit_timestamp), was returned as expected. However, afte...
- 2136 Views
- 2 replies
- 2 kudos
- 2 kudos
The issue stems from the interaction between the Change Data Feed (CDF) metadata columns (_change_type, _commit_version, _commit_timestamp) and the Delta Live Tables (DLT) library. After you import the dlt module, the behavior of reading the CDF-enab...
- 2 kudos
- 4303 Views
- 4 replies
- 6 kudos
Resolved! Serverless Compute no support for Caching data frames
Can anyone please tell me why df.cache() and df.persist() are not supported in Serevrless compute?Many Thanks
- 4303 Views
- 4 replies
- 6 kudos
- 6 kudos
What I do wish was possible was for serverless to warn that caching is not supported, but not error on a call. It makes switching between compute (serverless & all purpose) brittle and prevents code from easily being interoperable, no matter the comp...
- 6 kudos
- 2564 Views
- 1 replies
- 0 kudos
Cannot Create Table under catalog.schema
AnalysisException: [RequestId=75cd00bc-7274-48c5-bdb2-c86a05de227f ErrorClass=TABLE_DOES_NOT_EXIST.RESOURCE_DOES_NOT_EXIST] Table '643a51ba-70c9-41ac-b75d-9c0f9039e7c1' does not exist. I am getting this issue while creating the table under the catalo...
- 2564 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Kumaril_02,How are you doing today?, As per my understanding, It looks like the error you're getting—saying the table with a long ID doesn’t exist—is probably happening because Databricks is trying to reference a table using its internal ID rather...
- 0 kudos
- 1252 Views
- 2 replies
- 0 kudos
If/else task branches
Hi,I have an If/else task, say A and two other tasks B and C.For the false outcome I would like to execute task B. For the true branch I would like to execute task C followed with task B. What is the correct way to express the dependencies of B on th...
- 1252 Views
- 2 replies
- 0 kudos
- 0 kudos
For sure one solution is to package everything in a separate job. Other options?
- 0 kudos
- 2121 Views
- 2 replies
- 0 kudos
Serverless Cluster Issue
Hi,While using Serverless cluster I'm not able to access dbfs files, saying I don't have permission to the file.But while accessing them using All-purpose cluster I'm able to access them.Why am I facing this issue?
- 2121 Views
- 2 replies
- 0 kudos
- 0 kudos
You might be encountering this issue due to permission differences between the serverless and all-purpose clusters. Serverless environments often have restricted access for enhanced security. Make sure the appropriate IAM roles or access controls are...
- 0 kudos
- 1757 Views
- 1 replies
- 0 kudos
RLS and CLS with delta sharing
RLS and CLS is possible to apply in tables that are shared using unity catalog?
- 1757 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Vanessa, RLS and CLS is not directly supported in delta sharing but as an alternative you could add dynamic views to a share to filter rows and columns. Please find the documentation below - https://docs.databricks.com/aws/en/delta-sharing/create-...
- 0 kudos
- 1370 Views
- 1 replies
- 0 kudos
Difference between creating a schema manually vs schema through SQL code externally?
I have created a bronze schema manually using catalog->create schema. I have provided external location. The "details" table look like this:However, when I created silver schema but this time using sql script i.e. %sqlCREATE SCHEMA xyz.silverMANAGED ...
- 1370 Views
- 1 replies
- 0 kudos
- 0 kudos
When a schema is created manually via the Databricks catalog UI, ownership defaults to the user who created the schema, and ownership properties may not be explicitly recorded by default.On the other hand, when creating a schema using SQL commands, D...
- 0 kudos
- 4154 Views
- 1 replies
- 0 kudos
What is the possible cause in calling the dataricks Job permission API
PATCH /api/2.0/permissions/jobs/{job_id} { "error_code": "INVALID_PARAMETER_VALUE", "message": "Owner permissions cannot be modified via an update / PATCH request if the endpoint does not have a valid owner. Please use a set / PUT request ins...
- 4154 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi team,This seems to be an expected behaviour.The PATCH endpoint for updating job permissions is designed for incremental modifications of existing permissions. However, modifying owner permissions is restricted unless a valid owner already exists i...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
