- 1424 Views
- 1 replies
- 0 kudos
Is there a way to permanenetly purge data in Databricks based on certain condition ?
Is there a way to permanenetly purge data in Databricks based on certain condition ?Like, from a particular Databricks table, I want to permanently purge certain rows based on a specific condition e.g., WHERE <col1>="Val1" and <col2>="Val2"
- 1424 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @soumend7115! I will assume you are talking about managed tables in Unity Catalog here, if thats not the case, let me know. We can segregate this in two steps: You can use a DELETE FROM SQL statement to remove rows that match your condition. For e...
- 0 kudos
- 5814 Views
- 2 replies
- 0 kudos
Upload file to DBFS fails with error code 0
When trying to upload to DBFS from local machine getting error as "Error occurred when processing file ... : Server responded with 0 code" DBR 7.3 LTSSpark 3.0.1 Scala 2.12 Uploading the file using the "upload" in the Databricks cloud console, the c...
- 5814 Views
- 2 replies
- 0 kudos
- 0 kudos
@PramodNaikThe error you are facing not because of file size but because the file you are trying to upload contains PII or SPII data words like dob, Token, accesskey, password..,etc.Solution: Rename such data like date_of_birth, token_no, access_key...
- 0 kudos
- 10097 Views
- 6 replies
- 8 kudos
Connect Timeout - Error when trying to run a cell
Hello everybody.Whenever I am trying to run a simple cell I receive the following error message now:Notebook detached. Exception when creating expectation context: java.net.SocketTimeoutException: Connect Timeout.After that error message the cluster ...
- 10097 Views
- 6 replies
- 8 kudos
- 8 kudos
@HelloDatabricks The error you are facing is due to not detaching the old cluster, which is terminated/auto-terminated and when you run a cluster that connected to new cluster with same name and config. Which notebook tries to connect to old cluster ...
- 8 kudos
- 7995 Views
- 3 replies
- 0 kudos
UnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same
My Code:-- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT Key_ID, Distributor_ID, Customer_ID, Customer_Name, ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source table...
- 7995 Views
- 3 replies
- 0 kudos
- 0 kudos
@Prashant777 In your scenario at update section, you are trying to update primary keys aswell, which Delta Table can't differentiate when you re-run the same batch/file and throws error as all duplicates, to run without error/fail, remove (Target.Dis...
- 0 kudos
- 5672 Views
- 4 replies
- 2 kudos
Error in SQL Warehouse: User is not part of org
I tried to start the Databricks SQL Warehouse cluster today, but received the following error message:Clusters are failing to launch. Cluster launch will be retired Request to create a cluster failed with an exception: PERMISSION_DENIED: User xxxx is...
- 5672 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi All,What you need to do is set a new owner. You can do this by clicking on permissions, then the setup icon, and choosing Assign new owner.It works for me.Thanks
- 2 kudos
- 8553 Views
- 10 replies
- 4 kudos
ModuleNotFoundError when using foreachBatch on runtime 14 with Unity
Recently we have run into an issue using foreachBatch after upgrading our Databricks cluster on Azure to a runtime version 14 with Spark 3.5 with Shared access mode and Unity catalogue.The issue was manifested by ModuleNotFoundError error being throw...
- 8553 Views
- 10 replies
- 4 kudos
- 4 kudos
I am having the same issue using serverless compute. I think the issue comes from this documentation limitations https://docs.databricks.com/aws/en/structured-streaming/foreach#behavior-changes-for-foreachbatch-in-databricks-runtime-140
- 4 kudos
- 1988 Views
- 2 replies
- 0 kudos
Notebooks owner shows 'Unknown'
Hi All,We are using CI/CD to deploy notebooks from GitHub and authenticating via Azure Service Principals (SPNs). Until last week, the notebook owner was correctly displayed as the SPN. However, over the past few days, the owner is now shown as "Unkn...
- 1988 Views
- 2 replies
- 0 kudos
- 0 kudos
Same here, but indeed it does not seem to have any impact at all.So I guess something changed in the databricks backend as suggested before.
- 0 kudos
- 4269 Views
- 3 replies
- 2 kudos
Resolved! Job run failing to import modules
I have several notebooks that run code to ingest data from various APIs into our Data Warehouse. I have several modules that I reuse in multiple notebooks, things like redshift functions, string cleaning functions and json cleaning functions. Out of ...
- 4269 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @marcio_oliveira Thanks for sharing the error and the context — this intermittent module import issue in Databricks Serverless jobs is a known behavior in some environments,and here’s what’s likely going wrong :Root Cause:A race condition or cold-...
- 2 kudos
- 1714 Views
- 1 replies
- 0 kudos
Need advice for a big source table DLT Pipeline
Hi all,I was hoping to get advice from someone with DLT Pipelines, I want to apologize in advance if this is a noob question, I'm really new into DLT, materialized views and streaming tablesI have the following scenario, my source is a big sales delt...
- 1714 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MauricioS Absolutely not a noob question — you're touching on a common and important challenge in DLT pipelines,especially when dealing with large shared Delta tables and incremental ingestion from Unity Catalog sources.Let’s break it down so it’...
- 0 kudos
- 9745 Views
- 6 replies
- 0 kudos
Error in SQL statement: UnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same
My code:- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT Key_ID, Distributor_ID, Customer_ID, Customer_Name, ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source tableM...
- 9745 Views
- 6 replies
- 0 kudos
- 0 kudos
This error occurs; when we try to update all the cells of target_data without a single updated record in source_data(updates_data) , to resolve this issue add a update_time column with unix timestamp (or) make changes in at least one cell of streamin...
- 0 kudos
- 2677 Views
- 2 replies
- 2 kudos
Resolved! CDF metadata columns are lost after importing dlt
Hi Databricks Community, I attempted to read the Change Feed from a CDF-enabled table. Initially, the correct table schema, including the metadata columns (_change_type, _commit_version, and _commit_timestamp), was returned as expected. However, afte...
- 2677 Views
- 2 replies
- 2 kudos
- 2 kudos
The issue stems from the interaction between the Change Data Feed (CDF) metadata columns (_change_type, _commit_version, _commit_timestamp) and the Delta Live Tables (DLT) library. After you import the dlt module, the behavior of reading the CDF-enab...
- 2 kudos
- 5481 Views
- 4 replies
- 6 kudos
Resolved! Serverless Compute no support for Caching data frames
Can anyone please tell me why df.cache() and df.persist() are not supported in Serevrless compute?Many Thanks
- 5481 Views
- 4 replies
- 6 kudos
- 6 kudos
What I do wish was possible was for serverless to warn that caching is not supported, but not error on a call. It makes switching between compute (serverless & all purpose) brittle and prevents code from easily being interoperable, no matter the comp...
- 6 kudos
- 3227 Views
- 1 replies
- 0 kudos
Cannot Create Table under catalog.schema
AnalysisException: [RequestId=75cd00bc-7274-48c5-bdb2-c86a05de227f ErrorClass=TABLE_DOES_NOT_EXIST.RESOURCE_DOES_NOT_EXIST] Table '643a51ba-70c9-41ac-b75d-9c0f9039e7c1' does not exist. I am getting this issue while creating the table under the catalo...
- 3227 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Kumaril_02,How are you doing today?, As per my understanding, It looks like the error you're getting—saying the table with a long ID doesn’t exist—is probably happening because Databricks is trying to reference a table using its internal ID rather...
- 0 kudos
- 1575 Views
- 2 replies
- 0 kudos
If/else task branches
Hi,I have an If/else task, say A and two other tasks B and C.For the false outcome I would like to execute task B. For the true branch I would like to execute task C followed with task B. What is the correct way to express the dependencies of B on th...
- 1575 Views
- 2 replies
- 0 kudos
- 0 kudos
For sure one solution is to package everything in a separate job. Other options?
- 0 kudos
- 2502 Views
- 2 replies
- 0 kudos
Serverless Cluster Issue
Hi,While using Serverless cluster I'm not able to access dbfs files, saying I don't have permission to the file.But while accessing them using All-purpose cluster I'm able to access them.Why am I facing this issue?
- 2502 Views
- 2 replies
- 0 kudos
- 0 kudos
You might be encountering this issue due to permission differences between the serverless and all-purpose clusters. Serverless environments often have restricted access for enhanced security. Make sure the appropriate IAM roles or access controls are...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
