Databricks

oleole · 03-28-2023

Coming from MS SQL background, I'm trying to write a query in Spark SQL that simply update a column value of table A (source table) by INNER JOINing a new table B with a filter.MS SQL query looks like this:UPDATE T SET T.OfferAmount = OSE.EndpointEve...

oleole · 03-26-2023

I'm using spark version 3.2.1 on databricks (DBR 10.4 LTS), and I'm trying to convert sql server sql query to a new sql query that runs on a spark cluster using spark sql in sql syntax. However, spark sql does not seem to support XML PATH as a functi...

oleole · 03-19-2023

I have a daily job run that occasionally fails with the error: The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached. After I get the notification that this job failed on schedule, I manually run ...

oleole · 04-17-2023

According to this documentation, you can specify the wait time between the "start" of the first run and the retry start time.

oleole · 03-30-2023

Posting the solution that I ended up using:%sql DROP TABLE if exists UserCountry; CREATE TABLE if not exists UserCountry ( UserID INT, Country VARCHAR(5000) ); INSERT INTO UserCountry SELECT L.UserID AS UserID, CONCAT_WS(',', co...

oleole · 03-28-2023

Posting answer to my question: MERGE into TempOffer VIEW USING OfferSeq OSE ON VIEW.OfferId = OSE.OfferID AND OSE.OfferId = 1 WHEN MATCHED THEN UPDATE set VIEW.OfferAmount = OSE.EndpointEventAmountValue;

oleole · 03-27-2023

I think that's what you meant - it worked for me! Thanks for sharing your knowledge @Hubert Dudek

oleole · 03-27-2023

@Hubert Dudek Hello, I'm having the same issue with using UPDATE in spark sql and came across your answer. When you say "replace source_table_reference with view" in MERGE, do you mean to replace "P" with "VIEW" that looks something as below:%sql ME...

Databricks

User Stats

User Activity

MERGE to update a column of a table using Spark SQL

Using "FOR XML PATH" in Spark SQL in sql syntax

How to delay a new job run after job

Re: How to delay a new job run after job

Re: Using "FOR XML PATH" in Spark SQL in sql syntax

Re: MERGE to update a column of a table using Spark SQL

Re: Does Spark SQL can perform UPDATE with INNER JOIN and LIKE with '%' + [column] + '%' ?

Re: Does Spark SQL can perform UPDATE with INNER JOIN and LIKE with '%' + [column] + '%' ?