cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

oleole
by Contributor
  • 6741 Views
  • 1 replies
  • 1 kudos

Resolved! MERGE to update a column of a table using Spark SQL

Coming from MS SQL background, I'm trying to write a query in Spark SQL that simply update a column value of table A (source table) by INNER JOINing a new table B with a filter.MS SQL query looks like this:UPDATE T SET T.OfferAmount = OSE.EndpointEve...

  • 6741 Views
  • 1 replies
  • 1 kudos
Latest Reply
oleole
Contributor
  • 1 kudos

Posting answer to my question:   MERGE into TempOffer VIEW USING OfferSeq OSE ON VIEW.OfferId = OSE.OfferID AND OSE.OfferId = 1 WHEN MATCHED THEN UPDATE set VIEW.OfferAmount = OSE.EndpointEventAmountValue;

  • 1 kudos
JJL
by New Contributor II
  • 8643 Views
  • 3 replies
  • 3 kudos

Resolved! Does Spark SQL can perform UPDATE with INNER JOIN and LIKE with '%' + [column] + '%' ?

Hi All,I came from MS SQL and just started to learning more about Spark SQLHere is one part that I'm trying to perform. In MS SQL, it can be easily done, but it seems like it doesn't in SparkSo, I want to make a simple update to the record, if the co...

  • 8643 Views
  • 3 replies
  • 3 kudos
Latest Reply
oleole
Contributor
  • 3 kudos

@Hubert Dudek​ Hello, I'm having the same issue with using UPDATE in spark sql and came across your answer. When you say "replace source_table_reference with view" in MERGE, do you mean to replace "P" with "VIEW" that looks something as below:%sql ME...

  • 3 kudos
2 More Replies
Databricks_-Dat
by New Contributor II
  • 1497 Views
  • 2 replies
  • 4 kudos

what is the supported mssql connector for Databricks runtime 11.3LTS Scala 2.12 Spark 3.3.0?

We were using mssql connector -com.microsoft.azure:spark-mssql-connector_2.12_3.0:1.0.0-alpha with 10.3LTS DBR. As we need to upgrade to higher version of DBR to make use of new functions like unpivot/melt in the notebooks. -com.microsoft.azure:spark...

  • 1497 Views
  • 2 replies
  • 4 kudos
Latest Reply
ranged_coop
Valued Contributor II
  • 4 kudos

Is the spark 3.3 series even supported by the connector yet ?As per the [github link](https://github.com/microsoft/sql-spark-connector#current-releases) - assuming this is the library you are trying to use ?The latest Spark 2.4.x compatible connector...

  • 4 kudos
1 More Replies
jonathan-dufaul
by Valued Contributor
  • 1525 Views
  • 4 replies
  • 5 kudos

Why is writing to MSSQL Server 12.0 so slow directly from spark but nearly instant when I write to a csv and read it back

I have a dataframe that inexplicably takes forever to write to an MS SQL Server even though other dataframes, even much larger ones, write nearly instantly. I'm using this code:my_dataframe.write.format("jdbc") .option("url",sqlsUrl) .optio...

  • 1525 Views
  • 4 replies
  • 5 kudos
Latest Reply
yueyue_tang
New Contributor II
  • 5 kudos

I meet the same problem and I don't know how to write dataFrame to MS sql server quickly​

  • 5 kudos
3 More Replies
Labels