- 4122 Views
- 6 replies
- 6 kudos
I have a dataframe that inexplicably takes forever to write to an MS SQL Server even though other dataframes, even much larger ones, write nearly instantly. I'm using this code:my_dataframe.write.format("jdbc")
.option("url",sqlsUrl)
.optio...
- 4122 Views
- 6 replies
- 6 kudos
Latest Reply
Had a similar issue. I can do 1-4 million rows in 1 minute via SSIS ETL on SQL server. Table is 15 fields long. Looking at your code it seems you have many fields but nothing like 300-400 fields which can affect performance. You can check SQL Server ...
5 More Replies
- 13108 Views
- 1 replies
- 1 kudos
Coming from MS SQL background, I'm trying to write a query in Spark SQL that simply update a column value of table A (source table) by INNER JOINing a new table B with a filter.MS SQL query looks like this:UPDATE T
SET T.OfferAmount = OSE.EndpointEve...
- 13108 Views
- 1 replies
- 1 kudos
Latest Reply
Posting answer to my question:
MERGE into TempOffer VIEW
USING OfferSeq OSE ON VIEW.OfferId = OSE.OfferID AND OSE.OfferId = 1
WHEN MATCHED THEN UPDATE set VIEW.OfferAmount = OSE.EndpointEventAmountValue;
by
JJL
• New Contributor II
- 16543 Views
- 3 replies
- 3 kudos
Hi All,I came from MS SQL and just started to learning more about Spark SQLHere is one part that I'm trying to perform. In MS SQL, it can be easily done, but it seems like it doesn't in SparkSo, I want to make a simple update to the record, if the co...
- 16543 Views
- 3 replies
- 3 kudos
Latest Reply
@Hubert Dudek Hello, I'm having the same issue with using UPDATE in spark sql and came across your answer. When you say "replace source_table_reference with view" in MERGE, do you mean to replace "P" with "VIEW" that looks something as below:%sql
ME...
2 More Replies
- 2660 Views
- 2 replies
- 4 kudos
We were using mssql connector -com.microsoft.azure:spark-mssql-connector_2.12_3.0:1.0.0-alpha with 10.3LTS DBR. As we need to upgrade to higher version of DBR to make use of new functions like unpivot/melt in the notebooks. -com.microsoft.azure:spark...
- 2660 Views
- 2 replies
- 4 kudos
Latest Reply
Is the spark 3.3 series even supported by the connector yet ?As per the [github link](https://github.com/microsoft/sql-spark-connector#current-releases) - assuming this is the library you are trying to use ?The latest Spark 2.4.x compatible connector...
1 More Replies
