Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I have a dataframe that inexplicably takes forever to write to an MS SQL Server even though other dataframes, even much larger ones, write nearly instantly. I'm using this code:my_dataframe.write.format("jdbc")
.option("url",sqlsUrl)
.optio...
Had a similar issue. I can do 1-4 million rows in 1 minute via SSIS ETL on SQL server. Table is 15 fields long. Looking at your code it seems you have many fields but nothing like 300-400 fields which can affect performance. You can check SQL Server ...
We a facing a situation and I would like to understand from the Databricks side what is the best practice regarding that. Question: Is it possible to have a cluster with a fixed Global IP on Databricks?DetailsWe have a vendor that has a SQL Server da...
@Arnold Souza If you file a support to Azure support they can help customize the Vnet by unlocking it as the Azure Databricks resources are deployed in a managed resource group. Your plan B also should be the way to go if option 1 does not work as e...
I have a dms task that processing the full-load and replication ongoing tasksfrom source (MSSQL) to target (AWS S3)then use delta lake to handle the CDC logsI've a notebook that would insert data into mssql continuously (with id as primary key)then d...
When applying the following T-SQL I don't get any errors on MS SQL ServerSELECT DISTINCT
*
FROM dbo.account
LEFT OUTER JOIN dbo.crm2cburl_lookup
ON account.Id = CRM2CBURL_Lookup.[Key]
LEFT OUTER JOIN dbo.organizations
ON CRM2CBURL_Lookup.CB_UR...
![204293406-01bf6cc1-bb6f-42bb-9bfe-e9b1f5135ae9[1]](/t5/image/serverpage/image-id/1103i5E2E130C9E462F86/image-size/medium?v=v2&px=400)
