cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

carlosst01
by New Contributor II
  • 1555 Views
  • 0 replies
  • 0 kudos

java.lang.SecurityException: Could not verify permissions for DeleteFromTable ERROR IN EXTERNAL SYNAPSE TABLE

Hi guys, I am trying to delete an external table in Databricks that is connected to a Synapse table (via the com.databricks.spark.sqldw connector) but got the error shown below:Apart from that, when I try to execute an DELETE and INSERT actions (like...

WhatsApp Image 2023-05-25 at 12.02.52 PM WhatsApp Image 2023-05-25 at 12.08.16 PM WhatsApp Image 2023-05-25 at 12.13.14 PM
  • 1555 Views
  • 0 replies
  • 0 kudos
frank7
by New Contributor II
  • 2564 Views
  • 2 replies
  • 1 kudos

Resolved! Is it possible to write a pyspark dataframe to a custom log table in Log Analytics workspace?

I have a pyspark dataframe that contains information about the tables that I have on sql database (creation date, number of rows, etc)Sample data: { "Day":"2023-04-28", "Environment":"dev", "DatabaseName":"default", "TableName":"discount"...

  • 2564 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Bruno Simoes​ :Yes, it is possible to write a PySpark DataFrame to a custom log table in Log Analytics workspace using the Azure Log Analytics Workspace API.Here's a high-level overview of the steps you can follow:Create an Azure Log Analytics Works...

  • 1 kudos
1 More Replies
Ajay-Pandey
by Esteemed Contributor III
  • 1651 Views
  • 2 replies
  • 2 kudos

Why Azure Databricks needs to store data in temp storage in Azure before writing to the synapse.

I was following the tutorial about data transformation with azure databricks, and it says before loading data into azure synapse analytics, the data transformed by azure databricks would be saved on temp storage in azure blob storage first before loa...

  • 1651 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Ajay Pandey​ Saving the transformed data to temporary storage in Azure Blob Storage before loading into Azure Synapse Analytics provides a number of benefits to ensure that the data is accurate, optimized, and performs well in the target environmen...

  • 2 kudos
1 More Replies
RohitKulkarni
by Contributor II
  • 1906 Views
  • 3 replies
  • 4 kudos

TCP/IP Issue

Hello Team,I am trying to read the data from Synapse analytics from databricks.The below is the Query to read the table.%python### Read from Azure Synapse table via spark.read.load df = spark.read \ .format("com.databricks.spark.sqldw") \ .option("ur...

  • 1906 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 4 kudos
2 More Replies
TMNGB
by New Contributor II
  • 1395 Views
  • 0 replies
  • 2 kudos

Databricks to Azure Synapse SQL Server: error converting between Spark and Parquet column types

When writing data from Pyspark to Azure SQL Server (official databricks tutorial here) I am getting an error in the conversion between Spark and Parquet types.I believe this is caused in the temporary storage location mandatory when writing data from...

  • 1395 Views
  • 0 replies
  • 2 kudos
Akshith_Rajesh
by New Contributor III
  • 4158 Views
  • 4 replies
  • 6 kudos

Unable to write Data frame to Azure Synapse Table

When I am trying to insert records into the azure synapse Table using JDBC Its throwing below error com.microsoft.sqlserver.jdbc.SQLServerException: The statement failed. Column 'COMPANY_ADDRESS_STATE' has a data type that cannot participate ...

  • 4158 Views
  • 4 replies
  • 6 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 6 kudos

Hi @Rajesh Akshith​, We haven't heard from you on the last response from @Hubert Dudek​ , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to othe...

  • 6 kudos
3 More Replies
Labels