The default location of temporary file in Azure Synapse Connector(com.databricks.spark.sqldw)

Chengcheng — Fri, 30 Jun 2023 06:48:50 GMT

Hi everone, I'm trying to query data in Azure Synapse Dedicated SQL Pool according to the documentaion using:

.format("com.databricks.spark.sqldw")

It says that a abfss temporary location is needed.

But I found that even I don't specify the tempDir, the following code also works in DBR above the version 13.0.

I'd like to know whether there has a documentation of this driver/connector, and where it save temporary file when I don't specify one.

df = (spark.read .format("com.databricks.spark.sqldw") .option("url", url) # .option("tempDir", "abfss://tempdir@datalakefordatabricks555.dfs.core.windows.net/") # .option("forwardSparkAzureStorageCredentials", "true") .option("user", user) .option("password", password) .option("encrypt", "true") .option("trustServerCertificate", "false") .option("loginTimeout", "30") .option("query", pushdown_query) .option("fetchsize", 2000) .load() )

Should I worry about the following issue if I don't specify any tempDir? (which I don't want to do so if not necessary.)

"The Azure Synapse connector does not delete the temporary files that it creates in the Azure storage container. Databricks recommends that you periodically delete temporary files under the user-supplied tempDir location."

Temporary data management

Re: The default location of temporary file in Azure Synapse Connector(com.databricks.spark.sqldw)

lucasrocha — Wed, 05 Jun 2024 22:54:14 GMT

Hello @Chengcheng , I hope this message finds you well.

As per the documentation the "tempDir" parameter is a required one and there is no default value for it.
Databricks Synapse connector options reference: https://docs.databricks.com/en/connect/external-systems/synapse-analytics.html#databricks-synapse-connector-options-reference
Could you please share some logs of any of these runs where it was possible to be executed without specifying the "tempDir" parameter?
Maybe we can find some insights from there.

Best regards,
Lucas Rocha

topic Re: The default location of temporary file in Azure Synapse Connector(com.databricks.spark.sqldw) in Data Engineering

The default location of temporary file in Azure Synapse Connector(com.databricks.spark.sqldw)

Re: The default location of temporary file in Azure Synapse Connector(com.databricks.spark.sqldw)