[SQL_CONF_NOT_FOUND] The SQL config "/Volumes/xxx...." canot be found. Please verify that the confi

DataGeek_JT — Mon, 25 Mar 2024 15:04:25 GMT

I am getting the below error when trying to stream data from Azure Storage path to a Delta Live Table ([PATH] is the path to my files which I have redacted here):

[SQL_CONF_NOT_FOUND] The SQL config "/Volumes/[PATH]" cannot be found. Please verify that the config exists. SQLSTATE: 42K0I

I can read data from the volume in Unity Catalogue using spark.read.csv("/Volumes/[PATH]") but it seems to think there is an issue doing it this way.

The notebook code is:

import dlt

from pyspark.sql import functions as F

data_source_path = spark.conf.get("/Volumes/[PATH]")

@dlt.table

def ticks():

return (spark.readStream

.format("cscloudFilesv")

.option("cloudFiles.format", "csv")

.option("header", True)

.load(data_source_path )

.select("*", F.col("_metadata.file_path").alias("source_file_name"), F.split(F.col("_metadata.file_path"), "/")[8].alias("Symbol"))

)

Any idea what is wrong please?

Re: [SQL_CONF_NOT_FOUND] The SQL config "/Volumes/xxx...." canot be found. Please verify t

NandiniN — Thu, 09 Jan 2025 11:52:33 GMT

I believe you are not setting

spark.conf.set("/Volumes/[PATH]", "your_actual_path_here")

hence when you try to get the conf, it fails.

data_source_path = spark.conf.get("/Volumes/[PATH]")

"/Volumes/[PATH]" becomes the conf name, you would not want to name it with / and [] rather like read_path.

topic [SQL_CONF_NOT_FOUND] The SQL config "/Volumes/xxx...." canot be found. Please verify that the confi in Data Engineering

[SQL_CONF_NOT_FOUND] The SQL config "/Volumes/xxx...." canot be found. Please verify that the confi

Re: [SQL_CONF_NOT_FOUND] The SQL config "/Volumes/xxx...." canot be found. Please verify t