[SQL_CONF_NOT_FOUND] The SQL config "/Volumes/xxx...." canot be found. Please verify that the confi
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2024 08:04 AM
I am getting the below error when trying to stream data from Azure Storage path to a Delta Live Table ([PATH] is the path to my files which I have redacted here):
[SQL_CONF_NOT_FOUND] The SQL config "/Volumes/[PATH]" cannot be found. Please verify that the config exists. SQLSTATE: 42K0I
I can read data from the volume in Unity Catalogue using spark.read.csv("/Volumes/[PATH]") but it seems to think there is an issue doing it this way.
The notebook code is:
import dlt
from pyspark.sql import functions as F
data_source_path = spark.conf.get("/Volumes/[PATH]")
@dlt.table
def ticks():
return (spark.readStream
.format("cscloudFilesv")
.option("cloudFiles.format", "csv")
.option("header", True)
.load(data_source_path )
.select("*", F.col("_metadata.file_path").alias("source_file_name"), F.split(F.col("_metadata.file_path"), "/")[8].alias("Symbol"))
)
Any idea what is wrong please?
