DLT Pipeline, Autoloader, Streaming Query Exception: Could not find ADLS Gen2 Token

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

I have set up autoloader to form a streaming table in my DLT pipeline,

import dlt

@dlt.table

def streamFiles_new():

return (

spark.readStream.format("cloudFiles")

.option("cloudFiles.format", "json")

.option("cloudFiles.inferColumnTypes", "true")

.option("multiLine", "true")

.load("file_location")

)

When I run this cell in the notebook it goes through and infers the schema of the Delta Live Table. However, when I run the DLT pipeline that includes the notebook with this autoloader cell I get an error: "org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED]...terminated with exception: Could not find ADLS Gen2 Token"
I know we have ADLS Gen2 storage and in my DLT pipeline settings I have Compute set to "DLT Compute AAD Passthrough". I know the AAD pass through through is working and that the storage location is mounted because the DLT pipeline worked and was reading files from this location for materialized views. It just broke when I tried to instead implement autoloader for a streaming view. I do not have permissions under my Azure account to generate access keys for storage accounts.