@Ken Pendergast​  To setup Databricks with auto loader, please follow the below document.

https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.html

Fetching data from Glacier is not supported. however, you can try one of the following configurations in the cluster and it might help.

• Setting ignoreMissingFiles configuration on the notebook before the stream read?

spark.conf.set("spark.sql.files.ignoreMissingFiles", "true")

We should be able to ignore archived files with this.

OR

• Set badRecordsPath option on readStream (We would recommend setting the first one)

val df = spark.readStream
.option("badRecordsPath", "/tmp/badRecordsPath")
.format("cloudFiles")
.
.
.load()