Autoloader schema inference

[Cosimo_F_]

Hello,

is it possible to turn off schema inference with AutoLoader?

Thank you,

Cosimo

[Cosimo_F_]

Thank you for your reply!

The documentation mentions passing a schema to AutoLoader but does not explain how. The solution is simply to use the .schema method like so:

spark.\

 readStream.\

 schema(<schema>).\

 format("cloudFiles").\

load()

Best,

Cosimo.

[jose_gonzalez]

Do you mean? .option("cloudFiles.schemaLocation", "<path_to_checkpoint>")

If thats the case, then you can check the following docs https://docs.databricks.com/ingestion/auto-loader/options.html

[Cosimo_F_]

Hi Jose,

No, that's the location of the schema hints (which work together with schema inference). Specifying a schema location does not turn off schema inference as I wanted. In fact schemaLocation is a required option _unless_ the schema is passed explicitly as I showed.

Best,

Cosimo,