cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Autoloader schema inference

Cosimo_F_
Contributor

Hello,

is it possible to turn off schema inference with AutoLoader?

Thank you,

Cosimo

3 REPLIES 3

Thank you for your reply!

The documentation mentions passing a schema to AutoLoader but does not explain how. The solution is simply to use the .schema method like so:

spark.\

 readStream.\

 schema(<schema>).\

 format("cloudFiles").\

load()

Best,

Cosimo.

Do you mean? .option("cloudFiles.schemaLocation", "<path_to_checkpoint>")

If thats the case, then you can check the following docs https://docs.databricks.com/ingestion/auto-loader/options.html

Hi Jose,

No, that's the location of the schema hints (which work together with schema inference). Specifying a schema location does not turn off schema inference as I wanted. In fact schemaLocation is a required option _unless_ the schema is passed explicitly as I showed.

Best,

Cosimo,

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group