topic Re: Autoloader schema inference in Data Engineering

Autoloader schema inference

Cosimo_F_ — Tue, 06 Sep 2022 20:32:54 GMT

Hello,

is it possible to turn off schema inference with AutoLoader?

Thank you,

Cosimo

Re: Autoloader schema inference

Cosimo_F_ — Thu, 08 Sep 2022 18:35:53 GMT

Thank you for your reply!

The documentation mentions passing a schema to AutoLoader but does not explain how. The solution is simply to use the .schema method like so:

spark.\

readStream.\

schema(<schema>).\

format("cloudFiles").\

load()

Best,

Cosimo.

Re: Autoloader schema inference

jose_gonzalez — Fri, 09 Sep 2022 23:04:30 GMT

Do you mean? .option("cloudFiles.schemaLocation", "<path_to_checkpoint>")

If thats the case, then you can check the following docs https://docs.databricks.com/ingestion/auto-loader/options.html

Re: Autoloader schema inference

Cosimo_F_ — Sat, 10 Sep 2022 14:01:55 GMT

Hi Jose,

No, that's the location of the schema hints (which work together with schema inference). Specifying a schema location does not turn off schema inference as I wanted. In fact schemaLocation is a required option _unless_ the schema is passed explicitly as I showed.

Best,

Cosimo,

Re: Autoloader schema inference

shivagarg — Tue, 03 Dec 2024 15:26:31 GMT

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#language-python

you can enforce the schema or use the "cloudFiles.schemaHints" to override the Inference.

df = spark.readStream.format("cloudFiles") \
  .option("cloudFiles.format", "csv") \
  .option("header", "true") \
  .option("rescuedDataColumn", "_rescued_data") \ # makes sure that you don't lose data
  .schema(<schema>) \ # provide a schema here for the files
  .load(<path>