https://community.databricks.com/t5/data-engineering/autoloader-issue/m-p/53300#M29771 <P>I'm trying to ingest data from Parquet files using Autoloader. Now, I have my custom schema, I don't want to infer the schema from the parquet files.</P><P>During readstream everything is fine. But during writestream, it is somehow inferring the schema from the files and I'm getting a schema mismatch error.</P><P>Any idea why it is happening? Help will be appreciated.</P><P>#</P> Tue, 21 Nov 2023 13:57:42 GMT The_Demigorgan 2023-11-21T13:57:42Z https://community.databricks.com/t5/data-engineering/autoloader-issue/m-p/53300#M29771 <P>I'm trying to ingest data from Parquet files using Autoloader. Now, I have my custom schema, I don't want to infer the schema from the parquet files.</P><P>During readstream everything is fine. But during writestream, it is somehow inferring the schema from the files and I'm getting a schema mismatch error.</P><P>Any idea why it is happening? Help will be appreciated.</P><P>#</P> Tue, 21 Nov 2023 13:57:42 GMT https://community.databricks.com/t5/data-engineering/autoloader-issue/m-p/53300#M29771 The_Demigorgan 2023-11-21T13:57:42Z https://community.databricks.com/t5/data-engineering/autoloader-issue/m-p/101115#M40548 <P>In this case, please make sure you specify the schema explicitly when reading the Parquet files and do not specify any inference options.</P> <P>Something like</P> <LI-CODE lang="markup">spark.readStream.format("cloudFiles").schema(schema)...</LI-CODE> <P>If you want to more easily grab the schema, you can read with the batch reader and capture the schema:</P> <LI-CODE lang="markup">schema = spark.read.parquet("/your/path/here").schema</LI-CODE> Thu, 05 Dec 2024 17:22:43 GMT https://community.databricks.com/t5/data-engineering/autoloader-issue/m-p/101115#M40548 cgrant 2024-12-05T17:22:43Z