https://community.databricks.com/t5/data-engineering/unable-to-infer-schema-for-parquet-at/m-p/27316#M19193 <P>seems like an invalid parquet file. my guess is the incoming data has mixed types (for the same column) or a different/invalid structure.</P> Mon, 06 Sep 2021 06:30:19 GMT -werners- 2021-09-06T06:30:19Z https://community.databricks.com/t5/data-engineering/unable-to-infer-schema-for-parquet-at/m-p/27314#M19191 <P></P> <P> I have this code in a notebook:</P>val streamingDataFrame = incomingStream.selectExpr("cast (body as string) AS Content") .withColumn("Sentiment", toSentiment($"Content")) <P></P> <P>import org.apache.spark.sql.streaming.Trigger.ProcessingTime val result = streamingDataFrame .writeStream.format("parquet") .option("path", "/mnt/TwitterSentiment") .option("checkpointLocation", "/mnt/temp/check") .start() &lt;/p&gt;&lt;p&gt;...and it always results in this error. Am stumped, any advice?&lt;/p&gt;&lt;pre&gt;org.apache.spark.sql.AnalysisException: Unable to infer schema for Parquet at . It must be specified manually;&lt;br&gt; </P> <P></P> Wed, 29 Jan 2020 18:43:02 GMT https://community.databricks.com/t5/data-engineering/unable-to-infer-schema-for-parquet-at/m-p/27314#M19191 bciampa 2020-01-29T18:43:02Z https://community.databricks.com/t5/data-engineering/unable-to-infer-schema-for-parquet-at/m-p/27316#M19193 <P>seems like an invalid parquet file. my guess is the incoming data has mixed types (for the same column) or a different/invalid structure.</P> Mon, 06 Sep 2021 06:30:19 GMT https://community.databricks.com/t5/data-engineering/unable-to-infer-schema-for-parquet-at/m-p/27316#M19193 -werners- 2021-09-06T06:30:19Z