topic Re: How many records does Spark use to infer the schema? entire file or just the first "X" number of records? in Data Engineering

How many records does Spark use to infer the schema? entire file or just the first "X" number of records?

User15787040559 — Tue, 22 Jun 2021 23:09:52 GMT

It depends.

If you specify the schema it will be zero, otherwise it will do a full file scan which doesn’t work well processing Big Data at a large scale.

CSV files Dataframe Reader https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameReader.csv.html?highlight=dataframereader#pyspark-sql-dataframereader-csv samplingRatio will let you change how you sample the data on inference.

Re: How many records does Spark use to infer the schema? entire file or just the first "X" number of records?

aladda — Wed, 23 Jun 2021 04:09:15 GMT

As indicated there are ways to manage the amount of data being sampled for inferring schema. However as a best practice for production workloads its always best to define the schema explicitly for consistency, repeatability and robustness of the pipelines. It also helps with implementing effective data quality checks using features like schema enforcement and expectations in Delta Live Tables