How many records does Spark use to infer the schema? entire file or just the first "X" number of records?

User15787040559
Databricks Employee
Databricks Employee

‎06-22-2021 04:09 PM

It depends.

If you specify the schema it will be zero, otherwise it will do a full file scan which doesn’t work well processing Big Data at a large scale.

CSV files Dataframe Reader https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameReader.csv.html?h... samplingRatio will let you change how you sample the data on inference.

Labels:
0 Kudos

aladda
Databricks Employee
Databricks Employee

‎06-22-2021 09:09 PM

As indicated there are ways to manage the amount of data being sampled for inferring schema. However as a best practice for production workloads its always best to define the schema explicitly for consistency, repeatability and robustness of the pipelines. It also helps with implementing effective data quality checks using features like schema enforcement and expectations in Delta Live Tables

0 Kudos