shyam_9
Databricks Employee
Databricks Employee

Hi @sani nanda,

Please follow the below steps,

  • Read the schema file as a CSV, setting header to true. This will give an empty dataframe but with the correct header.
  • Extract the column names from that schema file.
column_names = spark.read.option("header",true).csv(schemafile).columns
  • Now read the datafile and change the default column names to the ones in the schema dataframe.
df = spark.read.option("header", "false").option("inferSchema", "true").csv(datafile).toDF(column_names: _*)