Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-24-2019 12:27 AM
Hi @sani nanda,
Please follow the below steps,
- Read the schema file as a CSV, setting header to true. This will give an empty dataframe but with the correct header.
- Extract the column names from that schema file.
column_names = spark.read.option("header",true).csv(schemafile).columns- Now read the datafile and change the default column names to the ones in the schema dataframe.
df = spark.read.option("header", "false").option("inferSchema", "true").csv(datafile).toDF(column_names: _*)