Show Existing Header From CSV I External Table

[Frantz]

Hello, is there a way to load csv data into an external table without the _c0, _c1 columns showing?

[Frantz]

The databricks community discussion post  creation sucks. I've been trying for the past 20 minutes to post a question but I keep getting a "Correct Highlighted errors" message. When I correct the "errors", the message still does not go through. I've resulted to posting test messages just to see what goes through. 

[Kaniz_Fatma]

Hi @Frantz, I'm sorry you're experiencing difficulties posting on the Databricks community discussion. Hang in there, and hopefully, we can get this sorted out soon so you can participate fully in the community discussions.

[Kaniz_Fatma]

Hi @Frantz! Setting the header option to true allows you to easily avoid the preset default column names (_c0, _c1, etc.) when using PySpark to load CSV data into an external table. This allows you to use the first row of your CSV file as the column names, which can be done as follows:

# Assuming you have a CSV file named "file.csv"
dff = spark.read.format("csv") \
   .option("delimiter", ",") \
   .option("header", "true") \
   .option("inferSchema", "true") \
   .load("file.csv")

This code snippet uses the header=True setting to ensure that the first row of the CSV file is recognized as column names. Additionally, the inferSchema=True setting allows for automatic inference of column data types. As a result, the DataFrame dff will contain the same column names as the original CSV file rather than the generic _c0, _c1, etc.

• https://docs.databricks.com/en/ingestion/copy-into/examples.html 

[Frantz · Accepted solution]

My question was answered in a separate thread here.