topic Databricks pyspark - Find columns in xls file. in Data Engineering

Databricks pyspark - Find columns in xls file.

weldermartins — Sat, 03 Sep 2022 15:14:54 GMT

Hello everyone, every day I extract data into xls files but the column position changes every day. Is there any way to find these columns within the file?

Here's a snippet of my code.

df = spark.read.format("com.crealytics.spark.excel")\
  .option("header", "true")\
  .schema(schema)\
  .option("dataAddress", "'releases'!A27:D78") \
  .load("dbfs:/FileStore/tables/invoice_september.xls")
df.display()

Re: Databricks pyspark - Find columns in xls file.

jose_gonzalez — Fri, 09 Sep 2022 23:18:01 GMT

You can also do df.printSchema() to check. Or even dbutils.fs.head(<file_path>) to check the header's position. Docs https://docs.databricks.com/dev-tools/databricks-utils.html

Re: Databricks pyspark - Find columns in xls file.

Vidula — Wed, 21 Sep 2022 05:20:47 GMT

Hi @welder martins

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Databricks pyspark - Find columns in xls file.

weldermartins — Thu, 06 Oct 2022 12:07:12 GMT

Hello, come shape. Thanks!

Re: Databricks pyspark - Find columns in xls file.

Debayan — Mon, 05 Sep 2022 22:29:37 GMT

Hi, Thanks for reaching out to community.databricks.com.

Please refer and let us know if this helps, you can find column names: https://sparkbyexamples.com/pyspark/pyspark-find-datatype-column-names-of-dataframe/