cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Pradeep_Namani
by New Contributor III
  • 2214 Views
  • 5 replies
  • 2 kudos

Date field getting changed when reading from excel file to dataframe in pyspark

The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/1947.In pyspark datafram...

  • 2214 Views
  • 5 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

how about using inferschema one single time to create a correct DF, then create a schema from the df-schema.something like this f.e.from pyspark.sql.types import StructType   # Save schema from the original DataFrame into json: schema_json = df.s...

  • 2 kudos
4 More Replies
sreedata
by New Contributor III
  • 2357 Views
  • 5 replies
  • 12 kudos

Resolved! Date field getting changed when reading from excel file to dataframe

The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/2022.In dataframe it is ...

  • 2357 Views
  • 5 replies
  • 12 kudos
Latest Reply
Pradeep_Namani
New Contributor III
  • 12 kudos

Hi Team, @Merca Ovnerud​ I am also facing same issue , below is the code snippet which I am using df=spark.read.format("com.crealytics.spark.excel").option("header","true").load("/mnt/dataplatform/Tenant_PK/Results.xlsx")I have a couple of date colum...

  • 12 kudos
4 More Replies
Swapnil1998
by New Contributor III
  • 586 Views
  • 0 replies
  • 2 kudos

Date Formats while extracting data from Cosmos Mongo DB using Azure Databricks.

I have been trying to extract one date field from cosmos which looks like as below:"lastModifiedDate" : {        "$date" : 1668443121840    }when the above field is extracted using Databricks it gets converted into a date format which looks like this...

  • 586 Views
  • 0 replies
  • 2 kudos
Labels