- 4724 Views
- 5 replies
- 2 kudos
Date field getting changed when reading from excel file to dataframe in pyspark
The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/1947.In pyspark datafram...
- 4724 Views
- 5 replies
- 2 kudos
- 2 kudos
how about using inferschema one single time to create a correct DF, then create a schema from the df-schema.something like this f.e.from pyspark.sql.types import StructType # Save schema from the original DataFrame into json: schema_json = df.s...
- 2 kudos
- 4914 Views
- 4 replies
- 10 kudos
Resolved! Date field getting changed when reading from excel file to dataframe
The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/2022.In dataframe it is ...
- 4914 Views
- 4 replies
- 10 kudos
- 10 kudos
Hi Team, @Merca Ovnerud​ I am also facing same issue , below is the code snippet which I am using df=spark.read.format("com.crealytics.spark.excel").option("header","true").load("/mnt/dataplatform/Tenant_PK/Results.xlsx")I have a couple of date colum...
- 10 kudos
- 1236 Views
- 0 replies
- 2 kudos
Date Formats while extracting data from Cosmos Mongo DB using Azure Databricks.
I have been trying to extract one date field from cosmos which looks like as below:"lastModifiedDate" : { "$date" : 1668443121840 }when the above field is extracted using Databricks it gets converted into a date format which looks like this...
- 1236 Views
- 0 replies
- 2 kudos
- 832 Views
- 0 replies
- 0 kudos
How Delta Lake parses through the files on querying the table with a filter condition
Consider a table that gets partitioned on a date field. But, I'm filtering a column that is not partitioned. Now, with this filter condition whether all the files are parsed to attain the required result set, or does any data skipping happens?
- 832 Views
- 0 replies
- 0 kudos