cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

b_1
by New Contributor II
  • 707 Views
  • 2 replies
  • 1 kudos

to_timstamp function in non-legacy mode does not parse this format: yyyyMMddHHmmssSS

I have this datetime string in my dataset: '2023061218154258' and I want to convert it to datetime, using below code. However the format that I expect to work, doesn't work, namely: yyyyMMddHHmmssSS. This code will reproduce the issue:from pyspark.sq...

  • 707 Views
  • 2 replies
  • 1 kudos
Latest Reply
b_1
New Contributor II
  • 1 kudos

Is there anybody who has the same issue or knows that this is in fact an issue?

  • 1 kudos
1 More Replies
cmilligan
by Contributor II
  • 4082 Views
  • 6 replies
  • 1 kudos

Resolved! Reference a single item tuple using .format() in spark.sql()

I'm trying to pass the elements of a tuple into a sql query using .format(). This works fine when I have multiple items in my tuple, but when using a single item in a tuple I get an error.tuple1 = (1,2,3) tuple2 = (5,)   combo = tuple1 + tuple2   pri...

Result
  • 4082 Views
  • 6 replies
  • 1 kudos
Latest Reply
Lakshay
Esteemed Contributor
  • 1 kudos

Could you please post the code and the error that you are getting?

  • 1 kudos
5 More Replies
Ericsson
by New Contributor II
  • 1720 Views
  • 2 replies
  • 1 kudos

SQL week format issue its not showing result as 01(ww)

Hi Folks,I've requirement to show the week number as ww format. Please see the below codeselect weekofyear(date_add(to_date(current_date, 'yyyyMMdd'), +35)). also plz refre the screen shot for result.

result
  • 1720 Views
  • 2 replies
  • 1 kudos
Latest Reply
Lauri
New Contributor III
  • 1 kudos

You can use lpad() to achieve the 'ww' format.

  • 1 kudos
1 More Replies
User16826994223
by Honored Contributor III
  • 2598 Views
  • 3 replies
  • 0 kudos

Resolved! Delta lake Check points storage concept

In which format the Checkpoints are stored in storage and , how does it help in delta to increase performance.

  • 2598 Views
  • 3 replies
  • 0 kudos
Latest Reply
aladda
Honored Contributor II
  • 0 kudos

Great points above on how checkpointing helps with performance. In additional Delta Lake also provides other data organization strategies such as compaction, Z-ordering to help with both read and write performance of Delta Tables. Additional details ...

  • 0 kudos
2 More Replies
Labels