cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Merchiv
by New Contributor III
  • 8503 Views
  • 4 replies
  • 3 kudos

Resolved! How can I add a duration in milliseconds to a timestamp?

Let's say I have a DataFrame with a timestamp and an offset column in milliseconds respectively in the timestamp and long format. E.g.from datetime import datetime df = spark.createDataFrame( [ (datetime(2021, 1, 1), 1500, ), (dat...

  • 8503 Views
  • 4 replies
  • 3 kudos
Latest Reply
Merchiv
New Contributor III
  • 3 kudos

Although @Lakshay Goel​'s solution works, we've been using an alternative approach, that we found to be a bit more readable:from pyspark.sql import Column, functions as f     def make_dt_interval_sec(col: Column): return f.expr(f"make_dt_interval...

  • 3 kudos
3 More Replies
vr
by Contributor
  • 3050 Views
  • 3 replies
  • 2 kudos

Resolved! Is timestamp difference always INTERVAL DAY TO SECOND?

My observations show that timestamp difference has type of INTERVAL DAY TO SECONDS:select typeof(getdate() - current_date()) ----------------------------------------- interval day to secondBut is it guaranteed? Can it be DAY TO MINUTE or, say, YEAR T...

  • 3050 Views
  • 3 replies
  • 2 kudos
Latest Reply
sher
Valued Contributor II
  • 2 kudos

you can check here for given example: https://docs.databricks.com/sql/language-manual/functions/minussign.htmlthis might help to you.

  • 2 kudos
2 More Replies
Labels