Let's say I have a DataFrame with a timestamp and an offset column in milliseconds respectively in the timestamp and long format.
E.g.
from datetime import datetime
df = spark.createDataFrame(
[
(datetime(2021, 1, 1), 1500, ),
(datetime(2021, 1, 2), 1200, )
],
["timestamp", "offsetmillis", ],
)
Now I want to add these offsets to the datetime, so that I get:
2021-01-01T00:00:01.500 and 2021-01-0T00:00:01.200
If I add these directly I get an error about type mismatch, which does make sense:
[DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES] Cannot resolve "(timestamp + offsetmillis)" due to data type mismatch: the left and right operands of the binary operator have incompatible types ("TIMESTAMP" and "BIGINT")
However I'm not sure how I can best cast this to a duration or interval.