cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Hubert-Dudek
by • Esteemed Contributor III
  • 1089 Views
  • 1 replies
  • 4 kudos

spark 3.4 and databricks 13 introduce two new types of timestamps for handling time zone information:- TIMESTAMP WITH LOCAL TIME ZONE: This type assum...

spark 3.4 and databricks 13 introduce two new types of timestamps for handling time zone information:- TIMESTAMP WITH LOCAL TIME ZONE: This type assumes that the input data is in the session's local time zone and converts it to UTC before processing....

timezone
Data Engineering
Reply
  • 1089 Views
  • 1 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

This is helpful! Timestamps are always the reason to mess up the business logic as we know.

  • 4 kudos
jonathan-dufaul
by • Valued Contributor
  • 3387 Views
  • 4 replies
  • 2 kudos

How can I store timestamps using the local (not offset) timestamp instead of storing as UTC-Offset?

I have data that looks like this:2021-11-25T19:00:00.000-0500 2021-03-03T13:00:00.000-0500 2021-03-09T15:00:00.000-0500 2021-03-13T16:00:00.000-0500 2021-03-19T03:00:00.000-0400 2021-05-28T03:00:00.000-0400which is accurate, except I'm pulling the da...

Data Engineering
Reply
  • 3387 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Jonathan Dufault​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...

  • 2 kudos
3 More Replies
najmead
by • Contributor
  • 15900 Views
  • 6 replies
  • 13 kudos

How to convert string to datetime with correct timezone?

I have a field stored as a string in the format "12/30/2022 10:30:00 AM"If I use the function TO_DATE, I only get the date part... I want the full date and time.If I use the function TO_TIMESTAMP, I get the date and time, but it's assumed to be UTC, ...

Data Engineering
Reply
  • 15900 Views
  • 6 replies
  • 13 kudos
View Replies
Latest Reply
Rajeev_Basu
Contributor III
  • 13 kudos

use from_utc_timestamp(to_timestam("<string>", <format>),<timezone>)

  • 13 kudos
5 More Replies
Anonymous
by • Not applicable
  • 1348 Views
  • 2 replies
  • 2 kudos

Time Parameter Returns Incorrect Time

For some reason, the parameter keeps showing me my local time in UTC. When converting this parameter from EST, the offset is wrong. In the screenshot, you can see that my local session TZ is set to ETC/UTC (for whatever reason). It looks like the par...

Data Engineering
Reply
  • 1348 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Zachary Higgins​ Hope everything is going great.Does @Prasad Wagh​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 2 kudos
1 More Replies
hitesh1
by • New Contributor III
  • 6127 Views
  • 1 replies
  • 5 kudos

java.util.NoSuchElementException: key not found

Hello,We are using a Azure Databricks with Standard DS14_V2 Cluster with Runtime 9.1 LTS, Spark 3.1.2 and Scala 2.12 and facing the below issue frequently when running our ETL pipeline. As part of the operation that is failing there are several joins...

Data Engineering
Reply
  • 6127 Views
  • 1 replies
  • 5 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 5 kudos

Hey man,Please use these configuration in your cluster and it will work,spark.sql.storeAssignmentPolicy LEGACYspark.sql.parquet.binaryAsString truespark.speculation falsespark.sql.legacy.timeParserPolicy LEGACYif it wont work let me know what problem...

  • 5 kudos
vamsi0132
by • New Contributor II
  • 937 Views
  • 0 replies
  • 1 kudos

BUG in TIME ZONE EST function

Hi,I found the bug while using in "from_utc_timestamp" function while using from UTC time stamp to EST time stampBelow is the Query Query:select trim(current_timestamp()) as Current,trim(from_utc_timestamp(current_timestamp(),'EST')) as EST,trim(from...

image
Data Engineering
Reply
  • 937 Views
  • 0 replies
  • 1 kudos
explore
by • New Contributor
  • 1338 Views
  • 0 replies
  • 0 kudos

Hi, Can we connect to the Teradata vantage installed in a vm via the community notebook. I am working on a POC to fetch data from Teradata vantate (just a teradata as it uses the jdbc) and process it in community notebook. Downloaded the terajdbc4.jar

from pyspark.sql import SparkSessionspark = SparkSession.builder.getOrCreate()def load_data(driver, jdbc_url, sql, user, password):  return spark.read \    .format('jdbc') \    .option('driver', driver) \    .option('url', jdbc_url) \    .option('dbt...

Data Engineering
Reply
  • 1338 Views
  • 0 replies
  • 0 kudos
Michael_Galli
by • Contributor III
  • 9351 Views
  • 6 replies
  • 3 kudos

Resolved! com.microsoft.sqlserver.jdbc.SQLServerException:The driver could not establish a secure connection to SQL Server by using SSL encr. Error: "Unexpected rethrowing"

Hi all,there is a random error when pushing data from Databricks to a Azure SQL Database.Anyone else also had this problem? Any ideas are appreciated.See stacktrace attached.Target: Azure SQL Database, Standard S6: 400 DTUsDatabricks Cluster config:"...

Data Engineering
Reply
  • 9351 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
Michael_Galli
Contributor III
  • 3 kudos

@Pearl Ubaru​ TLS 1.1 is already deprecated.Are there any concerns from your side to set TLS 1.2 in the connection string?

  • 3 kudos
5 More Replies
MartinB
by • Contributor III
  • 8848 Views
  • 4 replies
  • 3 kudos

Resolved! Interoperability Spark ↔ Pandas: can't convert Spark dataframe to Pandas dataframe via df.toPandas() when it contains datetime value in distant future

Hi,I have multiple datasets in my data lake that feature valid_from and valid_to columns indicating validity of rows.If a row is valid currently, this is indicated by valid_to=9999-12-31 00:00:00.Example:Loading this into a Spark dataframe works fine...

Example_SCD2
Data Engineering
Reply
  • 8848 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
shan_chandra
Esteemed Contributor
  • 3 kudos

Currently, out of bound timestamps are not supported in pyArrow/pandas. Please refer to the below associated JIRA issue. https://issues.apache.org/jira/browse/ARROW-5359?focusedCommentId=17104355&page=com.atlassian.jira.plugin.system.issuetabpanels%3...

  • 3 kudos
3 More Replies
User15787040559
by • New Contributor III
  • 1743 Views
  • 1 replies
  • 0 kudos

What timezone is the “timestamp” value in the Databricks Usage log?

What timezone is the “timestamp” value in the Databricks Usage log ?Is it UTC?timestamp2020-12-01T00:59:59.000ZNeed to match this to AWS Cost Explorer timezone for simplicity.It's UTC.Please see timestamp under Audit Log Schema https://docs.databrick...

Data Engineering
Reply
  • 1743 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User15725630784
New Contributor II
  • 0 kudos

UTC

  • 0 kudos
Labels