https://community.databricks.com/t5/data-engineering/timestamp-nanos-false-error/m-p/110742#M43667 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/135375">@staskh</a></P> <P>That error happens because data type is not supported natively. You can try with below spark setting:</P> <DIV id="C052QTRP5EE-1714617499.378639-thread-list-Thread_1714617499.378639" class="c-virtual_list__item" tabindex="0" role="listitem" aria-setsize="-1" data-qa="virtual-list-item" data-item-key="1714617499.378639"> <DIV class="c-message_kit__background c-message_kit__message c-message_kit__thread_message c-message_kit__thread_message--root" role="presentation" data-qa="message_container" data-qa-unprocessed="false" data-qa-placeholder="false"> <DIV class="c-message_kit__hover" role="document" aria-roledescription="message" data-qa-hover="true"> <DIV class="c-message_kit__actions c-message_kit__actions--inside"> <DIV class="c-message_kit__gutter"> <DIV class="c-message_kit__gutter__right" role="presentation" data-qa="message_content"> <DIV class="c-message_kit__blocks c-message_kit__blocks--rich_text"> <DIV class="c-message__message_blocks c-message__message_blocks--rich_text" data-qa="message-text"> <DIV class="p-block_kit_renderer" data-qa="block-kit-renderer"> <DIV class="p-block_kit_renderer__block_wrapper p-block_kit_renderer__block_wrapper--first"> <DIV class="p-rich_text_block" dir="auto"> <DIV class="p-rich_text_section">spark.conf.set(“spark.sql.legacy.parquet.nanosAsLong”, “true”)</DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> <DIV id="C052QTRP5EE-1714617499.378639-thread-list-Thread_separator" class="c-virtual_list__item" tabindex="-1" role="listitem" aria-setsize="-1" data-qa="virtual-list-item" data-item-key="separator"> <DIV class="p-thread_separator_row_generic p-thread_separator_row_generic--dark"> <DIV class="p-threads_flexpane__separator">&nbsp;</DIV> </DIV> </DIV> Thu, 20 Feb 2025 12:49:13 GMT Alberto_Umana 2025-02-20T12:49:13Z https://community.databricks.com/t5/data-engineering/timestamp-nanos-false-error/m-p/110740#M43665 <P>Hi,</P><P>I'm getting&nbsp;<STRONG><SPAN>Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false))&nbsp;</SPAN></STRONG><SPAN>error while trying to read a parquet file (generated outside of DataBricks)</SPAN><SPAN>. Unfortunately, due to security configuration, I do not have the ability to read it with pandas or similar - only&nbsp;</SPAN><SPAN>spark.read.</SPAN><SPAN>parquet()</SPAN><SPAN>. Any ideas what to do?</SPAN></P><P><SPAN>Regards</SPAN></P><P><SPAN>Stas&nbsp;</SPAN></P><P>&nbsp;</P> Thu, 20 Feb 2025 12:45:03 GMT https://community.databricks.com/t5/data-engineering/timestamp-nanos-false-error/m-p/110740#M43665 staskh 2025-02-20T12:45:03Z https://community.databricks.com/t5/data-engineering/timestamp-nanos-false-error/m-p/110742#M43667 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/135375">@staskh</a></P> <P>That error happens because data type is not supported natively. You can try with below spark setting:</P> <DIV id="C052QTRP5EE-1714617499.378639-thread-list-Thread_1714617499.378639" class="c-virtual_list__item" tabindex="0" role="listitem" aria-setsize="-1" data-qa="virtual-list-item" data-item-key="1714617499.378639"> <DIV class="c-message_kit__background c-message_kit__message c-message_kit__thread_message c-message_kit__thread_message--root" role="presentation" data-qa="message_container" data-qa-unprocessed="false" data-qa-placeholder="false"> <DIV class="c-message_kit__hover" role="document" aria-roledescription="message" data-qa-hover="true"> <DIV class="c-message_kit__actions c-message_kit__actions--inside"> <DIV class="c-message_kit__gutter"> <DIV class="c-message_kit__gutter__right" role="presentation" data-qa="message_content"> <DIV class="c-message_kit__blocks c-message_kit__blocks--rich_text"> <DIV class="c-message__message_blocks c-message__message_blocks--rich_text" data-qa="message-text"> <DIV class="p-block_kit_renderer" data-qa="block-kit-renderer"> <DIV class="p-block_kit_renderer__block_wrapper p-block_kit_renderer__block_wrapper--first"> <DIV class="p-rich_text_block" dir="auto"> <DIV class="p-rich_text_section">spark.conf.set(“spark.sql.legacy.parquet.nanosAsLong”, “true”)</DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> <DIV id="C052QTRP5EE-1714617499.378639-thread-list-Thread_separator" class="c-virtual_list__item" tabindex="-1" role="listitem" aria-setsize="-1" data-qa="virtual-list-item" data-item-key="separator"> <DIV class="p-thread_separator_row_generic p-thread_separator_row_generic--dark"> <DIV class="p-threads_flexpane__separator">&nbsp;</DIV> </DIV> </DIV> Thu, 20 Feb 2025 12:49:13 GMT https://community.databricks.com/t5/data-engineering/timestamp-nanos-false-error/m-p/110742#M43667 Alberto_Umana 2025-02-20T12:49:13Z