[INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER]

117074 — Tue, 05 Sep 2023 16:01:55 GMT

Hi all,

I'm trying to join 2 views in SQL editor for some analysis. I get the following error:

[INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] You may get a different result due to the upgrading to Spark >= 3.0: Fail to parse '22/12/978' in the new parser. You can set "legacy_time_parser_policy" to "LEGACY" to restore the behavior before Spark 3.0, or set to "CORRECTED" and treat it as an invalid datetime string.

I've tried correcting this by using the command: 'SET legacy_time_parser_policy = legacy;' before my code but it still errors.

Is anyone able to advise on how to solve this?

Thanks

Re: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER]

117074 — Thu, 05 Oct 2023 10:09:22 GMT

Hi Kaniz, I found the equivalent SQL code for this but it didn't seem to store the operation past the execution. I.e I would run the code to configure settings, then run the troublesome code afterwards and still get the same result.

The problem has been identified as source data missing a digit for date of birth year, and therefore having results before 1000AD. This has been rectified at source.

Thank you for your time in answering!

topic [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] in Data Engineering

[INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER]

Re: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER]