Hi all,
We have our medallion architecture transformation on databricks.
Im currently testing upgrading to 11.3 as 10.4 won't be supported anymore from March 2025.
However, I keep getting errors like this:
- Error inserting data into table. Type AnalysisException: cannot resolve 'CAST(v_staging_silverinsert_silver_docusign_docusign.RawDataStartDate AS BOOLEAN)' due to data type mismatch: cannot cast timestamp to boolean with ANSI mode on.
- [CAST_INVALID_INPUT] The value 'Default' of the type "STRING" cannot be cast to "BOOLEAN" because it is malformed.
I don't understand why this is happening, it seems to happen on this piece of code:
Raw.createOrReplaceTempView("v_staging_silverInsert_{0}_{1}".format(TargetSchema,RawObject))
print('Inserting data into table: {0}.{1}'.format(TargetSchema,RawObject))
try:
sqlstmt = 'INSERT INTO {0}.{1} select * from v_staging_silverInsert_{0}_{1}'.format(TargetSchema,RawObject)
sqlContext.sql(sqlstmt)
Continue = False
WriteDeletes = False
print('Silver has been written. Continue : {0}'.format(str(Continue)))
IsFinal = 'True'
except Exception as e:
hasError = True
Continue = False
Exception_type = type(e).__name__
ErrorMessages = '** Error inserting data into table. Type {}: {}'.format(Exception_type,e)
print(ErrorMessages)
Logging_Error(ErrorMessages)
What I don't understand is. I'm never casting these fields. So why is it that on databricks runtime 11.3 it tries to cast these fields?
I tried turning off the ANSI enforcement but unfortunately that is causing me other issues.