I have a scheduled job that is executed using a notebook. Within one of the notebook cells, there is a check to determine if a table exists. However, even when the table does exist, it incorrectly identifies it as non-existent and proceeds to execute the subsequent code.
spark.sql("CREATE SCHEMA IF NOT EXIST schema_name")
While executing the above code, it throws the below error:
shaded.v245.com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
Even though I haven't used the "<" character in the code, it is still throwing the same error. Error is only for 5-6 runs in a whole day, rest of the time it is working fine. How to avoid this kind of error?
Note: I am using Job Compute cluster to run this job.
