When running some SQL queries using spark.sql(...), we sometimes get a variant of the following error:

AnalysisException: Undefined function: current_timestamp. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.current_timestamp.; line 2 pos 35

The function that is missing sometimes changes (e.g. in other cases it's UUID()), but they are functions that are standard Databricks SQL builtins.

The SQL queries are being called using pyspark code that is inside a module.

Example:

(not actual code, but edited version since I can't paste everything here)

In the notebook we run:

from sql_utilities import example_log_status
 
example_log_status(id, status)

Which imports code from our module sql_utilities.py:

def example_log_status(id, status):
    query= f"""UPDATE foo.exampleTable
    SET Status="{status}", ModifiedAt=current_timestamp()
    WHERE RunLogId="{RunLogId}"
    """
    spark.sql(query)

Extra information:

Databricks runtime: 11.3 LTS

This only happens when jobs are scheduled from Azure DataFactory. We were not able to replicate this by manually running these queries. The jobs are typically scheduled in parallel on the same cluster.