getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4 LTS

llvu
New Contributor III

‎10-05-2023 07:33 AM

Hello,

I am trying to use the getArgument() function in a spark.sql query. It works fine if I run the notebook via an interactive cluster, but gives an error when executed via a job run in an instance Pool.

query:

OPTIMIZE <table>
where date = replace(regexp_extract(getArgument('folderPath'), '=\\\d{8}', 0), '=', '')

Error:

Exception: Undefined function: getArgument. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.getargument.; line 2 pos 49

 

I can't seem to figure out why this error occurs and how I can solve the issue. As the notebook used for the query is a common notebook used by multiple jobs, I would like to resolve the issue without changing anything to the notebook. A change to the query executed by spark.sql() is possible. Could someone help me out?

 

Interactive cluster settings:

Driver: Standard_DS3_v2

Workers: Standard_DS3_v2

10.4LTS (includes Apache Spark 3.2.1, Scala 2.12)

 

Pool settings:

Driver: Standard_DS3_v2

Workers: Standard_DS3_v2

10.4LTS (includes Apache Spark 3.2.1, Scala 2.12)

 

0 Kudos

llvu
New Contributor III
In response to Retired_mod

‎10-05-2023 11:49 AM

Hi @Retired_mod, thank you for your response. Would there be a way to still use the widget value? Or the pyspark parameter if this line is part of the code:

folder_name = dbutils.widgets.get('folderName')

Without rewriting spark.sql("""query""") into spark.sql(f"""query""")?

0 Kudos

llvu
New Contributor III
In response to Retired_mod

‎10-06-2023 04:12 AM

Hi @Retired_mod, thank you for the suggestion. I tried calling the parameter instead of the widget value. However, as I expected, this didn’t resolve the original problem of getArgument() not being recognised as a function.

query:

folder_path = dbutils.widgets.get('folderPath')

Spark.sql(“””

OPTIMIZE <table>
where date = replace(regexp_extract(getArgument('folder_path'), '=\\\d{8}', 0), '=', '') “””)
 
error:
Exception: Undefined function: getArgument. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.getargument.; line 2 pos 49
0 Kudos

llvu
New Contributor III

‎10-25-2023 06:47 AM

Hi @Retired_mod,

Would you be able to respond to my last comment? I couldn't manage to get it working yet.

Thank you in advance.