topic Re: getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4 in Get Started Discussions

getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4 LTS

llvu — Thu, 05 Oct 2023 14:33:23 GMT

Hello,

I am trying to use the getArgument() function in a spark.sql query. It works fine if I run the notebook via an interactive cluster, but gives an error when executed via a job run in an instance Pool.

query:

OPTIMIZE <table>

where date = replace(regexp_extract(getArgument('folderPath'), '=\\\d{8}', 0), '=', '')

Error:

Exception: Undefined function: getArgument. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.getargument.; line 2 pos 49

I can't seem to figure out why this error occurs and how I can solve the issue. As the notebook used for the query is a common notebook used by multiple jobs, I would like to resolve the issue without changing anything to the notebook. A change to the query executed by spark.sql() is possible. Could someone help me out?

Interactive cluster settings:

Driver: Standard_DS3_v2

Workers: Standard_DS3_v2

10.4LTS (includes Apache Spark 3.2.1, Scala 2.12)

Pool settings:

Driver: Standard_DS3_v2

Workers: Standard_DS3_v2

10.4LTS (includes Apache Spark 3.2.1, Scala 2.12)

Re: getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4

llvu — Thu, 05 Oct 2023 18:49:55 GMT

Hi @Retired_mod, thank you for your response. Would there be a way to still use the widget value? Or the pyspark parameter if this line is part of the code:

folder_name = dbutils.widgets.get('folderName')

Without rewriting spark.sql("""query""") into spark.sql(f"""query""")?

Re: getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4

llvu — Fri, 06 Oct 2023 11:12:12 GMT

Hi @Retired_mod, thank you for the suggestion. I tried calling the parameter instead of the widget value. However, as I expected, this didn’t resolve the original problem of getArgument() not being recognised as a function.

query:

folder_path = dbutils.widgets.get('folderPath')

Spark.sql(“””

OPTIMIZE <table>

where date = replace(regexp_extract(getArgument('folder_path'), '=\\\d{8}', 0), '=', '') “””)

error:

Exception: Undefined function: getArgument. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.getargument.; line 2 pos 49

Re: getArgument works fine in interactive cluster 10.4 LTS, raises error in interactive cluster 10.4

llvu — Wed, 25 Oct 2023 13:47:04 GMT

Hi @Retired_mod,

Would you be able to respond to my last comment? I couldn't manage to get it working yet.

Thank you in advance.