topic Re: Using variables in Spark SQL in Data Engineering

Using variables in Spark SQL

HariharaSam — Tue, 30 Aug 2022 09:38:10 GMT

Is there a way to declare variables in Spark SQL like we do it in T-SQL?

Re: Using variables in Spark SQL

Hubert-Dudek — Tue, 30 Aug 2022 20:26:29 GMT

Nice question. There was a discussion about the topic here https://community.databricks.com/s/question/0D53f00001c9RUYCA2/whats-the-equivalent-of-declare-in-databricks-sql-

So there are functions, widgets, or just combining Python with SQL. Probably I would choose SQL functions as they are permanent and stay in metastore (but can not be used for everything).

    CREATE OR REPLACE FUNCTION my_name(name STRING COMMENT 'my name')
      RETURNS STRING
      COMMENT 'just first name'
      CONTAINS SQL DETERMINISTIC
      RETURN name || 'Hubert'

Re: Using variables in Spark SQL

HariharaSam — Wed, 07 Sep 2022 10:13:31 GMT

Hi Hubert ,

As you have mentioned that it can not be used for everything , in my case also it doesn't suit as I have a lot variables declaration and having a function created for each variable doesn't look good.

Re: Using variables in Spark SQL

Debayan — Tue, 30 Aug 2022 21:10:31 GMT

Could you please follow the below link and let us know if this helps?

https://community.databricks.com/s/question/0D53f00001HKHa3CAH/how-do-i-pass-parameters-to-my-sql-statements