cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Using variables in Spark SQL

HariharaSam
Contributor

Is there a way to declare variables in Spark SQL like we do it in T-SQL?

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

Nice question. There was a discussion about the topic here https://community.databricks.com/s/question/0D53f00001c9RUYCA2/whats-the-equivalent-of-declare-in-da...

So there are functions, widgets, or just combining Python with SQL. Probably I would choose SQL functions as they are permanent and stay in metastore (but can not be used for everything).

    CREATE OR REPLACE FUNCTION my_name(name STRING COMMENT 'my name')
      RETURNS STRING
      COMMENT 'just first name'
      CONTAINS SQL DETERMINISTIC
      RETURN name || 'Hubert'

Hi Hubert ,

As you have mentioned that it can not be used for everything , in my case also it doesn't suit as I have a lot variables declaration and having a function created for each variable doesn't look good.

Debayan
Databricks Employee
Databricks Employee

Could you please follow the below link and let us know if this helps?

https://community.databricks.com/s/question/0D53f00001HKHa3CAH/how-do-i-pass-parameters-to-my-sql-st...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group