cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
lshar
by • New Contributor III
  • 29099 Views
  • 7 replies
  • 5 kudos

Resolved! How do I pass arguments/variables from widgets to notebooks?

Hello,I am looking for a solution to this problem, which is known since 7 years: https://community.databricks.com/s/question/0D53f00001HKHZfCAP/how-do-i-pass-argumentsvariables-to-notebooksWhat I need is to parametrize my notebooks using widget infor...

example_if_run
Data Engineering
Reply
  • 29099 Views
  • 7 replies
  • 5 kudos
View Replies
Latest Reply
T_Ash
New Contributor II
  • 5 kudos

Can we create paginated reports with multiple parameters(one parameter can dynamically change other parameter) or we can pass one variable from one dataset to other dataset like power bi paginated report using Databricks dashboard, please let me know...

  • 5 kudos
6 More Replies
Altay
by • New Contributor II
  • 669 Views
  • 0 replies
  • 0 kudos

Delta merge drops cached variables

Hi Everyone,I have an ingestion script where I use the delta merge to update and append newly incoming data in dataframe format to an existing delta table.I am experiencing an issue where all the variables that have been used previously loose their d...

Data Engineering
Reply
  • 669 Views
  • 0 replies
  • 0 kudos
Marcel
by • New Contributor III
  • 27252 Views
  • 4 replies
  • 2 kudos

Resolved! Set environment variables in global init scripts

Hi Databricks Community,I want to set environment variables for all clusters in my workspace.The goal is to have environment (dev, prod) specific environment variables values.Instead of set the environment variables for each cluster, a global script ...

Data Engineering
Reply
  • 27252 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
brickster
New Contributor II
  • 2 kudos

We have set the env variable at Global Init script as below,sudo echo DATAENV=DEV >> /etc/environmentand we try to access the variable in notebook that run with "Shared" cluster mode. import os print(os.getenv("DATAENV"))But the env variable is not a...

  • 2 kudos
3 More Replies
HariharaSam
by • Contributor
  • 16245 Views
  • 3 replies
  • 4 kudos

Using variables in Spark SQL

Is there a way to declare variables in Spark SQL like we do it in T-SQL?

Data Engineering
Reply
  • 16245 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 4 kudos

Could you please follow the below link and let us know if this helps? https://community.databricks.com/s/question/0D53f00001HKHa3CAH/how-do-i-pass-parameters-to-my-sql-statements

  • 4 kudos
2 More Replies
sgannavaram
by • New Contributor III
  • 3437 Views
  • 1 replies
  • 2 kudos

Resolved! How to pass variables into query string?

I have two variables StartTimeStmp and EndTimeStmp, i am going to assign the Start timestamp to it based on Last Successful Job Runtime and EndTimeStamp would be current time of system.SET StartTimeStmp = '2022-03-24 15:40:00.000';SET EndTimeStmp = '...

Data Engineering
Reply
  • 3437 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

@Srinivas Gannavaram​ , in python:spark.sql(f""" SELECT CI.CORPORATE_ITEM_INTEGRATION_ID , CI.CORPORATE_ITEM_CD WHERE CI.DW_CREATE_TS < '{my_timestamp_variable}' ; """)

  • 2 kudos
bdc
by • New Contributor III
  • 9878 Views
  • 4 replies
  • 3 kudos

Is it possible to access a variable in markdown cells?

I saw a similar question in a discussion 5 years ago and back then this option was not available. https://community.databricks.com/s/question/0D53f00001HKHhNCAX/markup-in-databricks-notebookI wonder if this feature has been added. It is possible to d...

Data Engineering
Reply
  • 9878 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
bdc
New Contributor III
  • 3 kudos

When I click on the feedback link under workspace/help, it opens a page with page not found.

  • 3 kudos
3 More Replies
MikeK_
by • New Contributor II
  • 14207 Views
  • 1 replies
  • 0 kudos

Resolved! SQL variables in a notebook

Hi, In an SQL notebook, using this link: https://docs.databricks.com/spark/latest/spark-sql/language-manual/set.html I managed to figure out to set values and how to get the value. SET my_val=10; //saves the value 10 for key my_val SET my_val; //dis...

Data Engineering
Reply
  • 14207 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
shyam_9
Databricks Employee
  • 0 kudos

Hi @Mike K.., you can do this with widgets and getArgument. Here's a small example of what that might look like: https://community.databricks.com/s/feed/0D53f00001HKHZfCAP

  • 0 kudos
tonyp
by • New Contributor II
  • 15566 Views
  • 1 replies
  • 1 kudos

How to pass a python variables to shell script.?

How to pass a python variables to shell script.in databricks notebook, The python parameters can passed from the 1 st cmd to next %sh cmd .?

Data Engineering
Reply
  • 15566 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
erikvisser1
New Contributor II
  • 1 kudos

I found the answer here: https://stackoverflow.com/questions/54662605/how-to-pass-a-python-variables-to-shell-script-in-azure-databricks-notebookbles basically: %python import os l =['A','B','C','D'] os.environ['LIST']=' '.join(l)print(os.getenv('L...

  • 1 kudos
samalexg
by • New Contributor III
  • 17920 Views
  • 13 replies
  • 1 kudos

How to add environment variable

Instead of setting the AWS accessKey and secret Key in hadoopConfiguration, I would like to add those in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. How can I do that in databricks?

Data Engineering
Reply
  • 17920 Views
  • 13 replies
  • 1 kudos
View Replies
Latest Reply
jric
New Contributor II
  • 1 kudos

It is possible! I was able to confirm that the following post's "Best" answer works: https://forums.databricks.com/questions/11116/how-to-set-an-environment-variable.htmlFYI for @Miklos Christine​  and @Mike Trewartha​ 

  • 1 kudos
12 More Replies
Labels