cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Can I use Widgets in a Delta Live Table pipeline

BigMF
New Contributor III

Hello, I'm pretty new to Databricks in general and Delta Live Tables specifically. My problem statement is that I'd like loop through a set of files and run a notebook that loads the data into some Delta Live Tables. Additionally, I'd like to include the name of the file as one of the columns that is created in my DLT.

What I've tried so far is to create a widget at the beginning of my notebook and include it in the SELECT portion of my CREATE or REFRESH STREAMING LIVE TABLE code. I've successfully tested the syntax for the widget creation. However, I'm getting a syntax error when I run the notebook as part of a pipeline. BTW, this widget is intended to receive the file name being passed from the MAIN notebook calling this one.

I may be going about this whole thing in a less-than-perfect/best practice way and am open to ideas.....or, would love some ideas on how to get the widget to work.....or, a confirmation that you CANNOT use widgets in DLT pipelines for some reason and therefore need to look for a different solution.

The widget is in a different cell from the table. Below is my code.

When I manually run just the create and select, the syntax works fine:

image 

When I run the code through a pipeline, I get this error:

image

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

No, as notebooks are run from Delta Live Tables Pipelines, and there is no option to pass parameters to notebooks.

View solution in original post

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III

No, as notebooks are run from Delta Live Tables Pipelines, and there is no option to pass parameters to notebooks.

Aviral-Bhardwaj
Esteemed Contributor III

may be it is possible in the job run file there will be configurations option the user can add there

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!