Hello, I'm pretty new to Databricks in general and Delta Live Tables specifically. My problem statement is that I'd like loop through a set of files and run a notebook that loads the data into some Delta Live Tables. Additionally, I'd like to include the name of the file as one of the columns that is created in my DLT.
What I've tried so far is to create a widget at the beginning of my notebook and include it in the SELECT portion of my CREATE or REFRESH STREAMING LIVE TABLE code. I've successfully tested the syntax for the widget creation. However, I'm getting a syntax error when I run the notebook as part of a pipeline. BTW, this widget is intended to receive the file name being passed from the MAIN notebook calling this one.
I may be going about this whole thing in a less-than-perfect/best practice way and am open to ideas.....or, would love some ideas on how to get the widget to work.....or, a confirmation that you CANNOT use widgets in DLT pipelines for some reason and therefore need to look for a different solution.
The widget is in a different cell from the table. Below is my code.
When I manually run just the create and select, the syntax works fine:
When I run the code through a pipeline, I get this error:
1 ACCEPTED SOLUTION
Accepted Solutions
