Databricks Community

MarkD

Hi,I'm trying to set a dynamic value to use in a DLT query, and the code from the example documentation does not work.SET startDate='2020-01-01'; CREATE OR REFRESH LIVE TABLE filtered AS SELECT * FROM my_table WHERE created_at > ${startDate};It is g...

MarkD

Hi,We have a DLT pipeline that has been running for a while with a Hive Metastore target that has stored billions of records. We'd like to move the data to a Unity Catalog. The documentation says "Existing pipelines that use the Hive metastore cannot...

MarkD

The reason I want to use a variable is so that I can use the same cut off time in multiple queries to make sure to make sure that I don't get records in one query with a later date than another.For exampleCREATE OR REFRESH LIVE TABLE filtered_one AS ...

MarkD

I've used the advanced configuration option for other variables, but in this case I wanted something dynamic. I want to `SET startDate = current_time()` and then use `startDate` through all my queries to make sure they are all using the same time com...

MarkD

The example I gave is directly from Databrick's DLT documentation, but as you say does not appear the be the right way to set a variable and doesn't work.Unfortunately SET VAR is not supported in Delta Live Tables.

Databricks Community

User Stats

User Activity

SET configuration in SQL DLT pipeline does not work

Is it possible to migrate data from one DLT pipeline to another?

Re: SET configuration in SQL DLT pipeline does not work

Re: SET configuration in SQL DLT pipeline does not work

Re: SET configuration in SQL DLT pipeline does not work