- 6679 Views
- 3 replies
- 37 kudos
Understanding DLT pipe lines in Databricks In Databricks, a DLT (Data Live Table) pipeline is a set of data transformations that are applied to data a...
Understanding DLT pipe lines in DatabricksIn Databricks, a DLT (Data Live Table) pipeline is a set of data transformations that are applied to data assets in a defined sequence, in order to clean, enrich, and prepare data for analysis or other purpos...
- 6679 Views
- 3 replies
- 37 kudos
- 4775 Views
- 2 replies
- 0 kudos
How to manage dev/prod environments with new unity catalog delta live table integration
Heres my current setup, dev workspace connected to dev keyvault and a prod workspace connected to a prod keyvault. There's a github repo and action syncing the two environments on pull request and all resources created through terraform. This is my n...
- 4775 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Jordan Fox Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 0 kudos
- 16331 Views
- 8 replies
- 13 kudos
Publish data to Azure ADLS Gen2 from Delta Live Tables pipelines?
I would like to publish data from from delta live table (DLT) to an Azure ADLS Gen2 storage. Checked the documentation but cannot get it to work. Any insights on how to do so?I've tried to specify the "Storage location" with many combinations of abfs...
- 16331 Views
- 8 replies
- 13 kudos
- 13 kudos
I am trying to load from ADLS with DLT, after add the spark configurations I got below errors:org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = 818323fc-80d5-4833-9f46-7d1afc9c5bf7, runId = 722e9aac-0fdd-4206-9d49-68...
- 13 kudos
- 4342 Views
- 2 replies
- 2 kudos
Delta Live tables publish to Unity Catalog
How to enable DLT pipelines for the tables which are in UC. Could you please suggest us the approach or if there is any workaround?
- 4342 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Janga Reddy Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 2 kudos
- 5195 Views
- 4 replies
- 5 kudos
DLT and Unity catalog best practices
What are the best practices when using DLT with a unity catalog aware workspace since it is not supported at the moment?Do you create a copy of every table you need in the hive metastore in order to reference them? Or is it recommended to wait until ...
- 5195 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @E H , the best way to manage this for the time being is to limit DLT to logic that is considered 'internal' i.e., leveraging the default hive metastore. For now at least, this should be treated as a transient landing zone. From here you can refi...
- 5 kudos
- 9096 Views
- 3 replies
- 4 kudos
Resolved! DLT not working with Unity Catalog
Hey guys,I am trying to create a delta live table in Unity Catalog as follows:CREATE OR REFRESH STREAMING LIVE TABLE <catalog>.<db>.<table_name> AS SELECT ... However, I get the error: org.apache.spark.sql.AnalysisException: Unsupported SQL statemen...
- 9096 Views
- 3 replies
- 4 kudos
- 4 kudos
@Anant Pingle As of right now, DLT doesn't support Unity Catalog yet. We can expect this feature in upcomig days.
- 4 kudos
- 3788 Views
- 1 replies
- 7 kudos
Resolved! Unity Catalog support for DLT
Hi,Does the Unity catalog support the Delta Live Tables? If yes, please help me with the documentation link. If not, any update on the feature readiness timeline would help. Thank you!
- 3788 Views
- 1 replies
- 7 kudos
- 7 kudos
Hi @Venkadeshwaran K ,No, UC does not support DLT as of today. We have this feature on the timeline. We will be having a private preview by end of this year for DLT and UC integration for SQL. There is no ETA yet for GA but you can definitely expect...
- 7 kudos