- 7662 Views
- 6 replies
- 4 kudos
DLT | Cluster terminated by System-User | INTERNAL_ERROR: Communication lost with driver. Cluster 0312-140502-k9monrjc was not reachable for 120 seconds
Dear Community, Hope you are doing well.For the last couple of days I am seeing very strange issues with my DLT pipeline, So every 60-70 mins it is getting failed in continuous mode, with the ERROR; INTERNAL_ERROR: Communication lost with driver. Clu...
- 7662 Views
- 6 replies
- 4 kudos
- 4 kudos
We had similar error for one the DLT pipeline, This could be some times because of compute size, we had increased compute size of server in your DLT pipelines, still we were seeing this error while processing very large file. we then added below para...
- 4 kudos
- 3892 Views
- 3 replies
- 13 kudos
Using foreachBatch within Delta Live Tables framework
Hey there!​I was wondering if there's any way of declaring a delta live table where we use foreachBatch to process the output of a streaming query.​Here's a simplification of my code:​def join_data(df_1, df_2): df_joined = ( df_1 ...
- 3892 Views
- 3 replies
- 13 kudos
- 13 kudos
foreachBatch support in DLT is coming soon, and you now have the ability to write to non-DLT sinks as well
- 13 kudos
- 7016 Views
- 5 replies
- 2 kudos
Resolved! Create Jobs and Pipelines in Workflows using API
I am trying to create Databricks Jobs and Delta live table(DLT) pipelines by using Databricks API.I would like to have the JSON code of Jobs and DLT in the repository(to configure the code as per environment) and execute the Databricks API by passing...
- 7016 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi Jose,Yes it answered my question. I am indeed using JSON file to create Jobs and pipelinesThanks.
- 2 kudos
- 4054 Views
- 5 replies
- 7 kudos
How to handle schema validation for Json file. Using Databricks Autoloader?
Following are the details of the requirement:1. I am using databricks notebook to read data from Kafka topic and writing into ADLS Gen2 container i.e., my landing layer.2. I am using Spark code to read data from Kafka and write into landing...
- 4054 Views
- 5 replies
- 7 kudos
- 7 kudos
just to clarify, are you reading kafka and writing into adls in json files? like for each message from kafka is 1 json file in adls ?
- 7 kudos
- 10462 Views
- 2 replies
- 0 kudos
Delta Live Tables + S3 | 5 tips for cloud storage with DLT
You’ve gotten familiar with Delta Live Tables (DLT) via the quickstart and getting started guide. Now it’s time to tackle creating a DLT data pipeline for your cloud storage–with one line of code. Here’s how it’ll look when you're starting:CREATE OR ...
- 10462 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi MadelynM,How should we handle Source File Archival and Data Retention with DLT? Source File Archival: Once the data from source file is loaded with DLT Auto Loader, we want to move the source file from source folder to archival folder. How can we ...
- 0 kudos
- 14263 Views
- 4 replies
- 1 kudos
Resolved! custom upsert for delta live tables apply_changes()
Hello community :).I am currently implementing some pipelines using DLT. They are working great for my medalion architecture for landed json in bronze -> silver (using apply_changes) then materialized gold views ontop.However, I am attempting to crea...
- 14263 Views
- 4 replies
- 1 kudos
- 1 kudos
Is it possible to have custom upserts for streaming tables in delta live tables?Im getting the error:pyspark.errors.exceptions.captured.AnalysisException: `blusmart_poc.information_schema.sessions` is not a Delta table.
- 1 kudos
- 9190 Views
- 6 replies
- 3 kudos
Delta Live Tables Incremental Batch Loads & Failure Recovery
Hello Databricks community,I'm working on a pipeline and would like to implement a common use case using Delta Live Tables. The pipeline should include the following steps:Incrementally load data from Table A as a batch.If the pipeline has previously...
- 9190 Views
- 6 replies
- 3 kudos
- 3 kudos
I totally agree that this is a gap in the Databricks solution. This gap exists between a static read and real time streaming. My problem (and suspect there are many use cases) is that I have slowly changing data coming into structured folders via ...
- 3 kudos
- 4393 Views
- 5 replies
- 2 kudos
Delta Live Tables: bulk import of historical data?
Hello! I'm very new to working with Delta Live Tables and I'm having some issues. I'm trying to import a large amount of historical data into DLT. However letting the DLT pipeline run forever doesn't work with the database we're trying to import from...
- 4393 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Sarah Guido​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 2 kudos
- 10750 Views
- 4 replies
- 0 kudos
Can I pass parameters to a Delta Live Table pipeline at running time?
I need to execute a DLT pipeline from a Job, and I would like to know if there is any way of passing a parameter. I know you can have settings in the pipeline that you use in the DLT notebook, but it seems you can only assign values to them when crea...
- 10750 Views
- 4 replies
- 0 kudos
- 0 kudos
This seems to be the key to this question:parameterize for dlt My understanding of this is that you can add the parameter either in the DLT settings UI via Advanced Config/Add Configuration, key, value dialog. Or via the corresponding pipeline set...
- 0 kudos
- 9604 Views
- 5 replies
- 0 kudos
Data Quality in Databricks
Hi Databricks Team, would like to implement data quality rules in Databricks, apart from DLT do we have any standard approach to perform/ apply data quality rules on bronze layer before further proceeding to silver and gold layer.
- 9604 Views
- 5 replies
- 0 kudos
- 5245 Views
- 6 replies
- 5 kudos
Workaround for GraphFrames not working on Delta Live Table?
According to this page, the GraphFrames package is included in the databricks runtime since at least 11.0. However trying to run a connected components algorithm inside a delta live table notebook yields the error java.lang.ClassNotFoundException: or...
- 5245 Views
- 6 replies
- 5 kudos
- 5 kudos
I'm also trying to use GraphFrames inside a DLT pipeline. I get an error that graphframes not installed in the cluster. i"m using it successfully in test notebooks using the ML version of the cluster. Is there a way to use this inside a DLT job?
- 5 kudos
- 1723 Views
- 1 replies
- 0 kudos
DLT aggregation problem
I'm utilizing SQL to perform aggregation operations within a gold layer of a DLT pipeline. However, I'm encountering an error when running the pipeline while attempting to return a data frame using spark.sql.Could anyone please assist me with the SQL...
- 1723 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Yash_542965 , I hope this message finds you well. Could you please share a sample of code you are using so that we can check it further? Best regards,Lucas Rocha
- 0 kudos
- 1200 Views
- 1 replies
- 0 kudos
- 1200 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @User16752244127 , I hope this message finds you well. Delta Live Tables supports loading data from any data source supported by Databricks. You can find the datasources supported here Connect to data sources, and JDBC is one of them. You can a...
- 0 kudos
- 7322 Views
- 2 replies
- 4 kudos
Resolved! Streaming Delta Live Tables
I'm a little confused about how streaming works with DLT. My first questions is what is the difference in behavior if you set the pipeline mode to "Continuous" but in your notebook you don't use the "streaming" prefix on table statements, and simila...
- 7322 Views
- 2 replies
- 4 kudos
- 4 kudos
Is it possible to have custom upserts in streaming tables in a delta live tables pipeline?Use case: I am trying to maintain a valid session based on timestamp column and want to upsert to the target table.Tried going through the documentations but dl...
- 4 kudos
- 7628 Views
- 9 replies
- 3 kudos
DLT managed by non-existent pipeline
I am building out a new DLT pipeline and have since had to rebuild it from scratch. Having deleted the old pipeline and constructed a new one I now get this error:Table 'X' is already managed by pipeline 'Y'. As I only have the one pipeline how would...
- 7628 Views
- 9 replies
- 3 kudos
- 3 kudos
rename your function from @Dlt.table, for exemple:@Dlt.table( comment="exemple", table_properties={"exemple": "exemple"}, partition_cols=["a", "b", "c"])def modify_this_name():
- 3 kudos