cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

bgerhardi
by New Contributor III
  • 8249 Views
  • 13 replies
  • 13 kudos

Surrogate Keys with Delta Live

We are considering moving to Delta Live tables from a traditional sql-based data warehouse. Worrying me is this FAQ on identity columns Delta Live Tables frequently asked questions | Databricks on AWS this seems to suggest that we basically can't cre...

  • 8249 Views
  • 13 replies
  • 13 kudos
Latest Reply
Pelle123
New Contributor II
  • 13 kudos

I am wondering if there is an answer to this question now.

  • 13 kudos
12 More Replies
tinai_long
by New Contributor III
  • 7842 Views
  • 10 replies
  • 4 kudos

Resolved! How to refresh a single table in Delta Live Tables?

Suppose I have a Delta Live Tables framework with 2 tables: Table 1 ingests from a json source, Table 2 reads from Table 1 and runs some transformation.In other words, the data flow is json source -> Table 1 -> Table 2. Now if I find some bugs in the...

  • 7842 Views
  • 10 replies
  • 4 kudos
Latest Reply
cpayne_vax
New Contributor III
  • 4 kudos

Answering my own question: nowadays (February 2024) this can all be done via the UI.When viewing your DLT pipeline there is a "Select tables for refresh" button in the header. If you click this, you can select individual tables, and then in the botto...

  • 4 kudos
9 More Replies
Oliver_Angelil
by Valued Contributor II
  • 2438 Views
  • 4 replies
  • 2 kudos

Resolved! Are data health check expectations available only on Delta Live tables?

I love the idea of "expectations" being available for Delta Live tables: https://docs.databricks.com/delta-live-tables/expectations.htmlI'd like to know if they are also available for regular delta tables?Thank you in advance!

  • 2438 Views
  • 4 replies
  • 2 kudos
Latest Reply
erigaud
Honored Contributor
  • 2 kudos

Hello @Oliver_Angelil, so have you found a way to implement something resembling expectations for delta tables outside of a DLT pipeline ? 

  • 2 kudos
3 More Replies
Prachi_Sankhala
by New Contributor
  • 8985 Views
  • 7 replies
  • 1 kudos

Resolved! What are the advantages of using Delta Live tables (DLT) over Data Build Tool (dbt) in Databricks?

Please explain with some use cases which show the difference between DLT and dbt.

  • 8985 Views
  • 7 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Prachi Sankhala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 1 kudos
6 More Replies
Abhradwip
by New Contributor II
  • 3023 Views
  • 3 replies
  • 0 kudos

How to create Delta Live table from Json files using Custom schema? I am getting the below error for the attached code # Error org.apache.spark.sql.AnalysisException: Table has a user-specified schema that is incompatible with the schema

#### Code# CodeImport DataTypefrom pyspark.sql.types import StructType, StructField, TimestampType, IntegerType, StringType, FloatType, BooleanType, LongType# Define Custom Schemacall_schema = StructType(  [    StructField("RecordType", StringType(),...

  • 3023 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Abhradwip Mukherjee​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 0 kudos
2 More Replies
colt
by New Contributor III
  • 2304 Views
  • 3 replies
  • 2 kudos

Using built-in SQL functions in Delta Live tables

Do Delta Live Tables have different built-in SQL functions than the corresponding Databricks runtime? I created a cluster with Databricks runtime 10.3 (the current DLT runtime) so I could test my Delta Live Tables code before running it as a pipeline...

  • 2304 Views
  • 3 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @Colt Kesselring​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
2 More Replies
fshimamoto
by New Contributor III
  • 1972 Views
  • 3 replies
  • 2 kudos

What are the best practices for schema drift using Delta Live tables, in a scenario where the main source is a no sql database and we have a lot of ch...

What are the best practices for schema drift using Delta Live tables, in a scenario where the main source is a no sql database and we have a lot of changes in the schema?​

  • 1972 Views
  • 3 replies
  • 2 kudos
Latest Reply
Vartika
Moderator
  • 2 kudos

Hey there @Fernando Martin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from...

  • 2 kudos
2 More Replies
NicolasJ
by New Contributor
  • 3174 Views
  • 1 replies
  • 0 kudos

Resolved! How to use Apache Sedona on Delta Live tables?

Hello,I am trying to run some geospatial transformations in Delta Live Table, using Apache Sedona.I tried defining a minimal example pipeline demonstrating the problem I encounter.First cell of my Notebook, I install apache-sedona Python package:%pip...

image
  • 3174 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Nicolas Jean​ , Unfortunately, installing the third-pa/rty Java libraries is not yet supported for the Delta Live Tables, so you can't use Sedona with DLT right now.But suppose you're interested in the geospatial things on Databricks. You may loo...

  • 0 kudos
Labels