cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

THIAM_HUATTAN
by Valued Contributor
  • 2630 Views
  • 6 replies
  • 6 kudos

Resolved! Delta Lake’s CDF Feature

https://www.databricks.com/notebooks/delta-lake-cdf.htmlI am trying to understand the above article. Could someone explain to be the below questions?a) From SELECT * FROM table_changes('gold_consensus_eps', 2)why is consensus_eps values of 2.1 and 2....

  • 2630 Views
  • 6 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @THIAM HUAT TAN​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 6 kudos
5 More Replies
Johny
by New Contributor III
  • 6196 Views
  • 2 replies
  • 4 kudos

Insert data to a CDF-enabled Delta table throwing java.lang.StackOverflowError

I am building a bronze table with CDF-enables in these steps:Initially, Reading json file from landing zone and write to table locationdf = spark.readStream.format("cloudFiles") \ .option("cloudFiles.schemaLocation", <schema_loc>) \ .option("clou...

  • 6196 Views
  • 2 replies
  • 4 kudos
Latest Reply
Johny
New Contributor III
  • 4 kudos

I tried with a simple csv file that only has one column. I got the same error.

  • 4 kudos
1 More Replies
Databrickguy
by New Contributor II
  • 1899 Views
  • 2 replies
  • 2 kudos

How to check/list the tables which has CDF enabled?

How to list all the tables which has CDF enabled?I can review a table to find out if CDF is enable with below code.SHOW TBLPROPERTIES tableA(delta.enableChangeDataFeed)The return key ...

  • 1899 Views
  • 2 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Hhi @Tim zhang​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 2 kudos
1 More Replies
RKNutalapati
by Valued Contributor
  • 3523 Views
  • 7 replies
  • 3 kudos

Resolved! Copy CDF enabled delta table from one location to another by retaining history

I am currently doing some use case testing. I have to CLONE delta table with CDF enabled to a different S3 bucket. Deep clone doesn't meet the requirement. So I tried to copy the files using dbutils.fs.cp, it is copying all the versions but the tim...

  • 3523 Views
  • 7 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Rama Krishna N​ , Because you're on Azure, you can use Azure Data Factory's Data Copy Tool as it's described in the documentation - delta tables are just files in the container, and this tool can copy data, and potentially it would be cheaper tha...

  • 3 kudos
6 More Replies
patputnam-db
by New Contributor II
  • 1093 Views
  • 1 replies
  • 0 kudos

When should Change Data Feed be used?

IHAC who has a Change Data Capture data flowing into a Delta table. They would like to propagate these changes from this table into another table downstream. Is this a good application for using Change Data Feed?

  • 1093 Views
  • 1 replies
  • 0 kudos
Latest Reply
patputnam-db
New Contributor II
  • 0 kudos

CDF simplifies the process of identifying the set of records that are updated, inserted, or deleted with each version of a Delta table. It helps to avoid having to implement downstream 'custom' filtration to identify these changes. This makes it an i...

  • 0 kudos
Labels