cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

noimeta
by Contributor II
  • 2154 Views
  • 4 replies
  • 1 kudos

Apply change data with delete and schema evolution

Hi,Currently, I'm using structure streaming to insert/update/delete to a table. A row will be deleted if value in 'Operation' column is 'deleted'. Everything seems to work fine until there's a new column.Since I don't need 'Operation' column in the t...

  • 2154 Views
  • 4 replies
  • 1 kudos
Latest Reply
User16753725469
Contributor II
  • 1 kudos

please go through this documentation https://docs.delta.io/latest/api/python/index.html

  • 1 kudos
3 More Replies
yatharthmahesh
by New Contributor III
  • 1592 Views
  • 4 replies
  • 6 kudos

ENABLE CHANGE DATA FEED FOR EXISTING DELTA-TABLE

I have a delta table already created, now I want to enable the change data feed. I read that I have to set delta.enableChangeDataFeed property to true. But however, this cannot be done using the Scala API. I tried using this but it didn't work. I am ...

  • 1592 Views
  • 4 replies
  • 6 kudos
Latest Reply
Kaniz
Community Manager
  • 6 kudos

Hi @Yatharth Maheshwari​, We haven’t heard from you on the last response from @Jose Gonzalez​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the community as it can be helpf...

  • 6 kudos
3 More Replies
hari
by Contributor
  • 4030 Views
  • 8 replies
  • 4 kudos

Resolved! How to write Change Data from Delta Lake to aws dynamodb

Is there some direct way to write data from DeltaLake to AWS DynamoDB.If there is none, Is there any way to do the same.

  • 4030 Views
  • 8 replies
  • 4 kudos
Latest Reply
jose_gonzalez
Moderator
  • 4 kudos

Hi @Harikrishnan P H​ ,Did @Werner Stinckens​ reply help you to resolved your issue? if yes, please mark it as best. if not, please let us know.

  • 4 kudos
7 More Replies
Suman
by New Contributor III
  • 1818 Views
  • 5 replies
  • 3 kudos

Resolved! Change Data Feed functionality from SQL Endpoint

I am trying to run command to retrieve change data from sql endpoint. It is throwing below error."The input query contains unsupported data source(s).Only csv, json, avro, delta, parquet, orc, text data sources are supported on Databricks SQL."But th...

  • 1818 Views
  • 5 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Suman Chowdhury​ , Change Data Feed is only available in Databricks Runtime 8.4 and above.

  • 3 kudos
4 More Replies
Ian
by New Contributor III
  • 3180 Views
  • 6 replies
  • 0 kudos

Resolved! Databricks-Connect and Change Data Feed query error

I have installed Databricks-Connect (9.1 LTS). I am able to send queries to the cluster. However, when the query includes a call to the 'table_changes' function that is a part of Change Data Feed, I get the following error:AnalysisException("could ...

  • 3180 Views
  • 6 replies
  • 0 kudos
Latest Reply
Ian
New Contributor III
  • 0 kudos

Hi @Kaniz Fatma​ , the table_changes function is an internal Databricks function used in Change Data Feed (CDF).Please refer to the article below. It discusses the table_changes function.https://docs.databricks.com/delta/delta-change-data-feed.html

  • 0 kudos
5 More Replies
User16826994223
by Honored Contributor III
  • 1269 Views
  • 1 replies
  • 1 kudos
  • 1269 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 1 kudos

# providing a starting version spark.readStream.format("delta") \ .option("readChangeFeed", "true") \ .option("startingVersion", 0) \ .table("myDeltaTable")   # providing a starting timestamp spark.readStream.format("delta") \ .option("readCh...

  • 1 kudos
User16826994223
by Honored Contributor III
  • 926 Views
  • 1 replies
  • 0 kudos
  • 926 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

ou must explicitly enable the change data feed option using one of the following methods:New table: Set the table property  delta.enableChangeDataFeed = true in the CREATE TABLE command.CREATE TABLE student (id INT, name STRING, age INT) TBLPROPERTIE...

  • 0 kudos
Labels