DLT behaving differently when used with python syntax vs when used with sql syntax to read CDF
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
I was trying t read CDF data of a table as a DLT materialized view.
It works fine with sql syntax reading all the columns of the source table along with the 3 CDF columns : _change_type,_commit_timestamp,_commit_version:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
But the same python code works fine when executed outside of a DLT pipeline. When I run the following in an interactive notebook it returns the source columns + CDF columns, which is logical because I am using the readChangeFeed option while reading.
spark.read.option('readChangeFeed','True').option('startingVersion',1).table(<source_table_name>)
The problem I stated occurs only when it is executed within a DLT pipeline which is strange.

