Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Hi,I was wondering that what are differences between Materialized view and Streaming table? which one should I use when I extract data from bronze table to silver table since I found that both CREATE LIVE TABLE and CREATE STREAMING LIVE TABLE could a...
Hi @Mike Chen Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wi...
My data is a dump of JSON response from an API. The schema of the json iscol_name data_type
data array<struct<attributes:struct<name: String, age: Int relationships:struct<address:struct<data:arraay<struct<id: long, type: string>>>>>>>
...
Hi All,I'm new to databricks and learning towards taking up Associate Engineer Certification.While going through the section "Build Data Pipelines with Delta Live Tables". I'm trying to implement Change Data Capture, but it is erroring out when execu...
Having had a quick look, I think your error is because you are trying to add SCD to a STREAMING LIVE table. I believe APPLY CHANGES INTO cannot be used on a streaming table.You can use a streaming table as a source though.Simply changing this line:C...