Hello,I have some data which are lying into Snowflake, so I want to apply CDC on them using delta live table but I am having some issues.Here is what I am trying to do: @dlt.view()
def table1():
return spark.read.format("snowflake").options(**opt...
I wanna apply a pivot on a dataframe in DLT but I'm having the following warningNotebook:XXXX used `GroupedData.pivot` function that will be deprecated soon. Please fix the notebook.I have the same warning if I use the the function collect.Is it risk...
Finally I followed steps from this blog, and everything works fine.https://www.databricks.com/blog/2022/04/25/simplifying-change-data-capture-with-databricks-delta-live-tables.htmlI just assumed that I have tables as sources and not flat files.Happy ...
Ok I got the point and thank you for your respond.So here is how my data is organisedI have 2 tables in Snowflaketable1 : weekly table containing all the good datatable2 : table contains only 1 week of logs for the changes that happened in the first ...
Hi @Kaniz Fatma , I get your point; it might be risky to use toPandas as you said but as I need just to retrieve distinct values from a specific column of the dataframe so toPandas might be used in this case.I will try this approch and let you know.
