What happens if we change the logic for the delta live tables and we do an incremental update. Does the table get reset (refresh) automatically or would it only apply the logic to new incoming data? would we have to trigger a reset in this case?
That implies that if there is any logic change for complete tables since it is recomputed on every update it is safe to not refresh the pipeline, where as with incremental tables if there is any change to recompute the changes we should refresh the p...
Here is my finding on when to refresh (reset) the table: If it is a complete table all the changes would be apply automatically. If the table is incremental table, you need to do a manually reset (full refresh).
If you have familiarity with Scala you can use Tika. Tika is a wrapper around PDFBox. In case you want to use it in Databricks I suggest you to go through this blog and Git repo. For python based codes you may want to use PyPDF2 as a pandas UDF in S...
@haseebkhan1421 , You can create a temporary view table in your python cell using "DataFrame.createOrReplaceTempView("table_name")" and query that table in SQL. check here for seeing an example
