Hi @ChristianRRL,Both modes incur cost from DBU/h usage based on your compute configuration. Keeping all other variables same, only way I can think of having a cost comparison is to set up two streams that point to the same dataset with different mod...
Hi @ChristianRRL Autoloader ingests your data incrementally regardless of whether you are on directory listing mode or file notification mode. The key difference lies in how it discovers new files. In directory listing mode, Autoloader queries the cl...
Hi @habyphilipose I believe this is due to the additive nature of the pipeline, designed to only add and update new tables. I can imagine a catastrophic loss if a table would be deleted in production just because it's commented out. So I believe that...
Hi @filipniziol ,I dealt with a large table of about a TB in size with liquid clustering enabled. Even with Liquid Clustering, selects and joins on the clustered columns took longer as the table grew. So I don't think it performs as fast as the table...
Hi Alena, Jobs API has update functionality to be able to do that: https://docs.databricks.com/api/workspace/jobs_21/updateIf for some reason you can’t update your pipeline before you trigger it you can also consider creating a new job with desired c...
