@filipniziol wrote:Hi @prad18,I have experience working with Databricks on Azure, and here are some considerations regarding your Unity Catalog implementation:Important Consideration:The most important fact on Azure is that you can only have a single...
Yes, too many small Parquet files in a Delta table can degrade write performance by increasing metadata overhead during commits. Regularly running OPTIMIZE helps reduce this impact and improve streaming latency.
You're right to be concerned — this sounds like a classic case of memory or resource leakage over time, which can affect long-running jobs even if metrics look okay on the surface. In triggered DLT (now Lakeflow) pipelines, tasksand state can accumul...
Hi Alex,You're definitely not alone — this is a common issue when using the Spotify API in environments like Databricks or Jupyter Notebooks, where user input through the console (like typing in a prompt) isn't supported.What's happening is that the ...
