hi everyone, I am a product manager in Amsterdam. I work on Jobs and Databricks SQL. In my spare time, I cook 菱 and send emojis . Looking forward to working with this community!
@iamgoda we are going to look into how to make this faster. There's a poll loop in Databricks Workflows for SQL notebooks (but not for SQL scripts) which causes things to slow down.
I'll try to answer this in the simplest possible way
1. Spark is an imperative programming framework. You tell it what it to do, it does it. DLT is declarative - you describe what you want the datasets to be (i.e. the transforms), and it takes care...