AlexVB
New Contributor III

Thanks for the quick reply. Sorry to hear that. I think that we will quickly grow tired of workflow bloating. We've also considering starting to use Databricks MLOps offering, where MLJobs etc are workflows which will further add noise to the workflows tab. Kinda sad. The UI is kinda shit as it is already.

I got an idea but haven't tried it though, you can do conditional tasks in workflows, do you think one is able to use those to have only parts of a workflow trigger when the workflow is triggered?

I have also considered creating a run table but then it quickly becomes work that I don't want to do and query that to determine what parts of the workflow to run. I really feel like this is an easy feature for Databricks to implement if they just start logging workflow runs (which they already do)