Commitchell
Databricks Employee
Databricks Employee

Hi there,

That's great to hear that you're looking to use Serverless. It's way less overhead and a better use experience than classic compute.

To answer your questions:

  1. Going forward, users should be required to select a budget policy on all new notebooks, jobs, or pipelines. The best solution to avoid them having to manually select a policy is if they only have one available policy, it should be auto-selected by default. The caveat to this is any existing content from before the policy was applied. Our documentation states, "Existing notebooks, jobs, and Lakeflow Spark Declarative Pipelines are not automatically assigned policies after their owners are granted access to a policy. To add a serverless budget policy to an existing asset, you must manually update the asset's serverless budget policy setting in the UI." Let us know if you're experiencing something different!
  2. Currently, Serverless Budget Policies, which facilitate tracking and notifications, is the only capability we have in this area. I have heard limits or throttling is an area we are working on. Hopefully, we'll see something in this space in the upcoming roadmap webinar next week.

 

View solution in original post