Update Databricks App compute

[juanjomendez96]

Hello community!

I have been using for a while the new Databricks feature 'Databricks Apps'. It has been incredible the amount of effort and time we have saved by using Databricks Apps to deploy our dashboards instead of deploying them directly in our cloud. 

Although, I'm still seeing some limitations, like being unable to have more than 20 secrets within an app. But this is not the most annoying one.

The most annoying one is to not being able to change the compute used by the dashboard. This limits a lot the computational power we can have when the application is running. In some situations, we need to work with multiple big files at the same time or retrieve a very big table, which requires a higher computational power to render in a Databricks App.

In our cloud, we could change the instance used and that's it (requires more DevOps efforts). However, with this new feature, I'm not able to control the compute used by the dashboard.

I was wondering if anyone from this community knows more about these missing (and very required) features from Databricks Apps.

Thanks a lot for the responses, I'm very happy to be part of this community!

PS. Don't get me wrong, we are very impressed with this Apps, but we are still wondering if we should move it to our cloud and use a more powerful instance or stick with Databricks Apps and try to optimize the code as much as possible (it is not possible in some cases, so we must be patient in some moments...).

[HariSankar · Accepted solution]

Hey @juanjomendez96 ,

You’ve explained this really well, and yes, what you’re experiencing is currently one of the main limitations of Databricks Apps.

Right now, these apps run on fixed managed compute controlled by Databricks. That means we, as users, don’t get the option to pick or scale the instance type.
The idea behind it is to make apps quick to deploy and lightweight, but it also means they can struggle when you’re working with large datasets or heavy computations.

In your cloud setup, you could easily switch to a bigger instance and get more power, but with Databricks Apps, that control isn’t available yet.

What You Can Try

--> Move heavy logic outside the app. You can run big joins or transformations in a separate job or SQL warehouse and let the app read from the preprocessed Delta table instead.
--> Use SQL warehouses for dashboards. If your app mostly displays query results, you can connect it to a SQL warehouse and scale that up when needed.
--> Optimize your data. Cache or pre-aggregate data so the app only loads what’s needed.

Databricks Apps are great for saving DevOps time and building internal dashboards quickly, but at the moment, they’re better suited for light to medium workloads.
For larger workloads, combining them with a SQL warehouse or an external job is usually the best way to go.

I’d also suggest sending this feedback through your Databricks account team or the feedback portal — several users have already requested configurable compute and higher secret
limits, so more voices definitely help.

Hope that helps clarify things a bit...

[juanjomendez96]

Hello @HariSankar , thanks a lot for the detailed explanation and for the suggestions (I think we will apply some of them to try to improve the user experience, so thanks for that 😉 ), I will try to escalate it and hopefully, we have this feature in a near future.

Thanks a lot!

Juanjo.