topic Re: Why Financial companies doesn't want to use databricks serverless and go with only classic Compu in Data Engineering

Why Financial companies doesn't want to use databricks serverless and go with only classic Compute?

kavithai — Fri, 20 Jun 2025 03:36:28 GMT

Re: Why Financial companies doesn't want to use databricks serverless and go with only classic Compu

Takuya-Omi — Sat, 21 Jun 2025 10:52:31 GMT

Hello.

Each company naturally has its own way of thinking, but for many the chief concern is security. Here, I’d like to look at the issue from a network-security standpoint.

Where the compute actually runs differs between serverless compute and classic compute. With classic compute, workloads run on your own cloud foundation, and you can strengthen the traffic between the compute layer and other resources with network-security features such as Private Link. Serverless compute, on the other hand, runs in cloud infrastructure managed by Databricks rather than by your organization. Because of this difference, some companies may have felt uneasy about security.

That, however, is now a matter of the past. It is now possible to apply Private Link on the network path from serverless compute to your own resources. Azure enabled this capability first, and this month AWS has also announced it, as shown in the releases below.

https://docs.databricks.com/aws/en/release-notes/product/2025/june#configure-privatelink-connections-from-serverless-compute-to-s3-storage-buckets-public-preview

https://docs.databricks.com/aws/en/release-notes/product/2025/june#privatelink-connections-from-serverless-compute-to-resources-in-your-vpc-through-an-nlb-is-in-public-preview

Re: Why Financial companies doesn't want to use databricks serverless and go with only classic Compu

jameshughes — Sat, 21 Jun 2025 14:36:38 GMT

@Takuya-Omi pretty much nailed it here. I work heavily with Health and Life Sciences customers which is another highly-regulated industry that requires much tighter control on the network and security layers as well as the forensic audit capabilities of classic compute in the event of any security incident. Serverless "black-boxes" a lot of things to make it easier for customers to use, but at the cost of decreased visibility and control.

Re: Why Financial companies doesn't want to use databricks serverless and go with only classic Compu

Isi — Sat, 21 Jun 2025 14:40:04 GMT

Hey @kavithai

Sometimes there are limitations in the laws of each country regarding "sharing" data outside private clouds or regions, which make it impossible to transmit data outside of your private networks. This is especially true for banks, which face many architectural limitations due to the need to comply with very strict security requirements to avoid data leaks and issues related to the transmission of data from millions of clients. I've worked with 3 banks, and we all faced the same problem. I've had to work with Spark on their own infrastructure, and it's terrible. The fact that you can use Databricks in many cases is already a huge advantage. 😆
And yet, when I've worked with other clients who have the option of Serverless, I still prefer classic because I have much more control over what gets executed, where, and under what circumstances.

Isi