Databricks delta sharing design

[noorbasha534]

Dears

I wanted to have a mindshare around delta sharing - how do you decide how many shares to be created and share with other departments if you are maintaining an enterprise wide data warehoouse/lakehouse using Azure Databricks. I see from the documentation that there can be 1000 shares at metastore level ((though this limit is not fixed)).

per request from a department, or per department etc. etc.

I see the access is at a share level, and we cannot further restrict access with a share; so, may be it ends up in one share per request from departments, then we end up in many shares..

appreciate thoughts exchange here...

[Isi]

Hi @noorbasha534 ,

Let me share a bit about our use case and how we’re handling Delta Sharing.

Delta Sharing is indeed a simple and lightweight solution, and one of its main advantages is that it’s free to use. However, it still has several limitations, especially when it comes to performance. The biggest challenge we’ve faced is that it loads the entire table into memory before returning the data, which becomes unsustainable with large datasets or wide tables.

In our case, we had to implement a custom workaround to make it usable, filtering the data at a fine-grained level before sharing, because in some situations you simply can’t afford to wait 7 minutes for a table to load, especially when the consumer has limited resources and the request just times out or fails.

My recomendations:

• Use Delta Sharing only for specific, low-volume scenarios, where response time isn’t critical.

• • In our case, we went with one share per team, as each team owns and controls both what they consume and what they expose.

  • For external clients, it’s always best to create a dedicated share.

    Regarding how many shares to create: that really depends on your use case and data sensitivity.

If the data volume is high:

I strongly recommend not using Delta Sharing for heavy use cases. Instead:

• Create a dedicated Service Principal

• Grant access to specific tables/views

• Make the data available through a SQL Warehouse (Classic or Serverless) depending on your budget and performance needs.

Hope this helps clarify things! 🙂

Isi