Cluster Modes - High Concurrency

[Pat]

It took me quite some time to find the option to create a cluster in High Concurrency mode. It was hidden in the new UI.

What should be the way to access the data with TAC?

What is the equivalent mode to work with TAC ?

Does it mean that we are being pushed to use the Unity Catalog? I guess this is the way, but at the moment there are some limitations with Unit Catalog - like DLT is not supported.

The documentation page mentions:

Table ACL only and Passthrough only, both as Legacy, but I cannot see those options in my UI.

https://docs.databricks.com/clusters/configure.html#cluster-mode

It's confusing, both differences in the UI and not up-to-date documentation I guess.

The main question is should I forget about using TAC and go with Unity Catalog with the new deployment?

[Prabakar]

@Pat Sienkiewicz​ If your workspace is assigned to a Unity Catalog metastore, High Concurrency clusters are not available. Instead, you use access mode to ensure the integrity of access controls and enforce strong isolation guarantees. 

My workspace is not UC assigned and I still have that option.

[Pat]

@Prabakar Ammeappin​ thanks for sharing this.

My cluster is not assigned to UC.

This is what I saw in the UI:

I was able to find it changing UI to legacy mode.

 

It's also confusing that it says High Concurrency mode is deprecated but still we can use it in the old UI.

I am just curious if this is not available in the new UI by default, does it mean we should forget about the High Concurrency clusters and TAC? Which means switching to using UC.

What if I want to use DLT? They are currently not supported when using UC.

[karthik_p]

@Pat Sienkiewicz​ Looks these changes are related to new UI scope . as Governance wise databricks is having good road map, they just updated clusters or access modes to sync up with Unity catalog. once you disable UI Preview mode to legacy, you should be able to see

[Pat · Accepted solution]

I thought @Prabakar Ammeappin​  answered the question here on the UI change, but I cannot see it anymore (?).

He mentioned the documentation page (https://docs.databricks.com/clusters/cluster-ui-preview.html)

I was looking for this explanation:

[Prabakar]

@Pat Sienkiewicz I wonder why I am unable to see the post here. I am checking this internally on how the post got deleted.

[Pat]

you helped not only me to find this documentation page.

[Prabakar]

Thanks. Always happy to help.