topic Re: Cluster Modes - High Concurrency in Data Engineering

Cluster Modes - High Concurrency

Pat — Mon, 29 Aug 2022 14:25:51 GMT

It took me quite some time to find the option to create a cluster in High Concurrency mode. It was hidden in the new UI.

What should be the way to access the data with TAC?

What is the equivalent mode to work with TAC ?

Does it mean that we are being pushed to use the Unity Catalog? I guess this is the way, but at the moment there are some limitations with Unit Catalog - like DLT is not supported.

The documentation page mentions:

Table ACL only and Passthrough only, both as Legacy, but I cannot see those options in my UI.

https://docs.databricks.com/clusters/configure.html#cluster-mode

It's confusing, both differences in the UI and not up-to-date documentation I guess.

The main question is should I forget about using TAC and go with Unity Catalog with the new deployment?

Re: Cluster Modes - High Concurrency

Prabakar — Wed, 31 Aug 2022 08:42:13 GMT

@Pat Sienkiewicz If your workspace is assigned to a Unity Catalog metastore, High Concurrency clusters are not available. Instead, you use access mode to ensure the integrity of access controls and enforce strong isolation guarantees.

My workspace is not UC assigned and I still have that option.

Re: Cluster Modes - High Concurrency

Pat — Wed, 31 Aug 2022 08:58:03 GMT

@Prabakar Ammeappin thanks for sharing this.

My cluster is not assigned to UC.

This is what I saw in the UI:

I was able to find it changing UI to legacy mode.

It's also confusing that it says High Concurrency mode is deprecated but still we can use it in the old UI.

I am just curious if this is not available in the new UI by default, does it mean we should forget about the High Concurrency clusters and TAC? Which means switching to using UC.

What if I want to use DLT? They are currently not supported when using UC.

Re: Cluster Modes - High Concurrency

karthik_p — Mon, 26 Sep 2022 20:35:06 GMT

@Pat Sienkiewicz Looks these changes are related to new UI scope . as Governance wise databricks is having good road map, they just updated clusters or access modes to sync up with Unity catalog. once you disable UI Preview mode to legacy, you should be able to see

Re: Cluster Modes - High Concurrency

Pat — Fri, 30 Sep 2022 07:09:03 GMT

I thought @Prabakar Ammeappin answered the question here on the UI change, but I cannot see it anymore (?).

He mentioned the documentation page (https://docs.databricks.com/clusters/cluster-ui-preview.html)

I was looking for this explanation:

Re: Cluster Modes - High Concurrency

Prabakar — Sat, 01 Oct 2022 10:17:04 GMT

@Pat Sienkiewicz I wonder why I am unable to see the post here. I am checking this internally on how the post got deleted.

Re: Cluster Modes - High Concurrency

Pat — Sat, 01 Oct 2022 10:19:06 GMT

you helped not only me to find this documentation page.

Re: Cluster Modes - High Concurrency

Prabakar — Sun, 02 Oct 2022 11:13:55 GMT

Thanks. Always happy to help.