Credential passthrough and Hive metastore table access controls are deprecated

[hzh]

Hello,

Based on the recent platform release, the credential passthrough will be deprecated for runtime 15.0 and later. 

Our current setup involves using Databricks alongside AWS Glue and Athena, i.e. registering delta tables in AWS Glue and running other workflows and processes in AWS. If credential passthrough is deprecated, does this mean won't be able to write table to the Hive metastore anymore?

If we migrate hive metastore tables to Unity catalog, is there an alternative method to register UC tables with the Glue catalog?

Migrating all downstream workflows and processes to Databricks simply isn't a feasible option.

[Kaniz]

Hi @hzh, 

• This deprecation specifically affects automatic authentication to S3 buckets from Databricks clusters using the identity you use to log in to Databricks.
• It doesn’t prevent you from writing tables to the Hive metastore. You can still interact with the Hive metastore even without credential passthrough.
• If you decide to migrate your Hive metastore tables to Unity Catalog, there are options:
  • Managed Tables in Unity Catalog:
    • Managed tables are the preferred way to create tables in Unity Catalog. They fully manage the lifecycle, file layout, and storage.
    • These tables always use the Delta table format and reside in a managed storage location reserved for Unity Catalog.
    • You can use CLONE or CREATE TABLE AS SELECT (CTAS) to copy existing Hive tables to Unity Catalog as ...².
  • External Tables in Unity Catalog:
    • External tables in Unity Catalog don’t manage data lifecycle, file layout, or storage location.
    • They’re useful when you need direct access to data using non-Databricks compute (not using Databricks clusters or Databricks SQL warehouses).
    • You can register existing data in Unity Catalog quickly without copying data, as external tables don....
  • If you want to continue using AWS Glue alongside Unity Catalog, consider the following:
    • External Tables in Unity Catalog:
      • Register Unity Catalog external tables with Glue catalog.
      • External tables allow you to maintain compatibility with Glue while benefiting from Unity Catalog features.
      • You won’t need to migrate all workflows to Databricks, as external tables don’t require managed stor....
  • Remember that Unity Catalog simplifies security and governance of your data, providing a central pla....

[hzh]

Thanks Kaniz, 

How to achieve the following under DLT pipeline?

• It doesn’t prevent you from writing tables to the Hive metastore. You can still interact with the Hive metastore even without credential passthrough.

Thanks