All Data Engineering posts

Re: Unity Catalog service credential get_token rejects api:// scope format — "not a valid URI&q

ShamenParis — Wed, 12 Aug 2026 13:44:15 GMT

Totally understand. Let's hope support comes through with a fix🤞

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Louis_Frolio — Wed, 12 Aug 2026 13:35:01 GMT

Hi @elizeu_reis , great detective work here. Cross-referencing system.billing.usage with system.query.history is exactly the right way to separate real usage from idle burn. I did some digging, and I can confirm most of what you observed. There are two different timeouts getting tangled up in this thread, so let's pull them apart.

The control @Niyojit pointed to (Settings > Compute > Serverless interactive execution timeout) caps how long a single statement can run before it gets cancelled. The default is 2.5 hours (9,000 seconds), admins can change it at the workspace level, and you can override it per notebook with spark.databricks.execution.timeout. It's a good guardrail against runaway queries, but it has nothing to do with idle session termination. That's why your 300 second setting had no effect on the 60 minute INACTIVITY termination. Docs here: https://docs.databricks.com/aws/en/compute/serverless/notebooks

Now, the direct answer to your question: today there is no supported user or admin control for the idle timeout on serverless notebook sessions. The platform manages the session lifecycle. The docs acknowledge that idle termination happens, but they don't publish the value or expose a knob. Your measurement of 60 minutes with termination_reason.code = INACTIVITY matches what others have seen. My colleague @Ashwin_DSA dug into the same question in a related thread, confirmed there's no setting to tune, and used the same billing table approach to verify charges: https://community.databricks.com/t5/data-engineering/serverless-notebook-idle-timeout-is-it-configurable-what-exactly/td-p/151133

As for why the default sits at 60 minutes, I honestly don't know, and I haven't found a published rationale. @RobertMitchel asked whether it's a product design choice or a technical constraint, and I won't guess on the record. What I can say is that on serverless the trade-off isn't startup time, it's session state. Terminating the session tears down your REPL, and every variable, temp view, and cached DataFrame goes with it. A generous idle window has historically been the protection against that.

Which brings me to the piece that changes the math: Automated session restoration for serverless notebooks. Turn it on under Settings > Developer > Experimental features. Databricks snapshots the notebook's Python and Spark state before idle termination, and when you come back a Reconnect banner restores your variables, functions, DataFrames, and temp views. Take a gander at https://www.databricks.com/blog/seamlessly-resume-sessions-serverless-notebooks and https://docs.databricks.com/aws/en/notebooks/notebook-compute. Once losing state stops hurting, terminating early (manually or otherwise) gets a lot easier to live with.

On your second observation, that an open tab kept the session alive past 60 minutes: that matches what others have reported. Termination appears to key off the notebook being disconnected, not off command execution alone. So any internal guidance you write should say "close the tab or hit Terminate," not "just stop running cells."

One caveat on the 99.7% figure. system.billing.usage is the source of truth for what you paid, and system.query.history is good activity context, but the two tables have different purposes and granularity. The comparison is a solid estimate of time with no recorded query, not an exact idle-charge ledger. Still plenty to make your case. References: https://docs.databricks.com/aws/en/admin/system-tables/billing and https://docs.databricks.com/aws/en/admin/usage/system-tables

Here's what I'd do today:

Publish internal guidance to hit Terminate (or close the notebook) when done, paired with the session restoration toggle so nobody fears losing their work.
Keep the execution timeout configured as a guardrail against runaway commands, knowing it won't touch the idle issue.
Keep the billing monitor you built, broken out by notebook, user, SKU, and time window. Your query is a good template for others. (@RobertMitchel made the same point about monitoring.)
Raise the ask through your Databricks account team or a support ticket so the product team can track it. Include your cloud and region, workspace IDs, notebook ID and path, timestamps, SKU, and the observed termination reason. You're not the first to ask for a configurable idle value, and the honest framing is a product and documentation gap: is 60 minutes intentional, and if so, can admins get a supported way to reduce it?

Also, +1 to @Armanillo_RoK's related gotcha: classic Personal Compute defaults auto-termination to 4320 minutes (3 days), so change that one right away. Just keep in mind classic auto-termination is a different compute-management model and doesn't carry over to serverless sessions.

One small caution: the x-databricks-attribution-tags header trick pokes at an internal endpoint. Smart way to investigate, but it's an implementation detail, not a supported API contract, and it could change without notice. I wouldn't build tooling on it.

Net takeaway: your diagnosis is right. The idle timeout on serverless notebooks is fixed at the platform level today, the execution timeout is a separate control, and until a knob exists your best tools are terminate discipline, session restoration, and the billing table.

Regards, Louis

Re: Unexpected behavior of Delta VACUUM – need explanation

Louis_Frolio — Wed, 12 Aug 2026 13:26:03 GMT

Greetings @bricks_2026 , I did some digging and here is what I found.

First, good question, and you've already pulled the right evidence from the table history. This looks like retention semantics rather than a PySpark quoting issue or a logging bug. Let me walk through both questions.

Question 1: why doesn't the history show the 168-hour retention?

When a RETAIN clause actually drives a VACUUM run, the START entry records it as specifiedRetentionMillis next to retentionCheckEnabled and defaultRetentionMillis. You can see an example in this thread: https://community.databricks.com/t5/data-engineering/vacuum-command-runs-without-any-retention-period-even-though-the/m-p/151616

Your START entry has no specifiedRetentionMillis and shows defaultRetentionMillis: 31536000000. That tells us the run used your table's 365-day retention property, not 168 hours. And on current runtimes, that's documented behavior, not a defect. The retention window for VACUUM is now controlled by the delta.deletedFileRetentionDuration table property. This applies in Databricks Runtime 18.0 and above, and for Unity Catalog managed tables it applies as far back as Runtime 13.3 LTS. Since you're using a three-level catalog.schema.table name, that likely covers you. Notice the current VACUUM syntax doesn't even document a RETAIN clause anymore: https://docs.databricks.com/aws/en/sql/language-manual/delta-vacuum and https://docs.databricks.com/aws/en/tables/operations/vacuum

So the answer to "is this a known issue or behavior" is: known behavior, assuming your runtime and table type match the above. Disabling spark.databricks.delta.retentionDurationCheck.enabled removes the safety check, but it doesn't make RETAIN override the table property. I'd confirm your exact runtime and whether the table is managed or external, since that determines which rule applies. One heads up if you ever run this on serverless: that Spark config isn't supported there at all: https://kb.databricks.com/delta/the-deltaretentiondurationcheck-property-is-not-recognized-when-using-serverless-compute

Question 2: why can you still time travel to the old key?

Given the above, the effective retention for your run was 365 days. A key updated several months ago is well inside that window, so its old files were never eligible for deletion, and time travel to those versions still works. Fully consistent, if not what you intended. Reference on retention and time travel: https://docs.databricks.com/aws/en/tables/history

One more thing to check, because it bites people even after retention is fixed: deletion vectors. If delta.enableDeletionVectors is true, an UPDATE marks old rows as deleted in metadata instead of rewriting the Parquet file. The original file stays referenced by the current version, and VACUUM never deletes files the current version references. To physically remove those old values, run REORG TABLE ... APPLY (PURGE) first, then VACUUM after the retention window passes: https://docs.databricks.com/aws/en/sql/language-manual/delta-reorg-table

The 16 candidates versus 0 deleted

Those metrics describe different stages. numFilesToDelete counts paths identified during the scan; numDeletedFiles counts paths where the actual delete call succeeded. Your sizeOfDataToDelete: 0 says the 16 candidates were zero-byte objects, so they weren't your key's data files in any case; they're likely non-data paths or directory markers, and your END entry's numVacuumedDirectories: 54 shows directory cleanup is tracked separately. In the open-source implementation, a path that has already disappeared or whose delete returns false counts as a candidate but not a deletion: https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/commands/VacuumCommand.scala

I'll be candid that I can't pin down those 16 paths from the history alone. A DRY RUN will show you exactly what's eligible, and the KB article on reading these metrics is here: https://kb.databricks.com/delta/track-deleted-files-from-vacuum-in-delta-table-history

What I'd do next

Confirm the runtime version of the job and whether the table is a Unity Catalog managed table (DESCRIBE DETAIL shows this).
Run SHOW TBLPROPERTIES catalog_name.schema_name.table_name and check delta.deletedFileRetentionDuration, delta.enableDeletionVectors, and delta.logRetentionDuration.
If seven days is the intended window, set it deliberately: ALTER TABLE ... SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 7 days').
Run VACUUM catalog_name.schema_name.table_name DRY RUN to inspect the candidate paths, then VACUUM ... FULL without relying on RETAIN.
If deletion vectors are enabled and you need old values physically gone, run REORG TABLE ... APPLY (PURGE) first, then VACUUM once the retention window allows.
If a DRY RUN lists real data files but VACUUM still reports zero deletions, compare those paths against cloud storage and check the compute identity's delete permissions. At that point I'd open a support ticket with both history entries and the DRY RUN output attached.

One caution before shrinking retention: dropping from 365 days to 7 days permanently removes time travel beyond a week once VACUUM runs, and on newer runtimes time travel requests older than the property are blocked outright. Verify no readers, streaming jobs, or recovery requirements need the longer history.

The takeaway: your VACUUM did exactly what the table property told it to do. Set delta.deletedFileRetentionDuration to your intended window, check for deletion vectors, and the behavior should match your expectations.

Regards, Louis

Re: Unity Catalog service credential get_token rejects api:// scope format — "not a valid URI&q

Oumeima — Wed, 12 Aug 2026 13:20:11 GMT

hello @ShamenParis,

sadly we cannot add another app ID URI for internal reasons. I filed for a support ticket and we'll see if they remove this restriction 🙂

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

elizeu_reis — Wed, 12 Aug 2026 12:05:17 GMT

Thanks for the suggestion! That setting is different, though. "Serverless interactive execution timeout" kills a single long-running query, not the idle session. What we're asking about is autotermination_minutes (idle-based auto-termination), which we confirmed is separate: we already have "Serverless interactive execution timeout" set to 300s in both of our workspaces, but a session we left idle still only terminated after 60 min, with termination_reason.code = INACTIVITY. We haven't found an admin control for that idle value on serverless, that's the actual gap we're flagging.

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Armanillo_RoK — Wed, 12 Aug 2026 09:30:44 GMT

That's an interesting find @elizeu_reis , let's see what DBX folks come up back with on that.

Here are my two cents re I see that autotermination_minutes defaults to 60 minutes
-> if one happens to be creating a personal compute (not serverless) in Databricks, the initial setting of termination of the cluster set to 4320 mins and (which equals to 3 days...what a lovely Easter egg :D!) So one needs to keep in mind to change it to something around 20 mins instead right away and once done with their work switch the cluster off best.

Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade

xwu — Wed, 12 Aug 2026 08:51:58 GMT

Hi everyone,

I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentation

Catalog commits | Databricks on AWS

When running the following command on a Managed Iceberg Table (contains VARIANT Type data):

ALTER TABLE xxx
SET TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported');

I get this error:

[DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES] Cannot combine upgrading table xxx to catalog-managed with other property changes. Properties provided: [delta.enableVariantShredding -> true, delta.feature.catalogmanaged -> supported]. SQLSTATE: 429BQ

Even if I set delta.enableVariantShredding = true in a separate command prior to running the upgrade, I still hit the exact same error.

Has anyone run into this issue before, and how did you resolve it?

Thanks in advance!

Re: Disabling Change Tracking and enabling Change Data Capture in SQL Server Lakeflow

balajij8 — Wed, 12 Aug 2026 08:10:22 GMT

Hi Upendra,

When both Change Tracking and Change Data Capture are enabled the connector generally prioritizes Change Tracking. Since CT requires the source table to have a primary key and table did not have one, it causes the pipeline to fail. Even after disabling CT, the same error may continue because the ingestion gateway caches the source database configuration and may take some time to detect changes. The gateway can take up to approximately 6 hours to refresh this information automatically. In addition, the pipeline may have existing streaming state or checkpoints that were established when CT was active.

On the SQL Server side, you can verify that CT has been disabled at both the database and table levels. You can also confirm that CDC is enabled at both the database and table levels. Once this is confirmed, restart the ingestion gateway. However, as the gateway may take time to pick up the updated configuration, the restart alone may not be sufficient to reinitialize the pipeline with CDC.

You need to perform a full refresh on the affected table. It forces the pipeline to re-snapshot the table and re-establish the change-tracking mechanism using the updated source configuration. You can trigger the full refresh from the pipeline UI by selecting Full Refresh for the specific table or use the Pipelines API with the parameters. This full refresh will take longer because the table needs to be re-snapshotted, but once it completes, incremental processing should resume using CDC.

You can try full refresh after switching from CT to CDC in dev. Confirm CT is disabled and CDC is enabled on SQL Server, restart the ingestion gateway, perform a full refresh for the affected table, monitor the pipeline update until it completes and then verify that incremental ingestion resumes using CDC. Switching to CDC can have a higher performance impact on the source database than CT, but it is necessary for a table without a primary key.

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

RobertMitchel — Wed, 12 Aug 2026 07:58:36 GMT

The distinction between serverless startup time and classic cluster provisioning is a good point. If serverless can start almost immediately, a 60-minute idle period can definitely result in unnecessary usage for notebooks that are left open.

I’d be interested to know whether the 60-minute value is mainly a product-design choice for interactive workflows or whether there are technical reasons for keeping the default that high. The observation that an open notebook may remain active even when no cells are running is particularly important from a cost-management perspective.

For teams concerned about unexpected usage, it seems worthwhile to monitor system.billing.usage alongside query history and establish internal guidance for disconnecting idle notebook sessions. A shorter configurable timeout would also give administrators more control over this type of workload.

After spending time investigating usage and billing, something lightweight like rollerballer.io can be a quick browser break.

Re: Disabling Change Tracking and enabling Change Data Capture in SQL Server Lakeflow

Upendra_Dwivedi — Wed, 12 Aug 2026 07:41:46 GMT

Hi Balaji,
Thankyou for responding to my question.

I tried implementing your solution and disabled CT only as CDC and CT both were enabled at database level and table level. Then i restarted gateway and expected it to not throw the same error i was getting previously which is:

Can you please tell why this is happening it should now pick CDC but still it is considering CT even if it is disabled. Please let me know if i misunderstood the error.

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Niyojit — Wed, 12 Aug 2026 07:00:00 GMT

Hi @elizeu_reis ,

You can change the Serverless Cluster timeout by going to Settings → Compute.

I’ve attached a screenshot for reference. I hope this helps!

Thanks,

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

elizeu_reis — Wed, 12 Aug 2026 01:52:20 GMT

Confirmed with an actual serverless session left idle until it hit the timeout on its own:

This INACTIVITY termination only happened because I closed the notebook and reopened it more than an hour later. When I left the notebook open (tab still active, no cells running) for over an hour, the serverless cluster did not auto-terminate after the 60-minute idle window. So the idle timeout appears to only kick in reliably when the notebook itself is closed/disconnected, not simply when there's no command execution while the notebook stays open.

Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

elizeu_reis — Tue, 11 Aug 2026 21:02:00 GMT

We've been digging into serverless interactive compute costs in notebooks, because our internal cost monitoring (cross-referencing system.billing.usage with system.query.history) showed a large share of the spend coming from idle time rather than actual query execution. To understand why, we wanted to inspect the actual cluster configuration behind a serverless notebook session — but the notebook UI doesn't expose a cluster ID anywhere for serverless, so there's no obvious way to query it via API or SDK.

How we found the serverless cluster ID (not exposed in the UI):

The workaround: open your browser's DevTools → Network tab, attach the notebook to Serverless, and run any cell. This triggers a request to the internal notebook/{notebook_id}/command endpoint. In that request's headers, look at x-databricks-attribution-tags — it's a URL-encoded JSON blob, and one of its fields (clusterId) contains the serverless cluster ID generated for that session. With that ID, you can query the cluster programmatically via the SDK:

import os from databricks.sdk import WorkspaceClient from databricks.sdk.service import compute w = WorkspaceClient() cluster_id = "your_serverless_cluster_id" cluster = w.clusters.get(cluster_id) print(cluster.autotermination_minutes)

The actual finding:

Querying the cluster this way, I see that autotermination_minutes defaults to 60 minutes — the same value historically used for classic all-purpose clusters, which take several minutes to spin up (which is why it makes sense not to tear down a classic cluster too quickly).

But serverless compute has the opposite value proposition: near-instant startup, with no provisioning cost for the user to wait through. That should allow for a much more aggressive idle timeout — without the "wait for the cluster to come back up" trade-off that justifies 60 minutes on classic clusters.

In practice, this generates real waste: we cross-referenced system.billing.usage with system.query.history to measure real usage vs. idle time in developer sessions, and in one specific case we found 99.7% of billed time was idle — a user who ran just over 1 minute of actual commands ended up billed for a 6-hour session, because the notebook was left open without ever hitting "Terminate."

When I query interactive serverless sessions for a user directly against the billing table, I can clearly see several sessions lasting exactly 60 minutes.

%sql -- 'Query generated by genie code' WITH prices AS ( SELECT sku_name, usage_unit, price_start_time, COALESCE(price_end_time, date_add(current_date, 1)) AS price_end_time_eff, pricing.default AS unit_price FROM system.billing.list_prices WHERE currency_code = 'USD' ) SELECT u.usage_start_time, u.usage_end_time, ROUND(timestampdiff(SECOND, u.usage_start_time, u.usage_end_time) / 60.0, 2) AS duration_minutes, u.sku_name, ROUND(u.usage_quantity, 4) AS dbu, ROUND(u.usage_quantity * p.unit_price, 4) AS cost_usd, u.usage_metadata.notebook_id AS notebook_id, u.usage_metadata.notebook_path AS notebook_path FROM system.billing.usage u LEFT JOIN prices p ON u.sku_name = p.sku_name AND u.usage_unit = p.usage_unit AND u.usage_end_time BETWEEN p.price_start_time AND p.price_end_time_eff WHERE u.billing_origin_product = 'INTERACTIVE' AND u.identity_metadata.run_as = 'USER_EMAIL' AND u.usage_date = '2026-08-10' AND u.workspace_id IN ('WORKSPACE_ID_1', 'WORKSPACE_ID_2') ORDER BY u.usage_start_time

So, why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Re: Pass variable from one notebook to another

adnan_alvee — Tue, 11 Aug 2026 17:52:31 GMT

Hi @ADB0513

With %run, you dont need to pass anything. %run runs the child inline in the same Python namespace, so the catalog variable set in the main notebook is already visible in the child. Just use it.

# main catalog = "dev_catalog" # %run ./child_notebook # child — catalog is already in scope; no widget, no config spark.sql(f"INSERT INTO `{catalog}`.target_schema.target_table " f"SELECT * FROM `{catalog}`.source_schema.source_table")

Regarding your other concerns:
- Do NOT use spark.conf for this because on serverless a custom key errors outright.
- Use a widget + dbutils.notebook.run() only if the child must be reusable / isolated (a real orchestration step), not for a plain %run helper.

Re: Pass variable from one notebook to another

N_N — Tue, 11 Aug 2026 16:27:12 GMT

@Retired_mod

yes, it would be great to get more details on any of these methods. I am trying to do something similar too. Thank you.

Re: Synced Tables - Partitioned Tables

GabFernandes — Tue, 11 Aug 2026 16:18:16 GMT

Hi @tpde ,

This is expected behavior — synced tables in Lakebase use hash partitioning on the primary key internally for sync pipeline performance (parallel upserts). There's currently no option to create a synced table without partitioning.

However, you shouldn't need per-partition grants. The fix is in how you're granting access. As databricks_superuser, run:

-- Grant schema-level usage first GRANT USAGE ON SCHEMA <your_schema> TO <role>; -- Then grant SELECT on the parent (partitioned) table GRANT SELECT ON <your_synced_table> TO <role>;

In PostgreSQL 11+ (which Lakebase uses), GRANT SELECT on a partitioned parent table automatically propagates to all existing partitions. If your downstream users still can't access it, the issue is likely one of:

Missing USAGE on the schema — without this, the table-level grant is invisible to the role.
Grants were issued before some partitions were created (race condition with the sync pipeline creating new partitions). Fix with:

-- Covers all existing tables in the schema, including partition tables GRANT SELECT ON ALL TABLES IN SCHEMA <your_schema> TO <role>; -- Ensures future partitions (created by the sync pipeline) inherit the grant ALTER DEFAULT PRIVILEGES IN SCHEMA <your_schema> GRANT SELECT ON TABLES TO <role>;

Important note: Synced tables are owned by the internal databricks_writer_<dbid> role (not by you), so only databricks_superuser can issue these grants. Regular users with pg_read_all_data (which databricks_superuser has) bypass partition-level checks entirely, which is why the creator can read it but other roles can't.

The ALTER DEFAULT PRIVILEGES approach is the "set it and forget it" solution — any new partitions created during future syncs will automatically inherit the SELECT grant.

If my answer was helpful, please consider marking it as accepted solution!

Re: Just a beginner in Data Engineer

ghn08 — Tue, 11 Aug 2026 15:30:51 GMT

@DataSax @szymon_dybczak @Rishabh-Pandey

Thank you all for the perfect question and answer. I am looking to transition myself into a Data Engineer role from a Data Analyst(10 years). I currently have experience working in DB2 and Hadoop with basic level of python programming.

Question : Does certification hold more weight to my profile than real time projects ?

I am excited to be part of this great community. Looking forward to get your insights

Synced Tables - Partitioned Tables

tpde — Tue, 11 Aug 2026 14:29:03 GMT

I am using synced tables with Lakebase Postgres and am seeing that all Postgres tables created via synced tables have a partition set based on the primary key I select when making the synced table. This is resulting in our downstream users in Postgres not having access to the table we made due to them not having partition access. Is there a way to denote a primary key without making the Postgres table a partitioned table? We are trying to grant select access on the whole table once instead of needing to maintain partition access.

Re: Disabling Change Tracking and enabling Change Data Capture in SQL Server Lakeflow

balajij8 — Tue, 11 Aug 2026 14:25:24 GMT

You can trigger a full refresh on this table to establish a clean CDC baseline. Pause the ingestion pipeline (and the gateway if running continuous Standard CDC) to prevent state conflicts while updating the source. Next, on the SQL Server instance, ensure CDC is enabled at the database level, then enable CDC on the target table. Because the table has no primary key, make sure the table CDC setup has net change support disabled, as supporting net changes requires a primary key or unique index. Once CDC is active on the table, explicitly disable Change Tracking on that table so Lake flow stops attempting to use it.

Before restarting the pipeline, verify through SQL Server system tables that the table reflects CDC as active and Change Tracking as disabled. Finally, re/start the gateway and pipeline, navigate to the table details within the Lake flow UI, and trigger a Full Refresh for the affected table so it can snapshot the current baseline and begin processing continuous CDC transactions seamlessly. Validate the changes in dev first. More details here

Re: Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found

ShamenParis — Tue, 11 Aug 2026 13:43:23 GMT

Hi @Shubh5073 ,

As per my understanding of the issue you are facing, the error indicates that Databricks cannot locate the Access Connector resource in Azure, rather than a problem with your ADLS permissions (since you already correctly assigned Storage Blob Data Contributor).

The most common cause for this specific "could not be found" error is missing Azure permissions. The identity you are using to log into the Databricks Account Console must have the Reader role on the Access Connector resource in Azure, otherwise Databricks cannot verify its existence (I would say check your access on the Access Connector). Additionally, ensure the Resource ID is copied exactly from the Azure Portal's JSON view, as it is strictly case-sensitive.

To address your configuration question: If you are setting up the root storage for a completely new Metastore, this must be done at the Account level. However, if you are simply creating an External Location for an existing Metastore, you can create the Storage Credential directly within the Workspace UI. You only need the appropriate metastore admin privileges to create the credential, which you can then use to configure your external locations.

For more information:

Manage External Locations: https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/manage-external-locations

Access Connector: https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/azure-managed-identities