All board posts in Databricks Platform Discussions

Azure Databricks Default Package Repository with Azure Key Vault-backed Secret Scope

kfadratek — Tue, 11 Aug 2026 08:19:09 GMT

Hi,

I’m trying to configure Default Python Package Repository in an Azure Databricks workspace and would like to clarify whether the predefined secret scope can be backed by Azure Key Vault.

According to the documentation, Default Package Repository can use the predefined secret scope:

databricks-package-management

with the following keys:

pip-index-url
pip-extra-index-urls
pip-cert

The documentation shows an example using:

databricks secrets create-scope databricks-package-management followed by put-secret commands.

However, I cannot find an explicit statement saying that databricks-package-management must be a Databricks-backed secret scope.

The documentation also describes two types of secret scopes, including Azure Key Vault-backed scopes. An Azure Key Vault-backed scope is read-only from the Databricks side, so the secrets would be created directly in Azure Key Vault rather than using databricks secrets put-secret.

My question is:

Has anyone successfully configured Default Package Repository using databricks-package-management as an Azure Key Vault-backed secret scope?

I have already tested this configuration. The Key Vault-backed scope exists and contains pip-index-url, and Databricks can read the secret through the scope.

However, I cannot specify the secret scope in the Default Package Repositories UI, and I would like to understand whether Databricks automatically resolves the predefined databricks-package-management scope or whether this functionality specifically requires a Databricks-backed scope.

Has anyone tested this configuration and got it working?

Alternatively, does anyone have official information or a technical reference confirming that databricks-package-management must use a Databricks-backed scope and cannot use an Azure Key Vault-backed scope?

Thanks!

Re: Google Drive ingestion pipeline failing – “Google Drive file system is not enabled”

sridhar_dbx — Tue, 11 Aug 2026 07:45:58 GMT

Lakeflow Connect for Google Drive requires Serverless Compute and Unity Catalog Account-level feature flags. Databricks Community / Free Edition runs on a restricted, shared classic compute environment where Lakeflow Connect preview connectors and serverless ingestion infrastructure are disabled.

Re: Google Drive ingestion pipeline failing – “Google Drive file system is not enabled”

ShamenParis — Tue, 11 Aug 2026 07:04:05 GMT

Hi @hj648 , Google authentication is limited on the Databricks free edition.

For more information: https://docs.databricks.com/aws/en/getting-started/free-edition-limitations

Solution: Simplify Genie Agent Instruction Updates Across Multiple Spaces

harisrinivasay — Tue, 11 Aug 2026 05:54:18 GMT

Dear All,

As the number of Genie Agents grows, maintaining consistent instructions across multiple Genie Spaces becomes increasingly cumbersome. Updating the same instructions manually in each space requires significant effort and is both time-consuming and error-prone.

To simplify this process, we have created a notebook-based solution that automatically updates Genie instructions across multiple spaces. To use it, simply replace the following values in the notebook:

Genie Space IDs
Databricks Workspace URL
Personal Access Token (PAT)
Desired Instructions

Once configured, the notebook will automatically apply the same instructions to all specified Genie Spaces, eliminating the need for repetitive manual updates.

This should significantly reduce maintenance effort and ensure consistency across your Genie Agents.

Attached notebook code:

================================start here====================================

import requests
import json

workspace_url = "<your databricks workspace url"
token = "<your PAT token"

headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}

space_ids = [
"<genie space id 1>",
"<genie space id 2>",
"<genie space id n>
]

NEW_INSTRUCTION = """
Your Role: Databricks SQL Analyst.
Rule 1: Use only tables, columns, joins, and logic defined in this Genie space. Never invent any.
Rule 2: Before generating a query, validate all rules.
<Your extended prompt Rules>
"""

for space_id in space_ids:
try:
# Get existing space
url = f"{workspace_url}/api/2.0/genie/spaces/{space_id}"
response = requests.get(
f"{url}?include_serialized_space=true",
headers=headers
)

response.raise_for_status()

space = response.json()

serialized = json.loads(space["serialized_space"])

# Replace instructions
serialized["instructions"]["text_instructions"] = [{
"content": [NEW_INSTRUCTION]
}]

payload = {
"serialized_space": json.dumps(serialized)
}

update_resp = requests.patch(
url,
headers=headers,
json=payload
)

print(
f"{space_id}: {update_resp.status_code}"
)
except requests.exceptions.RequestException as e:
print(f"{space_id}: Request failed - {e}")
except Exception as e:
print(f"{space_id}: Unexpected error - {e}")

=========================================end here================================

Re: Google Drive ingestion pipeline failing – “Google Drive file system is not enabled”

hj648 — Tue, 11 Aug 2026 03:53:24 GMT

Thanks for the suggestion. I’ll check whether this limitation is specific to the Free edition. Appreciate your help!

Re: Requesting free/discounted voucher for Data Engineer Associate certification exam

Sumit_7 — Tue, 11 Aug 2026 03:22:37 GMT

@Rahulkumar65 Vouchers are distributed to the eligible participants after the Learning Festival. Lookout for the next one in October. If your organisation is a partner of Databricks, then check with your admin.

Re: Google Drive ingestion pipeline failing – “Google Drive file system is not enabled”

Sumit_7 — Tue, 11 Aug 2026 03:19:30 GMT

@hj648 First assumption is that may be its the limitation of Free edition. Other members may help here.

Re: Creating Databricks agent

Sumit_7 — Tue, 11 Aug 2026 03:04:08 GMT

@DineshOjha This doc will be useful in proceeding https://docs.databricks.com/aws/en/agents/. Give it a try and let me know.

Creating Databricks agent

DineshOjha — Mon, 10 Aug 2026 22:30:42 GMT

Hi All,

We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new agent.

We are working on a migration project, moving data from Hive to Databricks, so we want to create a data validation agent which can query both hive and databricks.

Can you please guide us on how to proceed?

Google Drive ingestion pipeline failing – “Google Drive file system is not enabled”

hj648 — Mon, 10 Aug 2026 17:55:25 GMT

Hello Databricks Community,

I am using Databricks Free Edition and trying to ingest a CSV file from Google Drive using Data Engineering → Data Ingestion → Ingest data from Google Drive.

The pipeline validation is failing with the following error:

Pipeline validation error
Error while reading file from Google Drive
Google Drive file system is not enabled

I am the workspace admin, but under Settings → Workspace admin, I do not see a Previews option or any option to enable Google Drive/Lakeflow Connect for Google Drive.

I would like to know:

How can I enable the Google Drive file system in Databricks Free Edition?
Is Google Drive ingestion available for Free Edition workspaces?
If it requires account-level enablement, is there any way for a Free Edition workspace admin to enable it?
Is this error related to my Google Drive configuration, or is the Google Drive connector unavailable/not enabled in my workspace?

I am trying to ingest orderscsv.csv from my Google Drive into a Unity Catalog table in my Databricks workspace.

Thank you.

Re: Lakehouse Monitoring Solution

balajij8 — Mon, 10 Aug 2026 17:13:20 GMT

Hi Surya, You can begin with Data Quality Monitoring solution - already present. It has anomaly detection & Data Profiling (formerly known as Lakehouse Monitoring) solutions helping in Statistical Profiling, data quality, snapshot quality monitors, drift detection and time series monitors. More details here

Lakehouse Monitoring Solution

suryaprayaga — Mon, 10 Aug 2026 16:47:04 GMT

I wonder if there is any GitHub that specifically focuses on Databricks' Lakehouse monitoring solution with definitions and codes that can do this.

I want to onboard it for a customer of mine and don't want to write from scratch if one is already present or available.

Any pointers are welcome...

Thanks in advance

Surya

Re: How should schema evolution be handled across silver and gold layers in a medallion architecture

balajij8 — Mon, 10 Aug 2026 16:01:51 GMT

You can skip using traditional schema migration tools like Liquibase, Flyway or Alembic into Delta Lake as they were generally designed for relational databases where state is tracked through sequential DDL scripts.

Silver Layer

New Columns - You can let Spark handle column additions automatically during writes. For standard append or overwrite operations, configure the schema merge option on the Data Frame writer. You can check the cdc merge configuration & use it if feasible
Column Renames - You can treat renames as transformation logic rather than table migrations. Keep Bronze completely immutable. In your Silver transformations, read the incoming Bronze fields and alias them to the standardized Silver naming conventions. If you must physically rename a column on a Delta table without rewriting underlying data files, enable Delta Column Mapping on the Silver table properties which generally turns renames into instant metadata updates.
Dropped Columns - To keep Silver from accumulating dead columns over time, you can avoid using SELECT * in Silver job definitions explicitly project the required column list instead. If you need to physically purge a dropped column from the table metadata without triggering a heavy data rewrite, use Delta Column Mapping allows you to execute column drops as metadata operations if feasible.

Gold Layer

Automatic Column Sync - Configure the on_schema_change setting in your dbt incremental model config blocks or globally in your project file. Setting on_schema_change to sync_all_columns will generally add new columns to Gold and drop removed columns on the next incremental execution. Alternatively, append_new_columns will add incoming fields while preserving historical columns if an upstream field is dropped.
Renames & Deprecations - Handle column renames within your dbt model CTEs by aliasing the updated Silver column back to the expected Gold schema. For dropped columns where downstream BI dashboards still uses, you can supply default NULL values in the dbt SELECT statement before removing the column entirely via sync_all_columns.

Deployment

You can add a schema validation task at the start of your workflow that compares bronze and silver metadata, firing an alert when schema drift occurs to catch unexpected source changes before they hit reports.

How should schema evolution be handled across silver and gold layers in a medallion architecture?

temarych — Mon, 10 Aug 2026 15:15:58 GMT

We run a medallion pipeline on Databricks:

Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)
Silver: PySpark jobs clean and transform bronze data into Delta tables using batch writes and CDC merge
Gold: dbt incremental models aggregate silver into fact and measure tables

AutoLoader handles bronze schema evolution automatically. The problem is silver and gold.

When a source adds or renames a column, we currently handle it manually - authoring ALTER TABLE statements for silver Delta tables and running dbt run --full-refresh for gold. This works but is ad-hoc and error-prone, especially for column renames.

Our most frequent case is new columns being added by the source. We also care about dropped columns - we don't want dead columns accumulating in silver and gold tables over time.

Questions:

Is there a recommended pattern for managing silver Delta table schema changes in a DAB-deployed pipeline - beyond manual ALTER TABLE?
Should we introduce a schema migration tool like Alembic/Liquibase/Flyway for Delta tables?

Re: Synced table pipeline fails with permission denied for database

amirabedhiafi — Mon, 10 Aug 2026 11:21:48 GMT

Hello @KhaturiabPreet !

I like you analysis it made me do some reading this morning to freshen up my memory.

I checked the the doc and I can say that databricks_writer_<dbid> being unable to CONNECT does not look like an expected state because it is the system role used to create and manage synced tables and synced tables themselves are owned by that role. .

The doc does not say that users need to manually grant CONNECT to databricks_writer_<dbid> as part of synced table setup since its prerequisites concern the Lakebase project, UC source and permissions create_database_objects_if_missing = true is also explicitly supported when creating the synced tablehttps://docs.databricks.com/aws/en/oltp/projects/sync-tables

Therefore, given that the database was created through the Lakebase UI and every synced table failed until the writer privilege was corrected this looks much more like provisioning or bug than a skipped setup step.

Another thingGRANT CONNECT explains and fixes the failure but I would not consider manual modification of the system role the ideal long term configuration procedure as standard PostgreSQL GRANT commands are the documented mechanism for Lakebase database privileges including database privileges. however databricks_writer_<dbid> is categorized as a system managed role used by internal services.

so this:

GRANT CONNECT ON DATABASE <database_name>
TO databricks_writer_<dbid>;

is a technically correct remediation in my opinion as demonstrated by all three pipelines immediately succeeding but I would open a DBKS support case and ask them to confirm whether that database was incorrectly provisioned.

There is another strong point supporting your case which is the documented behavior says changing the pipeline run as identity does not reassign synced table ownership so recreating under different user or SP identities would not fix an ACL problem on databricks_writer_<dbid>.

and yes checking effective CONNECT is a sensible temporary readiness check especially for older or existing databases but I would avoid parsing pg_database.datacl directly and yoy have has_database_privilege() specifically for checking database connection permissions.

For example:

SELECT
    has_database_privilege(
        'databricks_writer_<dbid>',
        '<database_name>',
        'CONNECT'
    ) AS writer_can_connect;

you could check both relevant database privileges:

SELECT
    has_database_privilege(
        'databricks_writer_<dbid>',
        '<database_name>',
        'CONNECT'
    ) AS writer_can_connect,
    has_database_privilege(
        'databricks_writer_<dbid>',
        '<database_name>',
        'CREATE'
    ) AS writer_can_create;

This is preferable to inspecting pg_database.datacl because it tests the effective permission including inherited or public privileges rather than requiring your automation to interpret PostgreSQL ACL strings.

I didn't find anything in the doc until now a documented lakebase management API that exposes effective PostgreSQL database grants directly.

The lakebase Postgres REST API is primarily for infrastructure or resource management and DBKS distinguishes that API from actual database access. So I think for an automated readiness test today executing has_database_privilege() over a PostgreSQL connection is the cleaner supported approach.

Re: Databricks Support #00984257 - Password Reset Email is not working for me

Mubin4all — Mon, 10 Aug 2026 10:08:16 GMT

Issue Resolved with the help of DBX support team

Re: Databricks Support #00984257 - Password Reset Email is not working for me

amirabedhiafi — Mon, 10 Aug 2026 10:01:33 GMT

Hi @Mubin4all !

I would recommend not creating another Webassessor account as duplicate accounts can make the issue more difficult to resolve.

If the password reset email is not arriving even after checking youe spalm this will most likely need to be handled by the DBKS certification support team as they can check the Webassessor account associated with your login.

Try to check this old thread Solved: webassessor databricks login not working - Databricks Community - 64326

Hope it helps 🙂

Re: Is it safe to expose JWT in Databricks Job?

amirabedhiafi — Mon, 10 Aug 2026 09:45:32 GMT

Hello !

What I can say is that your conclusion is correct since DBKS does not currently expose the identity of the principal that called run-now to the running job through the Jobs API or notebook context and the audit event is the authoritative source but it is not designed for synchronous runtime identity resolution.

I would not pass the Databricks/Entra OAuth JWT access token itself as a job parameter.

DBKS explicitly shows resolved parameters on the job run details page. https://docs.databricks.com/aws/en/jobs/dynamic-value-references more importantly the runNow audit event includes job_parameters, notebook_params, python_params... in its request params and you can check this doc link to better understand the situation https://docs.databricks.com/aws/en/admin/account-settings/audit-logs)

which means something like:

{
  "job_id": 123,
  "job_parameters": {
    "jwt": "eyJhbGciOi..."
  }
}

can potentially make a bearer credential persist in places where you don't want credentials to exist like in job run metadata, audit records, logs or parameter inspection.

So what you need to do is simply treat tokens as secrets and avoid putting credentials directly into notebooks or jobs or logging them.

If that JWT is the token SPN B used to call:

POST /api/2.2/jobs/run-now
Authorization: Bearer <token>

then anybody obtaining it before expiration could potentially replay it with the permissions of SPN B.

Interestingly, what I really find amazing is that does know the caller since the jobs audit event represents an on demand invocation and every audit record has:

user_identity

which DBKS defines as the identity of the user initiating the request.

The problem is that DBKS does not propagate that audit user_identity into the job runtime and the documented dynamic values provide job.id, job.run_id, job.trigger.type, timestamps... but no triggered_by/caller identity https://docs.databricks.com/aws/en/jobs/dynamic-value-references

and unfortunately the system tables are explicitly not intended for realtime monitoring because in DBKS recent events may not appear immediately and Azure diagnostic log delivery isn't a reliable workaround either (auditable events typically arrive within about 15 minutes)

What I would recommend is if the caller identity is required inside the job immediately you can use a purpose built signed identity assertion rather than forwarding the OAuth access token.

for example SPN B (or preferably a trusted issuer) sends:

{
  "caller_assertion": "<signed-JWT>",
  "request_id": "8d6835..."
}

with claims such as:

{
  "sub": "SPN-B-application-id",
  "aud": "databricks-job-123",
  "iat": 1786352400,
  "exp": 1786352700,
  "jti": "8d6835..."
}

the job verifies:

signature
aud == expected job
exp < = 5 minutes
jti not previously used
issuer is trusted

so a JWT as a format is not inherently the problem. Passing a bearer/access JWT is.

For a prod security sensitive use case, I would go for :

SPN B
  │
  │ authenticated request
  ▼
thin trigger service or DBKS app
  │
  ├── validates caller = SPN B
  │
  ├── generates request_id
  │
  ├── creates short-lived signed assertion
  │
  ▼
DBKS run-now
  │
  │ caller_assertion + request_id
  ▼
Job running as SPN A
  │
  ├── verifies assertion
  └── knows caller = SPN B

this proxy does not need to be a large additional system since it can be a very thin authenticated endpoint whose only responsibility is identity propagation. A DBKS app is also worth considering because DBKS apps support OAuth for users or SP and explicitly recommend recording the original identity when performing actions on behalf of callers so if immediate caller identification is not required for authorization and is needed only for auditing or reporting, I would avoid all of this complexity and continue using:

SELECT
    event_time,
    user_identity,
    request_params['job_id'] AS job_id
FROM system.access.audit
WHERE service_name = 'jobs'
  AND action_name = 'runNow'

the audit table is actually the correct source of truth user_identity is the initiating identity while run_as is the execution identity.

Re: Building an Agentic HR Front Door on Databricks

kartikchoudhary — Mon, 10 Aug 2026 06:58:04 GMT

Interesting to see the shift from AI assistants that answer questions to agents that can actually complete tasks. For enterprise adoption, I’m curious how teams are thinking about measuring business impact here — is the focus more on reducing manual HR workflows, improving employee experience, or increasing the accuracy and speed of decisions?

Re: What does "Agent-Ready Data Governance" actually mean in production?

kartikchoudhary — Mon, 10 Aug 2026 06:46:50 GMT

One thing I’d be interested in hearing from teams running this across multiple business domains is how they handle business ownership and consistency of definitions. Technical access controls can restrict what an agent can see or invoke, but how are organizations ensuring that the agent interprets metrics and business terms consistently across domains?

Especially when the same metric can have different definitions across finance, sales, or operations, is the business glossary becoming part of the governance process rather than just metadata?