Hi Community members,I'm interested in how people monitor the effectiveness of Liquid Clustering as a Delta table grows over time.Consider below scenario:A table initially performs well, but MERGE operations gradually become slower even though the cl...
Setup:- Lakeflow Connect, SQL Server source, `connector_type: CDC`, integrated pipeline (gateway-less — extraction and apply in one triggered pipeline on classic compute)- Deployed with Declarative Automation Bundles, `databricks-bundles` 1.20.0, CLI...
Hi @Oumeima Just checking in to see whether the explanation above helped, and whether you were able to move forward, either by keeping the gateway-based pipeline for the tables that need include_columns or by projecting the columns downstream. If it...
Is the 401 expected behavior, and is it documented anywhere?Is there a supported way for an external system with only a static header to call a Databricks App?I want to receive webhooks from an external SaaS product. The sender can only POST to a pub...
Hi @databrickspooja Yes, the 401 is expected: the Databricks Apps proxy only accepts OAuth access tokens, not PATs, so it rejects the request before your code runs. A static header can't hold an OAuth token either, because M2M tokens expire after abo...
Hi all,I'm about to start managing Genie spaces as code with Asset Bundles and wantedto ask before I build it.From what I'm reading, the genie_spaces resource needs the tables in itsdata_sources to already exist when you deploy. That's fine for an en...
Short answer: you can keep everything in one bundle and one databricks bundle deploy. Add a job_runs resource that runs the job that builds the tables, and reference that run from the Genie agent so the CLI creates the agent last. On a redeploy the b...
I've created a foreign connection to an on prem sql server...CREATE CONNECTION IF NOT EXISTS `dagrbs-dr01` TYPE SQLSERVER OPTIONS ( host = secret('scope-kv-key-vault', 'ip-dagrbs-dr01'), port = '1433', user = secret('scope-kv-key-vault', '...
Hi @DaveCh, The error message is a strong clue here: "Option host must not be empty and should not contain invalid characters" points to the value being resolved bysecret(), not to the server name itself. As per this documentation, the recommended pa...
Hi,We are creating a lakehouse federated connection to our 2016 On Prem SQL Server. This has an instance in place, so we only want and need to connect to this instance. From this connection, we want to create a foreign catalog of a database on the ...
really appreciate everyone's feedback on this - I took the approach of avoiding the test connection and using only the ip not the secret, whcih included the port. when i created the catalog using a basic script the catolog was created no problem and ...
Hi everyone,I’m interested in learning how other teams manage user permissions and access control in Databricks workspaces, especially when multiple users and groups need different levels of access.For example, when working with SQL Warehouses, Unity...
Hello @oliviaanne In my experience, a scalable approach to managing permissions in Databricks is to combine group-based access control, centralized permission management, and regular access reviews rather than managing permissions individually throug...
Hi Team,Could someone please help me deploy the Supervisor Agent using Azure DevOps? I would appreciate it if you could guide me through the deployment steps as well.Thank you.
Hi @karuppusamy , This refers to Databricks’ managed multi-agent orchestration experience in the Agents UI, specifically Supervisor Agent. For Azure DevOps: Create the Supervisor Agent in Agents > Create Agent > Supervisor Agent and add at least one ...
The documentation at https://docs.databricks.com/aws/en/genie-agents/tune-quality#define-sql-expressions says that you can create a "Field" in a genie agent (in the Examples section of config) with the code CASE WHEN amount < 10000 THEN 'Small' WHEN ...
Hi @dkxxx-rc, Greetings. #1 — Documentation inconsistency: Yes, this appears to be an error in the docs. Notice that the Filter example on the same page correctly qualifies the column: orders.amount > 10000, but the Field example leaves amount unqual...
I have a data ingestion pipeline ingesting ServiceNow data into databricks. It has been working for several months, but it stopped working since October 1st.From what I see, the pipeline suffered connection time out.However, from the connection, I ca...
Hello @scott_kao, I took a look at both internal and external documentation and here is what I found. A working token refresh and an active ServiceNow account are good signs, but they don't rule out a connectivity or ServiceNow API performance proble...
Looks like this is finally becoming a platform feature: https://learn.microsoft.com/en-us/azure/databricks/jobs/triggers#add-multiple-triggers-to-a-job
Declarative Automation Bundle presets currently allow common deployment configuration to be reused across resources. For enterprise platform teams, it would be valuable if presets could also support custom_tags and budget_policy_id.This would allow o...
@browsingbeaver I did a quick bundle validation around this:presets.budget_policy_id is not currently supported: strict bundle validation reports it as an unknown field.budget_policy_id is valid at the individual job resource level.Adding it under pr...
Hi!I create a bundle with databricks bundle initand i let create a pipeline including a package. The pipeline definition looks like the following# The main pipeline for pipelinewithpackage resources: pipelines: pipelinewithpackage_etl: n...
Hi - Thank you for your quick response.I checked 1-3: everything seems to be fine:I can build manually and see all files workspace.file_path​contains the filesSo I provide additional information.Folder Treeª .gitignore ª AGENTS.md ª CLAUDE.md ª...
Hi,I'm experiencing an issue with a Databricks notebook after restarting a cluster that has a custom library installed.Here are the steps I followed:Created a Databricks compute cluster.Attached a notebook to the cluster.Installed a library required ...
Hello @DineshOjha, Short answer: this isn't expected behavior. When terminated compute starts again, Databricks recreates it, reinstalls the configured libraries, and reattaches notebooks. If a library is still pending, fails to install, or conflict...
We are using AUTO CDC (APPLY CHANGES INTO) to process change data capture events from an upstream source (via Autoloader ingesting CDC files landed in cloud storage) into our Silver layer tables. Our source occasionally delivers out-of-order events —...
Hello @Khasim_1, I took a look at both internal and external documentation and here is what I found. You're in good hands in this thread already. @Brahmareddy's STRUCT suggestion and @data_pulse's test both line up with the documented behavior, and @...
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
