- 1282 Views
- 10 replies
- 8 kudos
Resolved! Is Advanced edition required for Serverless pipeline?
I want to set Serverless pipeline with edition=PRO, but it raises the following error: cannot update pipeline: You must use the Advanced edition when using serverless compute.This is not specified anywhere in the documentation. It only says (see scr...
- 1282 Views
- 10 replies
- 8 kudos
- 8 kudos
Update: Confirmed by Databricks team that it's a documentation gap. Serverless pipelines require advanced pipeline edition.@aayush_410 @Khasim_1 @krishgarikipati @data_pulse
- 8 kudos
- 649 Views
- 2 replies
- 0 kudos
Unable to create Compute
Hello Community Support,I am not able to create the compute resources with my Azure Databricks subscription. Can anyone help?I was on a 14-day free premium trial plan. Now I am on Premium (+ Role-based access controls) plan. My azure account is pay ...
- 649 Views
- 2 replies
- 0 kudos
- 0 kudos
Greetings @sanketpatel, I did some digging and here is what I found. ShamenParis nailed it. This is an Azure subscription quota issue, not a problem with your workspace or Premium plan. From the screenshots, your standardDCASv5Family quota in East US...
- 0 kudos
- 317 Views
- 2 replies
- 0 kudos
Resolved! Genie "Field" error: Table name or alias is required for column 'amount'
The documentation at https://docs.databricks.com/aws/en/genie-agents/tune-quality#define-sql-expressions says that you can create a "Field" in a genie agent (in the Examples section of config) with the code CASE WHEN amount < 10000 THEN 'Small' WHEN ...
- 317 Views
- 2 replies
- 0 kudos
- 0 kudos
Greetings @dkxxx-rc, I did some digging and here is what I found. Vannurswamy Kuruba already nailed the core of this, so I'll build on his reply. On #1: yes, it's a doc inconsistency. On the same page, the Filter example qualifies the column (orders....
- 0 kudos
- 234 Views
- 2 replies
- 3 kudos
Lakeflow Connect SQL Server (integrated CDC pipeline): no way to Include/exclude columns
Setup:- Lakeflow Connect, SQL Server source, `connector_type: CDC`, integrated pipeline (gateway-less — extraction and apply in one triggered pipeline on classic compute)- Deployed with Declarative Automation Bundles, `databricks-bundles` 1.20.0, CLI...
- 234 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Oumeima Just checking in to see whether the explanation above helped, and whether you were able to move forward, either by keeping the gateway-based pipeline for the tables that need include_columns or by projecting the columns downstream. If it...
- 3 kudos
- 417 Views
- 3 replies
- 3 kudos
Resolved! Genie spaces with DABs - how do you deal with the table ordering?
Hi all,I'm about to start managing Genie spaces as code with Asset Bundles and wantedto ask before I build it.From what I'm reading, the genie_spaces resource needs the tables in itsdata_sources to already exist when you deploy. That's fine for an en...
- 417 Views
- 3 replies
- 3 kudos
- 3 kudos
Short answer: you can keep everything in one bundle and one databricks bundle deploy. Add a job_runs resource that runs the job that builds the tables, and reference that run from the Genie agent so the CLI creates the agent last. On a redeploy the b...
- 3 kudos
- 5804 Views
- 4 replies
- 1 kudos
Issue when trying to create a Foreign Catalog to a On Prem SQL Server Instance
Hi,We are creating a lakehouse federated connection to our 2016 On Prem SQL Server. This has an instance in place, so we only want and need to connect to this instance. From this connection, we want to create a foreign catalog of a database on the ...
- 5804 Views
- 4 replies
- 1 kudos
- 1 kudos
really appreciate everyone's feedback on this - I took the approach of avoiding the test connection and using only the ip not the secret, whcih included the port. when i created the catalog using a basic script the catolog was created no problem and ...
- 1 kudos
- 301 Views
- 3 replies
- 2 kudos
Supervisor Agent Deployments!
Hi Team,Could someone please help me deploy the Supervisor Agent using Azure DevOps? I would appreciate it if you could guide me through the deployment steps as well.Thank you.
- 301 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @karuppusamy , This refers to Databricks’ managed multi-agent orchestration experience in the Agents UI, specifically Supervisor Agent. For Azure DevOps: Create the Supervisor Agent in Agents > Create Agent > Supervisor Agent and add at least one ...
- 2 kudos
- 275 Views
- 2 replies
- 0 kudos
ServiceNow ingestion failed since October 1st.
I have a data ingestion pipeline ingesting ServiceNow data into databricks. It has been working for several months, but it stopped working since October 1st.From what I see, the pipeline suffered connection time out.However, from the connection, I ca...
- 275 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @scott_kao, I took a look at both internal and external documentation and here is what I found. A working token refresh and an active ServiceNow account are good signs, but they don't rule out a connectivity or ServiceNow API performance proble...
- 0 kudos
- 6607 Views
- 4 replies
- 2 kudos
- 6607 Views
- 4 replies
- 2 kudos
- 2 kudos
Looks like this is finally becoming a platform feature: https://learn.microsoft.com/en-us/azure/databricks/jobs/triggers#add-multiple-triggers-to-a-job
- 2 kudos
- 378 Views
- 2 replies
- 3 kudos
Resolved! Add custom_tags and budget_policy_id support to Declarative Automation Bundle presets
Declarative Automation Bundle presets currently allow common deployment configuration to be reused across resources. For enterprise platform teams, it would be valuable if presets could also support custom_tags and budget_policy_id.This would allow o...
- 378 Views
- 2 replies
- 3 kudos
- 3 kudos
@browsingbeaver I did a quick bundle validation around this:presets.budget_policy_id is not currently supported: strict bundle validation reports it as an unknown field.budget_policy_id is valid at the individual job resource level.Adding it under pr...
- 3 kudos
- 629 Views
- 4 replies
- 3 kudos
Notebook commands hang after restarting a cluster with custom library installed
Hi,I'm experiencing an issue with a Databricks notebook after restarting a cluster that has a custom library installed.Here are the steps I followed:Created a Databricks compute cluster.Attached a notebook to the cluster.Installed a library required ...
- 629 Views
- 4 replies
- 3 kudos
- 3 kudos
Hello @DineshOjha, Short answer: this isn't expected behavior. When terminated compute starts again, Databricks recreates it, reinstalls the configured libraries, and reattaches notebooks. If a library is still pending, fails to install, or conflict...
- 3 kudos
- 523 Views
- 4 replies
- 2 kudos
How does AUTO CDC resolve same-timestamp tie in SEQUENCE BY?
We are using AUTO CDC (APPLY CHANGES INTO) to process change data capture events from an upstream source (via Autoloader ingesting CDC files landed in cloud storage) into our Silver layer tables. Our source occasionally delivers out-of-order events —...
- 523 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @Khasim_1, I took a look at both internal and external documentation and here is what I found. You're in good hands in this thread already. @Brahmareddy's STRUCT suggestion and @data_pulse's test both line up with the documented behavior, and @...
- 2 kudos
- 6869 Views
- 13 replies
- 4 kudos
Databricks Job cluster for continuous run
Hi AllI am having situation where I wanted to run job as continuous trigger by using job cluster, cluster terminating and re-creating in every run within continuous trigger.I just wanted two know if we have any option where I can use same job cluster...
- 6869 Views
- 13 replies
- 4 kudos
- 4 kudos
Managing job clusters with continuous triggers can be challenging, especially when a cluster terminates and must be recreated for every run. Reusing the same cluster could potentially reduce startup delays and improve resource efficiency, depending o...
- 4 kudos
- 402 Views
- 4 replies
- 2 kudos
Propagating SCD2 History from Silver to Gold with AUTO CDC FLOW and Handling Full Refresh
I have a streaming SCD2 table in the Silver layer, and I need to propagate all historical and future SCD2 events into another SCD2 streaming table in the Gold layer.The Gold table is maintained through a DLT pipeline using two AUTO CDC FLOWs:Initial ...
- 402 Views
- 4 replies
- 2 kudos
- 2 kudos
I have made a few errors in my previous response. Here's the corrected version. What's documented about the ordering behavior For SCD Type 1 targets, you can combine one AUTO CDC FROM SNAPSHOT flow with one or more AUTO CDC flows. The snapshot versi...
- 2 kudos
- 654 Views
- 5 replies
- 3 kudos
DLT Pipeline - Overwrite except for one Append table
Hi all,I am a junior engineer and I am working on a use case involving DLT pipelines that read from a csv source at regular intervals and populate tables based upon processing of this. In most cases, I'd like the table contents to be overwritten with...
- 654 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi. Please find the another update on the response below:Use @DP.materialized_view for the overwrite table and an append-only streaming table (@DP.table over an Auto Loader read) for the history table. I also need to retract something from my last r...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
