Hi all,I have been working with Databricks Asset Bundles (now Declarative Automation Bundles) and kept running into the same friction point: there is no easy way to visually inspect a bundle locally before you deploy.Of course, you can read the YAML,...
I am working with a continuously running Spark Structured Streaming job in Databricks, deployed as a standalone job using continuous trigger mode via Databricks Asset Bundles (DABs).On top of the streaming output table (created via writeStream), I wa...
I think the easiest way to do that is to use DEEP CLONE. However, since the SET MANAGED approach was introduced in DBR 17, wouldn't it be better to first migrate the table as external and then convert it to managed using SET MANAGED? The Databricks a...
Hello @yanchr !Yes if the table can first be registered as a UC external table, then ALTER TABLE ... SET MANAGED is now the better practice than doing a direct DEEP CLONE especially on DBR 17+ / serverless.It is recommended to use SET MANAGED for con...
Hi all,When writing from Tableau Prep to Databricks on Azure, we receive the following error message:[Databricks][Hardy] (52) Error communicating with the service: 403 Interestingly, reading from Databricks works without any issues. The Databricks us...
Hello @Karsten !I don't think it is a read permission issue because Tableau prep write path is different from its read path.Tableau prep writes database output in stages so it generates rows, writes them to a temporary table/staging area then moves t...
We are using DABs to deploy our jobs. DABs have source set to git branch or git tag depending on the environment. Repository is structured in mono repo fashion. We don't use wheels for our modules. Sometimes when the jobs run they "randomly" fail th...
Hello @pepco !I will share with you my personal experience about a very similar behaviour I got like you.If you check DBKS doc you will find that git_source and task source: GIT are not recommended for DAB because local relative paths may not point ...
Hi all,I’m using Databricks Free Edition with a serverless SQL warehouse. I’m the only user in this workspace.Warehouse config:- Type: Serverless SQL- Size: Large- Max clusters: 2Query:SELECTw.workspace_name,ROUND(SUM(u.usage_quantity * COALESCE(lp.p...
Hi @Pradip007, This is expected in the free edition. In the free edition, your queries run on a shared regional serverless pool with a small effective warehouse capacity and best-effort access. Even if you’re the only user in your workspace, you’re s...
I read somewhere that there's a max_concurrent_task_runs property, but can't find it anywhere in the docs. So, how to limit the maximum concurrent tasks run in a job?
Hi @yit337, There isn’t a max_concurrent_task_runs setting in Databricks Jobs. The only setting you get is max_concurrent_runs, which limits how many runs of the same job can be active at once, plus a workspace-wide limit of 2000 concurrent task runs...
Hi, I want to change the cluster I am using. However, when I click on the "Compute" tab on the platform, I get automatically redirected to the "SQL Warehouses" page. I am not able to click and enter the "Compute" page. How can I solve this? Thank you
THIS IS NOT RELATED TO A FREE ACCOUNTI ran into the same issue—there was no option to create general compute, only SQL Warehouse or serverless.Solution:When creating a Databricks resource in the Azure portal, it defaults to a serverless workspace, wh...
Hi there, running into some trouble abstracting job_clusters configurations to improve reusability. At the moment, I have many job yaml files that require the following configuration:What would be the best approach(es) to remove this configuration fr...
Hello @ChristianRRL My doubt about your issue is happening in cluster_definitions.yml because it is not only defining a reusable cluster profile it is also redefining the same jobs that already exist in the individual fleet_*.yml files.Why ? because ...
In the January 2026 release notes, it was announced that: "Pipelines now support queued execution mode, where multiple update requests are automatically queued and executed sequentially instead of failing with conflicts. This simplifies operations fo...
Thank you very much for the detailed response! We unfortunately can't proceed with option 1, as we do require multiple places that can trigger the pipeline (an API call to the parent job, and a direct API call to the pipeline itself). This is due to ...
I am trying to stream my IoT data from azure event hub to databricks. Im running Databricks runtime 17.3 LTS with scala 2.13.
Hi @Areqio ,def getKafkaOptions( env: String, ehNameSpace: String, ehName: String, scopeName: String, kafkaOffset: String, ehConnKey: String, maxOffsetsPerTrigger: String = "50000" ): Map[String, String] = { val connStr = dbutils.sec...
[Sorry for a novice question.]I have multiple tables periodically updated from external sources (including insert, update, or delete). I need to update a target table, which is an outer join from multiple source tables without rewriting it each time....
EnvironmentCloud: AWSCompute: ServerlessTable: a_big_tableTable type: Streaming Table (SDP pipeline)Table size: 641 GB, 6,210 filesLiquid Clustering columns: [event_time, integer_userId]delta.dataSkippingStatsColumns:event_time, integer_userId, integ...
Hello @aonurdemir , I looked into your query and have compiled some helpful tips: I don't have direct access to your workspace internals, so I can't prove this definitively. But what you're seeing is consistent with how Delta's stats-based data skipp...
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
In this context, Elevate New is relevant as a digital content platform that covers technology trends, online platforms, software ecosystems, and modern internet-based solutions. As developers and tech communities continue discussing APIs, cloud ser...
Hi All,Lately I have had issues with my vector search index not syncing.The associated pipeline fails to create with error:failed to resolve flow: '__online_index_view'. com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [DELTA_UNSUPPORTED_...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 565 | |
| 349 | |
| 287 |
