- 66 Views
- 1 replies
- 1 kudos
DLT quota Issue
Hi Community,I am facing a weird problem within my Azure Databricks workspace, I am trying to create and run SDP, but somehow when I try to run more than 1 pipeline in parallel, it gives me an error (pasting the error message below). I currently only...
- 66 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @AyushPaldecha09! This error usually appears due to concurrency limits. If you're already on a Premium tier, you typically shouldn’t be hitting this cap, so the best next step is to open a Databricks Support ticket and request an increase to yo...
- 1 kudos
- 322 Views
- 4 replies
- 0 kudos
Time Travel Error when selecting from materialized view (Azure Databricks)
Hey - running into an error this morning that was brought to my attention via failed refreshes from PowerBI. We have a materialized view that, when queried with the standard pattern of 'select col1 from {schema}.table_name', returns an error of 'Cann...
- 322 Views
- 4 replies
- 0 kudos
- 0 kudos
OUR FIXEventually we learned via DESCRIBE EXTENDED, that the table had lost its Location path in production environment. Querying via the development environment to the production catalog the location path was still visible.Solved the issue (with t...
- 0 kudos
- 175 Views
- 3 replies
- 4 kudos
Workflow scheduling on particular working day of the month in ADB
Hi,I am looking to schedule a workflow to execute on 3rd working day. Working day here would be Mon-Fri of each month. I could not find any direct crontab solution but have created watcher file solution for it. Below code will create a watcher file a...
- 175 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @bianca_unifeye , @Poorva21 ,Thank you for your valuable responses.I have changed the main workflow trigger as file arrival in scheduling. I have scheduled the watcher file code to run daily in weekdays and as soon as the 3rd working day encounter...
- 4 kudos
- 76 Views
- 1 replies
- 0 kudos
Issue Genie API - different responses in UI and via API
Hi community,I created an agent with a genie tool, a wrapper around a GenieAgent connected to my Genie space (GENIE_SPACE_ID) that sends user questions and returns Genie’s textual response. I noticed I get 02 different responses when I post a questio...
- 76 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @D4F , What you’re seeing is normal behavior—and the good news is there are very real, very practical ways to make your Genie-based agent more consistent without resorting to a giant, brittle prompt. Let’s dig in. First, why the UI and API can r...
- 0 kudos
- 3113 Views
- 4 replies
- 3 kudos
Databricks asset bundle occasionally duplicating jobs
Since last year, we have adopted Databricks Asset Bundles for deploying our workflows to the production and staging environments. The tool has proven to be quite effective, and we currently use Azure DevOps Pipelines to automate bundle deployment, tr...
- 3113 Views
- 4 replies
- 3 kudos
- 3314 Views
- 4 replies
- 0 kudos
Delta Tables: Time-To-Live
I have seen somewhere (might have been in a Databricks Tech Talk) a Delta Table feature which allows to specify the "expiration date" of data stored in Delta Tables. Once rows surpass their time-to-live, they are automatically deleted or archived. Do...
- 3314 Views
- 4 replies
- 0 kudos
- 279 Views
- 6 replies
- 1 kudos
AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE when starting SQL Server Ingestion pipeline
Dear Community,I’m seeing a compute error when running a Databricks ingestion pipeline (Lakeflow managed ingestion) on AWS.Cloud : AWSRegion: ap−northeast−2Source: SQL Server ingestion pipelineWhen I start the ingestion pipeline, it fails with the f...
- 279 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi, I'm afraid you cannot edit compute instance type settings for SQL Server ingestion pipelines via the Databricks UI. Such changes can only be made via API.
- 1 kudos
- 367 Views
- 6 replies
- 4 kudos
Resolved! Accessing Databricks data in Salesforce via zero copy
I have uploaded clickstream data as shown belowDo i have to mandatorily share via Delta sharing for values to be exposed in Salesforce ?At the Salesforce end i have confirmed that i have a working connector where i am able to see samples data , but u...
- 367 Views
- 6 replies
- 4 kudos
- 4 kudos
HI Team ,Please help me my task is to connect Databrick and salesforce data cloud with zero copy . where we need databricks data in Salesforce data cloud , also just to mention my databricks workspace + ADLS stoarge is on private end point. any hel...
- 4 kudos
- 83 Views
- 0 replies
- 0 kudos
Mongo Db connector - Connection timeout when trying to connect to AWS Document DB
I am on Databricks Run Time LTE 14.3 Spark 3.5.0 Scala 2.12 and mongodb-spark-connector_2.12:10.2.0. Trying to connect to Document DB using the connector and all I get is a connection timeout. I tried using PyMongo, which works as expected and I can ...
- 83 Views
- 0 replies
- 0 kudos
- 144 Views
- 3 replies
- 1 kudos
Resolved! Issues Creating Genie Space via API Join Specs Are Not Persisted
Hi,I’m experimenting with the new API to create a Genie Space.I’m able to successfully create the space, but the join definitions are not created, even though I’m passing a join_specs object in the same format returned by GET /spaces/{id} for an exis...
- 144 Views
- 3 replies
- 1 kudos
- 1 kudos
@Raman_Unifeye@Raman_Unifeye Thanks for the detailed explanation — that really helps clarify why my join specs weren’t being persisted.Do you know if support for persisting join_specs, sql_snippets, and measures via the API is planned for an upcoming...
- 1 kudos
- 253 Views
- 4 replies
- 3 kudos
IDENTIFIER in SQL Views not supported?
Dear community,We are phasing out the dollar param `${catalog_name}` because it has been deprecated since runtime 15.2.We use this parameter in many queries and should now be replaced by the IDENTIFIER clause.In the query below where we retrieve data...
- 253 Views
- 4 replies
- 3 kudos
- 3 kudos
There are two options you may want to consider:Switch to using SQL UDTFs from views in certain casesFor each session, dynamically recreate the view using CREATE VIEW via EXECUTE IMMEDIATE or via Python string templating:
- 3 kudos
- 91 Views
- 1 replies
- 0 kudos
Is there a supported method to register a custom PySpark DataSource so that it becomes visible in th
Built a custom connector using the PySpark DataSource API (DataSource V2). The connector works programmatically, but it does not appear in the Databricks Ingestion UI (Add Data → Connectors) like the Salesforce connector.Is there a supported method t...
- 91 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @venkatesh557 ,Unfortunately, the answer is no - there isn’t a supported way for you to “register” an arbitrary PySpark DataSource V2 so that it appears as a tile in the Databricks Add data → Connectors (Ingestion) UI right now
- 0 kudos
- 456 Views
- 6 replies
- 6 kudos
Resolved! How can I pass parameters from DABs to something(like notebooks)?
I'm implementing DABs, Jobs, and Notebooks.For configure management, I set parameters on databricks.yml.but I can't get parameters on notebook after executed a job successfully. What I implemented ans Steps to the issue:Created "dev-catalog" on WEB U...
- 456 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @tak0519 I think I found the issue! Don't worry - your DABs configuration looks correct. The problem is actually about how you're verifying the results, not the configuration itself. What's happening In your last comment, you mentioned: "Manuall...
- 6 kudos
- 262 Views
- 6 replies
- 2 kudos
Materialized view always load full table instead of incremental
My delta table are stored at HANA data lake file and I have ETL configured like below@DP.materialized_view(temporary=True) def source(): return spark.read.format("delta").load("/data/source") @dp.materialized_view def sink(): return spark.re...
- 262 Views
- 6 replies
- 2 kudos
- 2 kudos
1 more note that I'm not using Unity Catalog here, not sure if it's relevant
- 2 kudos
- 259 Views
- 3 replies
- 0 kudos
Unable to configure custom compute for DLT pipeline
I am trying to configure cluster for a pipeline like above, However dlt keeps using the small cluster as usual, how to resolve this?
- 259 Views
- 3 replies
- 0 kudos
- 0 kudos
i updated my CLI and deployed the job, still i dont see the clusters updates in pipeline
- 0 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
