- 114 Views
- 1 replies
- 2 kudos
Notebooks suddenly fails to retrieve Databricks secrets
At around 5:30 am (UTC+11) this morning, a number of our scheduled serverless notebook jobs started failing when attempting to retrieve Databricks secrets.We are able to retrieve the secrets using the databricks CLI and the jobs are run as a user tha...
- 114 Views
- 1 replies
- 2 kudos
- 2 kudos
me tooBut it looks like there hasn't been any official reply regarding this matter yet.
- 2 kudos
- 68 Views
- 1 replies
- 1 kudos
Resolved! How to stop Databricks retaining widget selection between runs?
I have a Python notebook in Databricks. Within it I have a multiselect widget, which is defined like this:widget_values = spark.sql(f''' SELECT my_column FROM my_table GROUP BY my_column ORDER BY my_column ''') widget_values = widget_values.collect(...
- 68 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @SRJDB , What you’re running into isn’t your Python variable misbehaving—it’s the widget hanging onto its own internal state. A Databricks widget will happily keep whatever value you gave it, per user and per notebook, until you explicitly clea...
- 1 kudos
- 184 Views
- 3 replies
- 4 kudos
Workflow scheduling on particular working day of the month in ADB
Hi,I am looking to schedule a workflow to execute on 3rd working day. Working day here would be Mon-Fri of each month. I could not find any direct crontab solution but have created watcher file solution for it. Below code will create a watcher file a...
- 184 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @bianca_unifeye , @Poorva21 ,Thank you for your valuable responses.I have changed the main workflow trigger as file arrival in scheduling. I have scheduled the watcher file code to run daily in weekdays and as soon as the 3rd working day encounter...
- 4 kudos
- 173 Views
- 3 replies
- 1 kudos
Resolved! Issues Creating Genie Space via API Join Specs Are Not Persisted
Hi,I’m experimenting with the new API to create a Genie Space.I’m able to successfully create the space, but the join definitions are not created, even though I’m passing a join_specs object in the same format returned by GET /spaces/{id} for an exis...
- 173 Views
- 3 replies
- 1 kudos
- 1 kudos
@Raman_Unifeye@Raman_Unifeye Thanks for the detailed explanation — that really helps clarify why my join specs weren’t being persisted.Do you know if support for persisting join_specs, sql_snippets, and measures via the API is planned for an upcoming...
- 1 kudos
- 475 Views
- 6 replies
- 6 kudos
Resolved! How can I pass parameters from DABs to something(like notebooks)?
I'm implementing DABs, Jobs, and Notebooks.For configure management, I set parameters on databricks.yml.but I can't get parameters on notebook after executed a job successfully. What I implemented ans Steps to the issue:Created "dev-catalog" on WEB U...
- 475 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @tak0519 I think I found the issue! Don't worry - your DABs configuration looks correct. The problem is actually about how you're verifying the results, not the configuration itself. What's happening In your last comment, you mentioned: "Manuall...
- 6 kudos
- 253 Views
- 4 replies
- 1 kudos
Troubleshooting Azure Databricks Cluster Pools & spot_bid_max_price Validation Error
Hope you’re doing well!I’m reaching out for some guidance on an issue I’ve encountered while setting up Azure Databricks Cluster Pools to reduce cluster spin-up and scale times for our jobs.Background:To optimize job execution wait times, I’ve create...
- 253 Views
- 4 replies
- 1 kudos
- 1 kudos
Possible reasons:1. Setting spot_bid_max_price = -1 is not accepted by Azure poolsAzure Databricks only accepts:0 → on-demand onlypositive numbers → max spot price-1 is allowed in cluster policies, but not inside pools, so validation never completes....
- 1 kudos
- 172 Views
- 1 replies
- 0 kudos
Can't create a new ETL because of compute (?)
I just create a databricks workspace with GCP with "Use existing cloud account (Storage & compute)" option. I already add a few cluster for my task but when i try to create ETL, i always get this error notification. The file is created on the specifi...
- 172 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @molopocho We need to enable the feature in the workspace. If you don't see the option, then you need to reach out to the accounts team or create a ticket to databricks support team t get it enabled at the workspace level.
- 0 kudos
- 181 Views
- 2 replies
- 0 kudos
Declarative Pipeline Re-Deployment and existing managed tables exception
Hi,I am facing a issue regarding re deployment of declarative pipeline using asset bundle. On first deployment, I am able to run the pipeline successfully. On execution, pipeline, as expected create tables. However, when I try to re-deploy the pipeli...
- 181 Views
- 2 replies
- 0 kudos
- 0 kudos
Managed tables are “owned” by a DLT pipeline. Re-deploying a pipeline that references the same managed tables will fail unless you either:Drop the existing tables firstUse external tables that are not owned by DLTUse a separate development schema/pip...
- 0 kudos
- 153 Views
- 1 replies
- 1 kudos
Resolved! mongodb connector duplicate writes
Hi everyone,Has anyone run into this issue? I’m using the MongoDB Spark Connector on Databricks to expose data from Delta Lake to MongoDB. My workflow is:overwrite the collection (very fast),then create the indexes.Occasionally, I’m seeing duplicates...
- 153 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Aziz,What you’re seeing is an expected behaviour when combining Spark retries with non-idempotent writes.Spark’s write path is task-based and fault-tolerant. If a task fails part-way through writing to MongoDB, Spark will retry that task.From Spar...
- 1 kudos
- 653 Views
- 4 replies
- 5 kudos
How to trigger a Databricks job only after multiple other jobs have completed
We have a use case where Job C should start only after both Job A and Job B have successfully completed.In Airflow, we achieve this using an ExternalTaskSensor to set dependencies across different DAGs.Is there a way to configure something similar in...
- 653 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @dhruvs2 .A Lakeflow Job consists of tasks. The tasks can be things like notebooks or other jobs. If you want to orchestrate many jobs, I'd agree that having a job to do this is your best bet . Then you can setup the dependencies as you require.I...
- 5 kudos
- 129 Views
- 0 replies
- 0 kudos
OBO auth implementation in Streamlit not working
Hello,I am currently trying to implement OBO auth in a streamlit db app but I'm getting the following error message:requests.exceptions.HTTPError: 400 Client Error: PERMISSION_DENIED: User does not have USE CATALOG on Catalog '...'. Config: host=, a...
- 129 Views
- 0 replies
- 0 kudos
- 1874 Views
- 5 replies
- 2 kudos
How to run a python task that uses click for CLI operations
Hey,in my application I am using click to facilitate CLI operations. It works locally, in notebooks, when scripts are run locally, but it fails in Databricks. I defined a task that, as an entrypoint, accepts the file where the click-decorated functio...
- 1874 Views
- 5 replies
- 2 kudos
- 2 kudos
Hey guys,I think I managed to find a workaround. I will leave it here for everyone that is seeking the same answers, including future me.What I did is basically this piece of code:def main(): try: assign_variants(standalone_mode=False) ...
- 2 kudos
- 112 Views
- 1 replies
- 1 kudos
Error when triggering a single job run with a Table update trigger
When I trigger a single job run while having a Table update trigger, I get this error message.It has issues with this parameter: - name: updated_tables default: "{{job.trigger.table_update.updated_tables}}"I want to have the option of having the ta...
- 112 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ssommer-ai ,When you use a table update trigger with a parameter like {{job.trigger.table_update.updated_tables}}, this dynamic parameter only gets populated when the job is triggered by an actual table update event. When you manually click "Run ...
- 1 kudos
- 233 Views
- 3 replies
- 5 kudos
Resolved! How to restrict the values permitted in a job or task parameter?
Hi, apologies if this is a daft question - I'm relatively new to Databricks and still finding my feet!I have a notebook with a parameter set within it via a widget, like this:dbutils.widgets.dropdown("My widget", "A", ["A", "B", "C"]) my_variable = d...
- 233 Views
- 3 replies
- 5 kudos
- 5 kudos
Job/task parameters are free-form strings (or JSON) that get pushed down into tasks; there’s no built‑in way in Jobs to constrain them to an enum list like A/B/C in the UI or API. You can override them at run time, but they’re not validated against t...
- 5 kudos
- 821 Views
- 1 replies
- 0 kudos
Salesforce LakeFlow connect - Deletion Salesforce records
Hello, I am new in databricks and related to data engineering. I am running a POC to sync data between a Salesforce sandbox and Databricks using LakeFlow connect.I already make the connection and i successfully sync data between salesforce and databr...
- 821 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @isai-ds Could you please refer to the document below? https://www.databricks.com/blog/introducing-salesforce-connectors-lakehouse-federation-and-lakeflow-connect https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/salesforce-faq
- 0 kudos