- 15 Views
- 2 replies
- 1 kudos
Spark JDBC Write Fails for Record Not Present - PK error
Good afternoon everyone,I’m writing this post to see if anyone has encountered this problem and if there is a way to resolve it or understand why it happens. I’m working in a Databricks Runtime 15.4 LTS environment, which includes Apache Spark 3.5.0 ...
- 15 Views
- 2 replies
- 1 kudos
- 1 kudos
@jorperort When writing to SQL Server tables with composite primary keys from Databricks using JDBC, unique constraint violations are often caused by Spark’s distributed retry logic https://docs.databricks.com/aws/en/archive/connectors/jdbcSolution...
- 1 kudos
- 829 Views
- 3 replies
- 0 kudos
how to pass secrets keys using a spark_python_task
Hello community,I was searching a way to pass secrets to spark_python_task. Using a notebook file is easy, it's only to use dbutils.secrets.get(...) but how to do the same thing using a spark_python_task set using serveless compute?Kind regards,
- 829 Views
- 3 replies
- 0 kudos
- 0 kudos
@Renu_ but passing them as spark_env will not work with serverless I guess? See also the limitations on the docs Serverless compute limitations | Databricks on AWS
- 0 kudos
- 18 Views
- 0 replies
- 0 kudos
VS Code Python file execution
Hi Everyone,I'm using the Databricks VS Code Extension to develop and deploy Asset Bundles. Usually we work with Notebooks and use the "Run File as Workflow" function. Now I'm trying to use pure python file for a new use case and tried to use the "Up...
- 18 Views
- 0 replies
- 0 kudos
- 12 Views
- 0 replies
- 0 kudos
Databricks Workflow
I have a query. I have multiple job (workflow)present in my workspace. Those job runs regularly. Multiple task present in those jobs. Few task having notebook that contain for each code in it. now when a job runs that particular task execute the for ...
- 12 Views
- 0 replies
- 0 kudos
- 119 Views
- 5 replies
- 3 kudos
Resolved! Pass parameters between jobs
Hello I have jobIn that job, it runs a task (GetGid) that executes a notebook and obtains some value using dbutils.jobs.taskValuesSete.g. dbutils.jobs.taskValuesSet(key = "gid", value = gid)As a result, I can use this and pass it to another task for ...
- 119 Views
- 5 replies
- 3 kudos
- 3 kudos
Thanks @Hubert-Dudek and @ilir_nuredini I see this nowI'm setting using:dbutils.jobs.taskValues.Set()passing to the job task using Key - gid; Value - {{tasks.GetGid.values.gid}}Then reading using: pid = dbutils.widgets.get()
- 3 kudos
- 786 Views
- 3 replies
- 0 kudos
To identify deleted Runs in Workflow.Job UI in "system.lakeflow"
Hi,I executed a few runs in a Workflow.Jobs UI. I then deleted some of them. I am seeing the deleted runs in "system.lakeflow.job_run_timeline". How do i know which runs are the deleted ones? Thanks
- 786 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @AlleyCat , Hope you are doing well! The jobs table includes a delete_time column that records the time when the job was deleted by the user. So to identify deleted jobs, you can run a query like the following: SELECT * FROM system.lakeflow.jobs ...
- 0 kudos
- 38 Views
- 2 replies
- 0 kudos
Error trying to edit Job Cluster via Databricks CLI
Good Day all,After having issues with Cloud resources allocated to Lakeflow jobs and Gateways I am trying to apply a policy to the cluster that is allocated to the Job. I am very new to a lot of the databricks platform and the administration so all h...
- 38 Views
- 2 replies
- 0 kudos
- 0 kudos
Good Afternoon Louis,Thank you for the detailed answer. The issue I face is that the default gateway is allocating Virtual CPUs which is not in our Quotas so I need to apply the Compute policy at the creation stage. At this point in the pipelines I c...
- 0 kudos
- 65 Views
- 2 replies
- 1 kudos
Resolved! SQL Stored Procedures - Notebook to always run the CREATE query
I have a stored procedure that is saved as a query file. I can run it and the proc is created. However I want to take this one step further. I want my notebook to run the query file called sp_Remit.sql so if there is any changes to the proc between t...
- 65 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you. I did find this about an hour after I posted. Thank you Kevin
- 1 kudos
- 24 Views
- 1 replies
- 0 kudos
Databricks to snowflake data load
Hi Team, I’m trying to load data from Databricks into Snowflake using the Snowflake Spark connector. I’m using a generic username and password, but I’m unable to log in using these credentials directly. In the Snowflake UI, I can only log in through ...
- 24 Views
- 1 replies
- 0 kudos
- 0 kudos
@SuMiT1 The recommended method to connect to snowflake from databricks is OAuth with Client Credentials Flow.This method uses a registered Azure AD application to obtain an OAuth token without user interaction.Steps:Register an app in Azure AD and c...
- 0 kudos
- 3077 Views
- 1 replies
- 0 kudos
not able to install library in normal site while in community version it working please help
I am not able to install library in normal version while in community editioin i am able to add libray using compute how to install in normal databricks same as community edition.
- 3077 Views
- 1 replies
- 0 kudos
- 0 kudos
Greetings @ayush667787878 , The difference comes down to the type of compute you’re using. In your standard Databricks workspace, you’re running on Serverless — which doesn’t support installing additional libraries on the cluster. In Community Editi...
- 0 kudos
- 32 Views
- 1 replies
- 1 kudos
When to Use and when Not to Use Liquid Clustering?
Hi everyone,I’m looking for some practical guidance and experiences around when to choose Liquid Clustering versus sticking with traditional partitioning + Z-ordering.From what I’ve gathered so far:For small tables (<10TB), Liquid Clustering gives s...
- 32 Views
- 1 replies
- 1 kudos
- 1 kudos
Greeting @pooja_bhumandla , Thanks for laying out your current understanding — here’s practical guidance, trade-offs, and field rules-of-thumb for choosing between Liquid Clustering and traditional partitioning + Z-ordering, along with gotchas to ...
- 1 kudos
- 3087 Views
- 2 replies
- 0 kudos
Is it possible to turn off the redaction of secrets? Is there a better way to solve this?
As part of our Azure Data Factory pipeline, we utilize Databricks to run some scripts that identify which files we need to load from a certain source. This list of files is then passed back into Azure Data Factory utilizing the Exit status from the n...
- 3087 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, as of today, is it still not possible to unredact secret names?
- 0 kudos
- 29 Views
- 1 replies
- 2 kudos
Cannot Login to Databricks Customer Academy (Data Engineering courses)
When I am login to the course it shows me this error.You are not authorized to access https://customer-academy.databricks.com. Please select a platform you can access from the following list.
- 29 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @osamR! If you’re encountering this error while trying to log in to Customer Academy, it may indicate that your account is registered under the Partner Academy. You can try accessing your courses through the Partner Academy.However, if your org...
- 2 kudos
- 70 Views
- 8 replies
- 1 kudos
Unable to run sql alert task using databricks job using service principal
I am trying to run a SQL alert task in data bricks job, Service principal is the run_as as set to the job, and task fails with below error message. I also checked if SPN can be granted permission on SQL alert and SQL query, turns out only user id or...
- 70 Views
- 8 replies
- 1 kudos
- 1 kudos
unfortunately, only these options are available in my UI, is it the new UI?
- 1 kudos
- 38 Views
- 1 replies
- 0 kudos
Declative Pipeline: Can pipeline or job be deployed run_as using asset bundle
Hi, I have very interesting scenario. I am trying to use Declarative pipelines for first time. The platform team has made workspace artefacts as devops based deployment [infra as code], meaning, I cannot create compute. I have to create compute with ...
- 38 Views
- 1 replies
- 0 kudos
- 0 kudos
@crami wrote:Hi, I have very interesting scenario. I am trying to use Declarative pipelines for first time. The platform team has made workspace artefacts as devops based deployment [infra as code], meaning, I cannot create compute. I have to create ...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 479 | |
| 349 | |
| 287 |
