- 1214 Views
- 5 replies
- 1 kudos
Autoloader cleansource option does not take any effect
Hello everyone,I was very keen to try out the Autoloader's new cleanSource option so we can clean up our landing folder easily.However I found out it does not have any effect whatsoever. As I cannot create a support case I am creating this post.A sim...
- 1214 Views
- 5 replies
- 1 kudos
- 2841 Views
- 4 replies
- 1 kudos
Resolved! How to capture dlt pipeline id / name using dynamic value reference
Hi there,I have a usecase where I want to set the dlt pipeline id in the configuration parameters of that dlt pipeline.The way we can use workspace ids or task id in notebook task task_id = {{task.id}}/ {{task.name}} and can save them as parameters a...
- 2841 Views
- 4 replies
- 1 kudos
- 3449 Views
- 1 replies
- 0 kudos
Call SQL Function via API
Background - I created a SQL function with the name schema.function_name, which returns a table, in a notebook, the function works perfectly, however, I want to execute it via API using SQL Endpoint. In API, I got insufficient privileges error, so gr...
- 3449 Views
- 1 replies
- 0 kudos
- 0 kudos
Do you know if API service principal / user has USAGE on the database itself? This seems like the most likely issue based on information on the question. Quick Fix Checklist: Run these commands in order (replace api_user with the actual user from ...
- 0 kudos
- 2405 Views
- 4 replies
- 1 kudos
CDC with Snapshot - next_snapshot_and_version() function
I am trying to use create_auto_cdc_from_snapshot_flow (formerly apply_changes_from_snapshot()) (see: https://docs.databricks.com/aws/en/dlt/cdc#cdc-from-snapshot)I am attempting to do SCD type 2 changes using historic snapshot data.In the first coup...
- 2405 Views
- 4 replies
- 1 kudos
- 1 kudos
I have the same issue - very big limitation of create_auto_cdc_from_snapshot_flow and no solution
- 1 kudos
- 944 Views
- 3 replies
- 0 kudos
how to pass secrets keys using a spark_python_task
Hello community,I was searching a way to pass secrets to spark_python_task. Using a notebook file is easy, it's only to use dbutils.secrets.get(...) but how to do the same thing using a spark_python_task set using serveless compute?Kind regards,
- 944 Views
- 3 replies
- 0 kudos
- 0 kudos
@Renu_ but passing them as spark_env will not work with serverless I guess? See also the limitations on the docs Serverless compute limitations | Databricks on AWS
- 0 kudos
- 334 Views
- 5 replies
- 3 kudos
Resolved! Pass parameters between jobs
Hello I have jobIn that job, it runs a task (GetGid) that executes a notebook and obtains some value using dbutils.jobs.taskValuesSete.g. dbutils.jobs.taskValuesSet(key = "gid", value = gid)As a result, I can use this and pass it to another task for ...
- 334 Views
- 5 replies
- 3 kudos
- 3 kudos
Thanks @Hubert-Dudek and @ilir_nuredini I see this nowI'm setting using:dbutils.jobs.taskValues.Set()passing to the job task using Key - gid; Value - {{tasks.GetGid.values.gid}}Then reading using: pid = dbutils.widgets.get()
- 3 kudos
- 822 Views
- 3 replies
- 0 kudos
To identify deleted Runs in Workflow.Job UI in "system.lakeflow"
Hi,I executed a few runs in a Workflow.Jobs UI. I then deleted some of them. I am seeing the deleted runs in "system.lakeflow.job_run_timeline". How do i know which runs are the deleted ones? Thanks
- 822 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @AlleyCat , Hope you are doing well! The jobs table includes a delete_time column that records the time when the job was deleted by the user. So to identify deleted jobs, you can run a query like the following: SELECT * FROM system.lakeflow.jobs ...
- 0 kudos
- 159 Views
- 2 replies
- 1 kudos
Resolved! SQL Stored Procedures - Notebook to always run the CREATE query
I have a stored procedure that is saved as a query file. I can run it and the proc is created. However I want to take this one step further. I want my notebook to run the query file called sp_Remit.sql so if there is any changes to the proc between t...
- 159 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you. I did find this about an hour after I posted. Thank you Kevin
- 1 kudos
- 100 Views
- 1 replies
- 1 kudos
Databricks to snowflake data load
Hi Team, I’m trying to load data from Databricks into Snowflake using the Snowflake Spark connector. I’m using a generic username and password, but I’m unable to log in using these credentials directly. In the Snowflake UI, I can only log in through ...
- 100 Views
- 1 replies
- 1 kudos
- 1 kudos
@SuMiT1 The recommended method to connect to snowflake from databricks is OAuth with Client Credentials Flow.This method uses a registered Azure AD application to obtain an OAuth token without user interaction.Steps:Register an app in Azure AD and c...
- 1 kudos
- 3203 Views
- 1 replies
- 1 kudos
not able to install library in normal site while in community version it working please help
I am not able to install library in normal version while in community editioin i am able to add libray using compute how to install in normal databricks same as community edition.
- 3203 Views
- 1 replies
- 1 kudos
- 1 kudos
Greetings @ayush667787878 , The difference comes down to the type of compute you’re using. In your standard Databricks workspace, you’re running on Serverless — which doesn’t support installing additional libraries on the cluster. In Community Editi...
- 1 kudos
- 195 Views
- 1 replies
- 1 kudos
When to Use and when Not to Use Liquid Clustering?
Hi everyone,I’m looking for some practical guidance and experiences around when to choose Liquid Clustering versus sticking with traditional partitioning + Z-ordering.From what I’ve gathered so far:For small tables (<10TB), Liquid Clustering gives s...
- 195 Views
- 1 replies
- 1 kudos
- 1 kudos
Greeting @pooja_bhumandla , Thanks for laying out your current understanding — here’s practical guidance, trade-offs, and field rules-of-thumb for choosing between Liquid Clustering and traditional partitioning + Z-ordering, along with gotchas to ...
- 1 kudos
- 3121 Views
- 2 replies
- 0 kudos
Is it possible to turn off the redaction of secrets? Is there a better way to solve this?
As part of our Azure Data Factory pipeline, we utilize Databricks to run some scripts that identify which files we need to load from a certain source. This list of files is then passed back into Azure Data Factory utilizing the Exit status from the n...
- 3121 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, as of today, is it still not possible to unredact secret names?
- 0 kudos
- 102 Views
- 1 replies
- 2 kudos
Resolved! Cannot Login to Databricks Customer Academy (Data Engineering courses)
When I am login to the course it shows me this error.You are not authorized to access https://customer-academy.databricks.com. Please select a platform you can access from the following list.
- 102 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @osamR! If you’re encountering this error while trying to log in to Customer Academy, it may indicate that your account is registered under the Partner Academy. You can try accessing your courses through the Partner Academy.However, if your org...
- 2 kudos
- 214 Views
- 8 replies
- 1 kudos
Unable to run sql alert task using databricks job using service principal
I am trying to run a SQL alert task in data bricks job, Service principal is the run_as as set to the job, and task fails with below error message. I also checked if SPN can be granted permission on SQL alert and SQL query, turns out only user id or...
- 214 Views
- 8 replies
- 1 kudos
- 1 kudos
unfortunately, only these options are available in my UI, is it the new UI?
- 1 kudos
- 172 Views
- 1 replies
- 0 kudos
Resolved! Declative Pipeline: Can pipeline or job be deployed run_as using asset bundle
Hi, I have very interesting scenario. I am trying to use Declarative pipelines for first time. The platform team has made workspace artefacts as devops based deployment [infra as code], meaning, I cannot create compute. I have to create compute with ...
- 172 Views
- 1 replies
- 0 kudos
- 0 kudos
@crami wrote:Hi, I have very interesting scenario. I am trying to use Declarative pipelines for first time. The platform team has made workspace artefacts as devops based deployment [infra as code], meaning, I cannot create compute. I have to create ...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 484 | |
| 349 | |
| 287 |
