- 1187 Views
- 3 replies
- 3 kudos
Adding dependencies to Serverless compute with concurrency slows processing right down
I am trying to run a job using the For Each command with many concurrent processes using serverless compute.To add dependencies to serverless jobs, it seems you have to add them to the notebook, rather than configure them on the tasks screen like you...
- 1187 Views
- 3 replies
- 3 kudos
- 3258 Views
- 7 replies
- 1 kudos
JDBC Connection query row limit
Anyone know how to increase the amount of rows returned in a JDBC query? Currently we're receiving 1000 rows per query.Have tried adding a LIMIT 5000 to the end of the query, but no luck.
- 3258 Views
- 7 replies
- 1 kudos
- 1 kudos
Thanks all for your help.Looks like the bottleneck is the tool I'm using the make the connection (Appian). It limits JDBC responses to 1000 rows.
- 1 kudos
- 2535 Views
- 3 replies
- 0 kudos
How to Permanently Disable Serverless Compute in Azure Databricks?
Hi,I was wondering how to completely disable serverless compute in Azure Databricks. I am certain that it was disabled in my workspace before, but now it seems to be constantly available at the notebook level.Did Databricks release any recent updates...
- 2535 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey @noorbasha534 , I guess we dont have any feature to enable/disable databricks serverless compute at workspace level. You can confirm this once with your databricks account executive team. They might have a solution for this.
- 0 kudos
- 3833 Views
- 5 replies
- 5 kudos
Resolved! Partitioning vs. Clustering for a 50 TiB Delta Lake Table on Databricks
Hello everyone,I’m planning to create a Delta Lake table on Databricks with an estimated size of ~50 TiB. The table includes three date columns — year, month, and day — and most of my queries will filter on these fields.I’m trying to decide whether t...
- 3833 Views
- 5 replies
- 5 kudos
- 5 kudos
Hey Yutaro,Thank you so much for the kind words—it honestly means a lot! I'm really glad the guidance helped and that you're feeling more confident moving forward. You're doing all the right things by asking the right questions and planning ahead. If...
- 5 kudos
- 11079 Views
- 13 replies
- 9 kudos
Resolved! File information is not passed to trigger job on file arrival
We are using the UC mechanism for triggering jobs on file arrival, as described here: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/file-arrival-triggers.Unfortunately, the trigger doesn't actually pass the file-path that is gener...
- 11079 Views
- 13 replies
- 9 kudos
- 9 kudos
@007 - Review the link https://community.databricks.com/t5/data-engineering/file-arrival-trigger/m-p/94069/highlight/true#M38808
- 9 kudos
- 2252 Views
- 1 replies
- 0 kudos
Resolved! Run failed with termination code: RunExecutionError
I'm getting an error of RunExecutionError with no tasks having run in a notebook. The clusters spin up and then 5 mins later I am getting this error and all cells in the task notebook say cancelled. I don't see any issues with the clusters has they h...
- 2252 Views
- 1 replies
- 0 kudos
- 0 kudos
This was due to a %run notebook command where the cluster could not locate the notebook. I was using a relative to root path from the github repo which did work when running the notebook book on a cluster but did not work when I put it into a job. Ho...
- 0 kudos
- 5887 Views
- 3 replies
- 0 kudos
Resolved! Asset Bundles Email/Notifications Prod ONly
Howdy, I've got a job 'job1' and my dev/stg/prod target in my databricks.yamlCurrently, I have this configurationfor my job:email_notifications: on_success: - me@myorg.com on_failure: - me@myorg.comwebhook_notifications: on_failure: - id: ${var.w...
- 5887 Views
- 3 replies
- 0 kudos
- 0 kudos
Hmmm so something like this:targets: dev: resources: jobs: Workflow1: email_notifications: {} webhook_notifications: {} stage: resources: jobs: Workflow1: email_notifications: on_s...
- 0 kudos
- 14561 Views
- 15 replies
- 11 kudos
- 14561 Views
- 15 replies
- 11 kudos
- 11 kudos
Hi folks - if you're unsure who your account team is and you're interested in the app delegated auth preview, please contact us via aibi-previews [at] databricks [dot] com
- 11 kudos
- 1446 Views
- 1 replies
- 0 kudos
Passing values from a CTE (Common Table Expression) to user-defined functions (UDF) in Spark SQL
Hello everyone, I'm trying to pass a value from a CTE to my function (UDF). Unfortunately, it's not working.Here is the first variant: WITH fx_date_new AS ( SELECT CASE WHEN '2025-01-01' > current_date() THEN CAST(date_format...
- 1446 Views
- 1 replies
- 0 kudos
- 0 kudos
I think the issue is in your subquery. You shouldn't have the entire cte query in parentheses. Only the column from your CTE. Your FROM clause is inside your udf arguments. See if you can use the example below to fix the issue.CREATE OR REPLACE FUNC...
- 0 kudos
- 3682 Views
- 4 replies
- 0 kudos
How to use Iceberg SQL Extensions in a notebook?
I'm trying to use Iceberg's SQL extensions in my Databricks Notebook, but I get a syntax error. Specifically, I'm trying to run 'ALTER TABLE my_iceberg_table WRITE LOCALLY ORDERED BY timestamp;'. This command is listed as part of Iceberg's SQL extens...
- 3682 Views
- 4 replies
- 0 kudos
- 0 kudos
val dfh = spark.sql(s"""CALL glue_catalog.system.create_changelog_view( table => '<>table', options => map('start-snapshot-id', '$startSnapshotId', 'end-snapshot-id', '$endSnapshotId'), changelog_view => table_v)""")lead to ParseException: [PROCED...
- 0 kudos
- 4756 Views
- 3 replies
- 5 kudos
Resolved! How do I use databricks-cli without manual configuration
I want to use databricks cli:databricks clusters listbut this requires a manual step that requires interactive work with the user:databricks configure --tokenIs there a way to use databricks cli without manual intervention so that you can run it as p...
- 4756 Views
- 3 replies
- 5 kudos
- 5 kudos
You can set two environment variables: DATABRICKS_HOST and DATABRICKS_TOKEN, and databricks-cli will use them. See the example of that in the DevOps pipelinesee the full list of environment variables at the end of the Authentication section of docume...
- 5 kudos
- 1691 Views
- 1 replies
- 1 kudos
Query related to Job cluster verses All Purpose Cluster
Currently if run the Notebook with Job Cluster it will show status as Pending for 2 minutes but All Purpose Cluster shows only 6seconds as pending and completes the run quickly. How to improve startup time of Job Cluster to mmatch with startup time ...
- 1691 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Dileep_Karanki Thank you for reaching out and for your question. Here is a link for documentation about the available cluster types: https://docs.databricks.com/aws/en/compute#types-of-compute All-Purpose or Interactive Clusters: These are flex...
- 1 kudos
- 577 Views
- 1 replies
- 1 kudos
Schedule & Triggers
Hi guys, having some issue when I use Schedule and trigger in workflow when file arrives condition, if a place a file with same name it won't trigger is there any configuration I need to do in databricks? thanks
- 577 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Leog Thank you for reaching out and for your sending your question. Per the documentation: https://docs.databricks.com/gcp/en/jobs/file-arrival-triggers Only new files trigger runs. Overwriting an existing file with a file of the same name does ...
- 1 kudos
- 1246 Views
- 4 replies
- 0 kudos
Unable to connect to the cluster
Container launch failure:An unexpected error was encountered while launching containers on worker instances for the cluster. Please retry and contact Databricks if the problem persists.Instance ID: i-02aadc9a7c3532af0Internal error message: Failed to...
- 1246 Views
- 4 replies
- 0 kudos
- 11536 Views
- 8 replies
- 3 kudos
[Azure Databricks] Create an External Location to Microsoft Fabric Lakehouse
Hi,I want to create an external location from Azure Databricks to a Microsoft Fabric Lakehouse, but seems I am missing something.What did I do:I created an "Access Connector for Azure Databricks" in Azure PortalI created a storage credential for the ...
- 11536 Views
- 8 replies
- 3 kudos
- 3 kudos
@stefanberreiter were you able to access Fabric onelake data from databricks unity catalog using Service prinicipal?
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 792 | |
| 553 | |
| 349 | |
| 287 |
