- 3707 Views
- 3 replies
- 0 kudos
How to Permanently Disable Serverless Compute in Azure Databricks?
Hi,I was wondering how to completely disable serverless compute in Azure Databricks. I am certain that it was disabled in my workspace before, but now it seems to be constantly available at the notebook level.Did Databricks release any recent updates...
- 3707 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey @noorbasha534 , I guess we dont have any feature to enable/disable databricks serverless compute at workspace level. You can confirm this once with your databricks account executive team. They might have a solution for this.
- 0 kudos
- 7285 Views
- 5 replies
- 5 kudos
Resolved! Partitioning vs. Clustering for a 50 TiB Delta Lake Table on Databricks
Hello everyone,I’m planning to create a Delta Lake table on Databricks with an estimated size of ~50 TiB. The table includes three date columns — year, month, and day — and most of my queries will filter on these fields.I’m trying to decide whether t...
- 7285 Views
- 5 replies
- 5 kudos
- 5 kudos
Hey Yutaro,Thank you so much for the kind words—it honestly means a lot! I'm really glad the guidance helped and that you're feeling more confident moving forward. You're doing all the right things by asking the right questions and planning ahead. If...
- 5 kudos
- 13972 Views
- 13 replies
- 9 kudos
Resolved! File information is not passed to trigger job on file arrival
We are using the UC mechanism for triggering jobs on file arrival, as described here: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/file-arrival-triggers.Unfortunately, the trigger doesn't actually pass the file-path that is gener...
- 13972 Views
- 13 replies
- 9 kudos
- 9 kudos
@007 - Review the link https://community.databricks.com/t5/data-engineering/file-arrival-trigger/m-p/94069/highlight/true#M38808
- 9 kudos
- 3195 Views
- 1 replies
- 0 kudos
Resolved! Run failed with termination code: RunExecutionError
I'm getting an error of RunExecutionError with no tasks having run in a notebook. The clusters spin up and then 5 mins later I am getting this error and all cells in the task notebook say cancelled. I don't see any issues with the clusters has they h...
- 3195 Views
- 1 replies
- 0 kudos
- 0 kudos
This was due to a %run notebook command where the cluster could not locate the notebook. I was using a relative to root path from the github repo which did work when running the notebook book on a cluster but did not work when I put it into a job. Ho...
- 0 kudos
- 8542 Views
- 3 replies
- 0 kudos
Resolved! Asset Bundles Email/Notifications Prod ONly
Howdy, I've got a job 'job1' and my dev/stg/prod target in my databricks.yamlCurrently, I have this configurationfor my job:email_notifications: on_success: - me@myorg.com on_failure: - me@myorg.comwebhook_notifications: on_failure: - id: ${var.w...
- 8542 Views
- 3 replies
- 0 kudos
- 0 kudos
Hmmm so something like this:targets: dev: resources: jobs: Workflow1: email_notifications: {} webhook_notifications: {} stage: resources: jobs: Workflow1: email_notifications: on_s...
- 0 kudos
- 18315 Views
- 15 replies
- 11 kudos
- 18315 Views
- 15 replies
- 11 kudos
- 11 kudos
Hi folks - if you're unsure who your account team is and you're interested in the app delegated auth preview, please contact us via aibi-previews [at] databricks [dot] com
- 11 kudos
- 2927 Views
- 1 replies
- 0 kudos
Passing values from a CTE (Common Table Expression) to user-defined functions (UDF) in Spark SQL
Hello everyone, I'm trying to pass a value from a CTE to my function (UDF). Unfortunately, it's not working.Here is the first variant: WITH fx_date_new AS ( SELECT CASE WHEN '2025-01-01' > current_date() THEN CAST(date_format...
- 2927 Views
- 1 replies
- 0 kudos
- 0 kudos
I think the issue is in your subquery. You shouldn't have the entire cte query in parentheses. Only the column from your CTE. Your FROM clause is inside your udf arguments. See if you can use the example below to fix the issue.CREATE OR REPLACE FUNC...
- 0 kudos
- 5330 Views
- 4 replies
- 0 kudos
How to use Iceberg SQL Extensions in a notebook?
I'm trying to use Iceberg's SQL extensions in my Databricks Notebook, but I get a syntax error. Specifically, I'm trying to run 'ALTER TABLE my_iceberg_table WRITE LOCALLY ORDERED BY timestamp;'. This command is listed as part of Iceberg's SQL extens...
- 5330 Views
- 4 replies
- 0 kudos
- 0 kudos
val dfh = spark.sql(s"""CALL glue_catalog.system.create_changelog_view( table => '<>table', options => map('start-snapshot-id', '$startSnapshotId', 'end-snapshot-id', '$endSnapshotId'), changelog_view => table_v)""")lead to ParseException: [PROCED...
- 0 kudos
- 5804 Views
- 3 replies
- 5 kudos
Resolved! How do I use databricks-cli without manual configuration
I want to use databricks cli:databricks clusters listbut this requires a manual step that requires interactive work with the user:databricks configure --tokenIs there a way to use databricks cli without manual intervention so that you can run it as p...
- 5804 Views
- 3 replies
- 5 kudos
- 5 kudos
You can set two environment variables: DATABRICKS_HOST and DATABRICKS_TOKEN, and databricks-cli will use them. See the example of that in the DevOps pipelinesee the full list of environment variables at the end of the Authentication section of docume...
- 5 kudos
- 2863 Views
- 1 replies
- 1 kudos
Query related to Job cluster verses All Purpose Cluster
Currently if run the Notebook with Job Cluster it will show status as Pending for 2 minutes but All Purpose Cluster shows only 6seconds as pending and completes the run quickly. How to improve startup time of Job Cluster to mmatch with startup time ...
- 2863 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Dileep_Karanki Thank you for reaching out and for your question. Here is a link for documentation about the available cluster types: https://docs.databricks.com/aws/en/compute#types-of-compute All-Purpose or Interactive Clusters: These are flex...
- 1 kudos
- 988 Views
- 1 replies
- 1 kudos
Schedule & Triggers
Hi guys, having some issue when I use Schedule and trigger in workflow when file arrives condition, if a place a file with same name it won't trigger is there any configuration I need to do in databricks? thanks
- 988 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Leog Thank you for reaching out and for your sending your question. Per the documentation: https://docs.databricks.com/gcp/en/jobs/file-arrival-triggers Only new files trigger runs. Overwriting an existing file with a file of the same name does ...
- 1 kudos
- 1968 Views
- 4 replies
- 0 kudos
Unable to connect to the cluster
Container launch failure:An unexpected error was encountered while launching containers on worker instances for the cluster. Please retry and contact Databricks if the problem persists.Instance ID: i-02aadc9a7c3532af0Internal error message: Failed to...
- 1968 Views
- 4 replies
- 0 kudos
- 16540 Views
- 8 replies
- 3 kudos
[Azure Databricks] Create an External Location to Microsoft Fabric Lakehouse
Hi,I want to create an external location from Azure Databricks to a Microsoft Fabric Lakehouse, but seems I am missing something.What did I do:I created an "Access Connector for Azure Databricks" in Azure PortalI created a storage credential for the ...
- 16540 Views
- 8 replies
- 3 kudos
- 3 kudos
@stefanberreiter were you able to access Fabric onelake data from databricks unity catalog using Service prinicipal?
- 3 kudos
- 2753 Views
- 3 replies
- 4 kudos
Materialized view resources created after creation query failed
Hello, Im testing with Materialized Views and I ran the following query:create materialized view `xx`.`xx`.`stg_sales` PARTITIONED BY (id, external_id, tenant_id) TBLPROPERTIES ('cluster_by' = 'id') as with stg_accounts_events as ( select * from ...
- 2753 Views
- 3 replies
- 4 kudos
- 4 kudos
You are always welcome, Vitor. Do let me know for any queries. Good day.
- 4 kudos
- 1893 Views
- 1 replies
- 0 kudos
How to Configure a Policy in Job Cluster Using Databricks Linked Service in ADF?
I am trying to configure a policy in a job cluster using Databricks Linked Service in Azure Data Factory (ADF). The goal is to restrict access to a specific group or individuals who can use this cluster.I want to know:How to define and enforce a clus...
- 1893 Views
- 1 replies
- 0 kudos
- 0 kudos
In the Linked Services panel in the Azure Data Factory UI, find your Databricks Linked Service and click the squiggly braces {} to edit the service. From here, you'll see a definition in JSON. Create a new key: properties.typeProperties.policyId and ...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
