- 1851 Views
- 8 replies
- 7 kudos
[Errno 11] resource temporarily unavailable
Hi Databricks Community,We faced a strange error today where the error below was returned when a notebook was being run. It only happens on git connected notebooks and on rerun it succeeds. What is the issue?
- 1851 Views
- 8 replies
- 7 kudos
- 7 kudos
Just follow https://status.azuredatabricks.net, there you'll see an active incident in West Europe
- 7 kudos
- 1218 Views
- 3 replies
- 0 kudos
Databricks CLI how to start a job and pass the parameters?
I try to start a job ID 85218616788189 and pass one parameters 'demo' in Windows Shell.This works:databricks jobs run-now 85218616788189 If I try this one,databricks jobs run-now --json '{"job_id":85218616788189,"notebook_params": {"demo":"parameter...
- 1218 Views
- 3 replies
- 0 kudos
- 0 kudos
@jim12321 I got the same error with CLI version 0.223.1, but I could resolve it by escaping the quotation marks:databricks jobs run-now --json '{\"job_id\":85218616788189,\"notebook_params\": {\"demo\":\"parameter successful\"}}'Seems to be fixed by ...
- 0 kudos
- 550 Views
- 4 replies
- 1 kudos
CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE) for workflow job with all-purpose cluster
One of our databricks workflow job is failing occasionally with below error, after re-running and working fine without any issue.What is the exact reason for the issue and how can we fix itError:Unexpected failure while waiting for the cluster to be ...
- 550 Views
- 4 replies
- 1 kudos
- 1 kudos
These are cloud provider related errors and we will not have much error details from the error message. Based on the error message and also, that you have enough CPU/VM quota available, I think the issue might be due to the storage creation stage in ...
- 1 kudos
- 2643 Views
- 4 replies
- 0 kudos
Deploy lock force acquired error when deploying asset bundle using databricks cli
I'm running this command on a DevOps pipeline.databricks bundle deploy -t devI receive this error and have tried using --force-lock but it still doesn't work.Error: deploy lock force acquired by name@company.com at 2024-02-20 16:38:34.99794209 +0000 ...
- 2643 Views
- 4 replies
- 0 kudos
- 1502 Views
- 3 replies
- 0 kudos
Jobs API "run now" - How to set task wise parameters
I have a job with multiple tasks like Task1 -> Task2 -> Task3. I am trying to call the job using api "run now". Task details are belowTask1 - It executes a Note Book with some input parametersTask2 - It runs using "ABC.jar", so its a jar based task ...
- 1502 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,It would be a good feature to pass parameters at task level. We have scenarios where we would like to create a job with multiple tasks (notebook/dbt) and pass parameters at task level.
- 0 kudos
- 508 Views
- 3 replies
- 1 kudos
How to define DLT table with cyclic reference
@Dlt.table def table_A(): return ( dlt.read_stream(...) ) @dlt.table def table_join_A_and_C(): df_A = dlt.read_stream(table_A) df_C = dlt.read_stream(table_C) return ( ....df_A.join(df_C) ) @dlt.table def table_C(): return ( ...
- 508 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @hadoan , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback ...
- 1 kudos
- 389 Views
- 2 replies
- 0 kudos
Uploading wheel using `dbutils.fs.cp` to workspace and install it in Runtime>15
I have been trying to find an alternative to copying a wheel file from my local file system to Databricks and then installing it into the cluster. Doing this databricks_client.dbutils.fs.cp("file:/local..../..whl", "dbfs:/Workspace/users/..../..whl")...
- 389 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @safoineext , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedb...
- 0 kudos
- 334 Views
- 2 replies
- 0 kudos
System Access Column lineage showing inaccurate results
Hi All,I have been trying to leverage the system column lineage table to check the overall journey of a column. But i am getting inaccurate results wherever unpivot transformations are used.Instead of showing the results in a way that 20 columns are ...
- 334 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Mahesh_Yadav , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your fee...
- 0 kudos
- 379 Views
- 2 replies
- 0 kudos
How to get data permissions from Sharepoint and Confluence to Unity Catalog for RAG LLM chatbot
We're implementing a chatbot where documents in SharePoint and pages in Confluence augment the results. We want to adhere to existing RBAC policies in these data sources so that the chatbot doesn't produce results that someone should not see. Are you...
- 379 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @beautrincia , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feed...
- 0 kudos
- 975 Views
- 5 replies
- 3 kudos
How can I preserve the data type of the delta tables while writing to Azure blob storage ?
I am writing a file using this but the data type of columns get changed while reading. df.write.format("com.crealytics.spark.excel").option("header", "true").mode("overwrite").save(path) Due to this I have to manual change every time as I can't chang...
- 975 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Tiwarisk , Thank you for reaching out to our community! We're here to help you.To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...
- 3 kudos
- 10426 Views
- 5 replies
- 3 kudos
Resolved! Executor memory increase limitation based on node type
Hi Databricks community,I'm using Databricks Jobs Cluster to run some jobs. I'm setting the worker and driver type to AWS m6gd.large, which has 2 cores and 8G of memory each.After seeing it's defaulting executor memory to 2G, I wanted to increase it,...
- 10426 Views
- 5 replies
- 3 kudos
- 3 kudos
I think I found the right answer here: https://kb.databricks.com/en_US/clusters/spark-shows-less-memoryIt seems it sets fixed size of ~4GB is used for internal node services. So depending on the node type, `spark.executor.memory` is fixed by Databric...
- 3 kudos
- 610 Views
- 3 replies
- 0 kudos
Delta Live tables stream output to Kafka
Hello,Wanted to Know if we can write the stream output to a Kafka topic in DLT pipeline?Please let me know.Thankyou.
- 610 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi ! Ensure your code is set up to use these libraries. Here is the complete example: Navigate to your cluster configuration:Go to your Databricks workspace.Click on "Clusters" and select your cluster.Go to the "Libraries" tab. Install the necessar...
- 0 kudos
- 3970 Views
- 10 replies
- 6 kudos
access to event_log TVF
Hi, According to the documentation:https://docs.databricks.com/en/delta-live-tables/observability.html"The event_log TVF can be called only by the pipeline owner and a view created over the event_log TVF can be queried only by the pipeline owner. The...
- 3970 Views
- 10 replies
- 6 kudos
- 6 kudos
As per this documentation, https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/unity-catalog, the issue here is documented as a current Limitation:LimitationsThe following are limitations when using Unity Catalog with Delta Live Tabl...
- 6 kudos
- 958 Views
- 4 replies
- 1 kudos
SQL compilation error while connecting to snowflake from Databricks
Hi All,I'm trying to connect to the snowflake database from databricks notebook either to read the data or write the data. However I'm getting an weird error. The code and error are provided as follows, snowflake_table = (spark.read .format("snowfla...
- 958 Views
- 4 replies
- 1 kudos
- 305 Views
- 2 replies
- 0 kudos
Autoloader - Ingestion of CSV files when there is not operation column
Hi,We are working on ingesting multiple files from S3. The files name are fixed based on our source system, Files get replaced frequently with full feed. In DLT when we process new file we have to delete the records processed earlier of the same file...
- 305 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @2vinodhkumar, APPLY CHANGES INTO API with Delta Live Tables (DLT): DLT provides a declarative way to implement CDC. You can use the apply_changes API to identify changes and apply them across tables.Here’s an example of how you can use it in ...
- 0 kudos