cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Kayla
by • Valued Contributor
  • 1086 Views
  • 3 replies
  • 1 kudos

Resolved! Datadog Installation

Is anyone familiar with installing the Datadog agent on clusters? We're not having much luck.  We honestly might not be having the init script run since we're not seeing it in the log, but we can get just a generic "hellow world" init script to run a...

Data Engineering
Reply
  • 1086 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kayla
Valued Contributor
  • 1 kudos

Responding here with the solution I found. Hopefully it'll help anyone with similar issues.First, the Datadog install script is practically a matryoshka doll- the script creates another script which creates a YAML file.One of the consequences of that...

  • 1 kudos
2 More Replies
erigaud
by • Honored Contributor
  • 1747 Views
  • 4 replies
  • 0 kudos

Pass Dataframe to child job in "Run Job" task

Hello,I have a Job A that runs a Job B, and Job A defines a globalTempView and I would like to somehow access it in the child job. Is that in anyway possible ? Can the same cluster be used for both jobs ? If it is not possible, does someone know of a...

Data Engineering
Reply
  • 1747 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
rahuja
New Contributor III
  • 0 kudos

Hi @ranged_coop Yes, we are using the same job compute for using different workflows. But I think different tasks are like different docker containers so that is why it becomes an issue. It would be nice if you can explain a bit about the approach yo...

  • 0 kudos
3 More Replies
tf32
by • New Contributor II
  • 1482 Views
  • 2 replies
  • 1 kudos

Resolved! ERROR com.databricks.common.client.DatabricksServiceHttpClientException: DEADLINE_EXCEEDED

Hi,I got this error "com.databricks.WorkflowException: com.databricks.common.client.DatabricksServiceHttpClientException: DEADLINE_EXCEEDED" during the run of a job workflow with an interactive cluster, at the start of this. It's a job that has been ...

Data Engineering
Reply
  • 1482 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
tf32
New Contributor II
  • 1 kudos

Yes, subsequent runs have been successful.Thank you for the explanation. 

  • 1 kudos
1 More Replies
Avinash_Narala
by • Contributor
  • 1410 Views
  • 2 replies
  • 2 kudos

Resolved! Databricks AI Assistant Cost Implications

I'm worried about how much the Databricks AI assistant will cost me.I need to understand what I'll be charged for, especially when I give a prompt to the AI Assistant Pane and how it will operate in the background.

Data Engineering
Reply
  • 1410 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Avinash_Narala
Contributor
  • 2 kudos

Is there any token limit? like in response or the prompt we send?

  • 2 kudos
1 More Replies
brian999
by • Contributor
  • 1335 Views
  • 4 replies
  • 2 kudos

Resolved! Managing libraries in workflows with multiple tasks - need to configure a list of libs for all tasks

I have workflows with multiple tasks, each of which need 5 different libraries to run. When I have to update those libraries, I have to go in and make the update in each and every task. So for one workflow I have 20 different places where I have to g...

Data Engineering
Reply
  • 1335 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
brian999
Contributor
  • 2 kudos

Actually I think I found most of a solution here in one of the replies: https://community.databricks.com/t5/administration-architecture/installing-libraries-on-job-clusters/m-p/37365/highlight/true#M245It seems like I only have to define libs for the...

  • 2 kudos
3 More Replies
praful
by • New Contributor
  • 1285 Views
  • 0 replies
  • 0 kudos

Recover Lost Notebook

Hi Team, I was using Databricks community edition for learning purpose. I had an account https://community.cloud.databricks.com/?o=6822095545287159 where I stored all my learning notebooks. Unfortunately, this account suddenly stopped working, and I ...

Data Engineering
Reply
  • 1285 Views
  • 0 replies
  • 0 kudos
Jreco
by • Contributor
  • 2170 Views
  • 2 replies
  • 1 kudos

Resolved! SQLServer Incorrect syntax near the keyword 'WITH'

Hi Mates!I'm trying to get some data from an SQLServer using a query; the query has a WITH statement but I'm getting the following error:raise convert_exception(pyspark.errors.exceptions.connect.SparkConnectGrpcException: (com.microsoft.sqlserver.jdb...

Data Engineering
Reply
  • 2170 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Hi @Jreco ,You need to use prepare query option and then query like below: url = "jdbc:sqlserver://server_name:1433;database=db_name" df = spark.read \ .format("jdbc") \ .option("url", url) \ .option("prepareQuery", "with cte as ( SELECT ...

  • 1 kudos
1 More Replies
gaurav_khanna
by • New Contributor II
  • 5330 Views
  • 4 replies
  • 3 kudos

Resolved! Notebook is not attaching to a cluster, asks to contact your administrator. Completely stumped. Please help.

Data Engineering
Reply
  • 5330 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
BartRJD
New Contributor II
  • 3 kudos

I am having the same issue (Azure Databricks).I have a running compute cluster analytics-compute-cluster running in Single User access mode.  The Event Log for the cluster says the cluster is running and the "Driver is healthy".I have Manage permissi...

  • 3 kudos
3 More Replies
Magesh2798
by • New Contributor II
  • 563 Views
  • 1 replies
  • 0 kudos

Databricks to IDT Connection using Databricks Service Principal

Hello all,I'm trying to create a connection from Databricks to Information Design Tool using access token generated using Databricks Service Principal.While testing the connection I'm getting this error: [Databricks][JDBCDriver](500593) Communication...

Data Engineering
Reply
  • 563 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello all,I'm trying to create a connection from Databricks to Information Design Tool using access token generated using Databricks Service Principal.While testing the connection I'm getting this error: [Databricks][JDBCDriver](500593) Communication...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
leungi
by • Contributor
  • 729 Views
  • 1 replies
  • 0 kudos

Error on ANALYZE TABLE [...] COMPUTE STATISTICS FOR COLUMNS [...]

End goal is to apply OPTIMIZE and ZORDER table.However, one of the columns to be ZORDER doesn't have stats collected.Running ANALYZE generates the error below.QueryANALYZE TABLE <catalog>.<schema>.<table> COMPUTE STATISTICS FOR COLUMNS my_col_1, my_c...

Data Engineering
Reply
  • 729 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
End goal is to apply OPTIMIZE and ZORDER table.However, one of the columns to be ZORDER doesn't have stats collected.Running ANALYZE generates the error below.QueryANALYZE TABLE <catalog>.<schema>.<table> COMPUTE STATISTICS FOR COLUMNS my_col_1, my_c...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
duttong
by • New Contributor III
  • 3089 Views
  • 8 replies
  • 7 kudos

[Errno 11] resource temporarily unavailable

Hi Databricks Community,We faced a strange error today where the error below was returned when a notebook was being run. It only happens on git connected notebooks and on rerun it succeeds. What is the issue? 

duttong_0-1719413522488.png
Data Engineering
Reply
  • 3089 Views
  • 8 replies
  • 7 kudos
View Replies
Latest Reply
Witold
Honored Contributor
  • 7 kudos

Just follow https://status.azuredatabricks.net, there you'll see an active incident in West Europe

  • 7 kudos
7 More Replies
jim12321
by • New Contributor II
  • 2158 Views
  • 3 replies
  • 0 kudos

Databricks CLI how to start a job and pass the parameters?

I try to start a job ID 85218616788189 and pass one parameters 'demo' in Windows Shell.This works:databricks jobs run-now 85218616788189  If I try this one,databricks jobs run-now --json '{"job_id":85218616788189,"notebook_params": {"demo":"parameter...

jim12321_0-1710267172994.png
Data Engineering
Reply
  • 2158 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Husky
New Contributor III
  • 0 kudos

@jim12321 I got the same error with CLI version 0.223.1, but I could resolve it by escaping the quotation marks:databricks jobs run-now --json '{\"job_id\":85218616788189,\"notebook_params\": {\"demo\":\"parameter successful\"}}'Seems to be fixed by ...

  • 0 kudos
2 More Replies
SrinuM
by • New Contributor III
  • 970 Views
  • 4 replies
  • 1 kudos

CLOUD_PROVIDER_LAUNCH_FAILURE (CLOUD_FAILURE) for workflow job with all-purpose cluster

One of our databricks workflow job is failing occasionally with below error, after re-running and working fine without any issue.What is the exact reason for the issue and how can we fix itError:Unexpected failure while waiting for the cluster to be ...

Data Engineering
Reply
  • 970 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
PSR100
New Contributor III
  • 1 kudos

These are cloud provider related errors and we will not have much error details from the error message. Based on the error message and also, that you have enough CPU/VM quota available, I think the issue might be due to the storage creation stage in ...

  • 1 kudos
3 More Replies
RKNutalapati
by • Valued Contributor
  • 1795 Views
  • 3 replies
  • 0 kudos

Jobs API "run now" - How to set task wise parameters

I have a job with multiple tasks like Task1 -> Task2 -> Task3. I am trying to call the job using api "run now". Task details are belowTask1 - It executes a Note Book with some input parametersTask2 - It runs using "ABC.jar", so its a jar based task ...

Data Engineering
Reply
  • 1795 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Harsha777
New Contributor III
  • 0 kudos

Hi,It would be a good feature to pass parameters at task level. We have scenarios where we would like to create a job with multiple tasks (notebook/dbt) and pass parameters at task level.

  • 0 kudos
2 More Replies
hadoan
by • New Contributor II
  • 825 Views
  • 2 replies
  • 1 kudos

How to define DLT table with cyclic reference

 @Dlt.table def table_A(): return ( dlt.read_stream(...) ) @dlt.table def table_join_A_and_C(): df_A = dlt.read_stream(table_A) df_C = dlt.read_stream(table_C) return ( ....df_A.join(df_C) ) @dlt.table def table_C(): return ( ...

Data Engineering
Reply
  • 825 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 1 kudos

Hi @hadoan , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback ...

  • 1 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels