Databricks Community

Hi,Could you please share the error message or any logs you’re seeing when the task gets stuck? This will help in diagnosing the issue and identifying potential solutions.

A typical recommendation is to not do any transformations as the data lands into the bronze layer (ELT). The idea is that you want your bronze layer to be as close of a representation of your source data as possible so if there are any mistakes later...

issue resolved:for .py, i was using spark, and I have to explicitly create the spark session so that it can be run properly and insert data. 

There isn't a direct way to achieve this within the current DLT framework. When a DLT table is undeclared, it is designed to be removed from the pipeline, which includes the underlying data. However, there are a few strategies you can consider to spe...

So, what use would it be in a situation where I have a Docker image that runs as a job using Databricks Compute. Here the Job has "Run As" which is set to a service principal, say "svc1" which is a databricks managed service principal. I believe that...

Hi @AcrobaticMonkey, How are you triggering the queries using a service principal, is it through a workflow job? If so then you can use Job notifications. I would need more details but one possible solution is to implement Alerts: https://docs.databr...

with the databricks cli you can get all infos about a job run with this command:databricks jobs get-run <run-id> replace <run-id> with your actual run-id

@MuthuLakshmi Thank you very much for the reply! Yes, I have serverless compute enabled on the workspace, but the cluster I am using is not serverless. The error happens whenever access mode is "Single user" (see pictures below). It also happens when...

Hello @Sanjeev, You might want to try this option:   Please refer to below document.You can use cron syntax to schedule jobs. https://docs.databricks.com/en/jobs/scheduled.html

@skarpeck does your input df contain any filters? The empty codes variable could be due to empty microbatches maybe. Please check the numInputRows from your query's Stream Monitoring Metrics. I recommend you to check if there are input rows for the b...

Hello @Syed-SnapLogic, Databricks does not support the password grant type for generating access tokens. The supported grant types for generating access tokens in Databricks are client_credentials and authorization_code. For more information, you can...

Any update on this? I am also facing this issue.

Hello @jeremy98, Yes, it is possible to configure a YAML file to run a workflow only if the previous job run of the same workflow has finished. You can achieve this by defining dependencies between tasks within the workflow. You can specify task depe...