- 2473 Views
- 3 replies
- 3 kudos
Resolved! How to access the job-Scheduling Date from within the notebook?
I have created a job that contains a notebook that reads a file from Azure Storage. The file-name contains the date of when the file was transferred to the storage. A new file arrives every Monday, and the read-job is scheduled to run every Monday. I...
- 2473 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Karolin Albinsson , Just a friendly follow-up. Do you still need help, or @Hubert Dudek (Customer) 's response help you to find the solution? Please let us know.
- 3 kudos
- 15108 Views
- 10 replies
- 7 kudos
How to make the job fail via code after handling exception
Hi , We are capturing the exception if an error occurs using try except. But we want the job status to be failed once we got the exception. Whats the best way to do that. We are using pyspark.
- 15108 Views
- 10 replies
- 7 kudos
- 7 kudos
you can do some hack arround dbutils = get_dbutils(spark) tables_with_exceptions = [] for table_config in table_configs: try: process(spark, table_config) except Exception as e: exception_detail = f"Error p...
- 7 kudos
- 2350 Views
- 5 replies
- 2 kudos
Job cluster support in jobs/runs/submit API
We are using jobs/runs/submit API of databricks to create and trigger a one-time run with new_cluster and existing_cluster configuration. We would like to check if there is provision to pass "job_clusters" in this API to reuse the same cluster across...
- 2350 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi, Any update on the above mentioned issue? Unable to submit a one time new job run (api/2.0 or 21/jobs/runs/submit) with shared job cluster or one new cluster has to be used for all TASKs in the job
- 2 kudos
- 6534 Views
- 10 replies
- 11 kudos
How to know, who started a job run?
Hi there!We have different jobs/workflows configured in our Databricks workspace running on AWS and would like to know who actually started the job run? Are they started by a user or a service principle using curl?Currently one can only see, who is t...
- 6534 Views
- 10 replies
- 11 kudos
- 11 kudos
Hi,Any update on this feature?With the info that is currently available, we cannot track the user who initiated the job run. The job run shows whether its scheduled or manual run but doesn't provide info on the user who triggered the manual run. "RUN...
- 11 kudos
- 19453 Views
- 3 replies
- 4 kudos
Resolved! How to get the Job ID and Run ID and save into a database
We are having Databricks Job running with main class and JAR file in it. Our JAR file code base is in Scala. Now, when our job starts running, we need to log Job ID and Run ID into the database for future purpose. How can we achieve this?
- 19453 Views
- 3 replies
- 4 kudos
- 4 kudos
That article is for members only, can we also specify here how to do it (for those that are not medium members?). Thanks!
- 4 kudos
- 2657 Views
- 3 replies
- 2 kudos
Dropdown for parameters in a job
I want to be able to denote the type of run from a predetermined list of values that a user can choose from when kicking off a run using different parameters. Our team does standardized job runs on a weekly cadence but can have timeframes that change...
- 2657 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @cmilligan , I have a similar requirement and would really be grateful if you could provide me with any information on how to fix this issue. Thanks a lot!
- 2 kudos
- 2682 Views
- 2 replies
- 0 kudos
Batch Doesn't Exist Failure
I have a job that's been working perfectly fine since I deployed it earlier this month. Last night, however, one of the tasks within the job started failing with the following error:java.lang.IllegalStateException: batch 4 doesn't exist at org.apac...
- 2682 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @JordanYaker, The error message java.lang.IllegalStateException: batch 4 doesn't exist is thrown when Apache Spark™’s Structured Streaming job tries to access a batch that doesn’t exist in the metadata. This can happen for various reasons, such as...
- 0 kudos
- 2126 Views
- 2 replies
- 1 kudos
Resolved! Job ID value out of range - Azure Logic App Connector
Hello everybody,i tried to build a Logic App Custom Connector following this one explanation. (https://medium.com/@poojaanilshinde/create-azure-logic-apps-custom-connector-for-azure-databricks-e51f4524ab27)Now i run in the following Problem and wante...
- 2126 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey Lukas,I can totally relate to the frustration of encountering those confounding errors when building custom connectors in Azure Logic Apps. The "Job ID value out of range" issue can be quite perplexing, but fear not, for there's a solution on the...
- 1 kudos
- 2357 Views
- 2 replies
- 0 kudos
Resolved! The driver is temporarily unavailable
My job fails with Driver is temporarily unavailable. Apparently, it's permanently unavailable, because the job is not pausing but failing.
- 2357 Views
- 2 replies
- 0 kudos
- 0 kudos
I am facing the same issues . I am writing in batches using a simple for loop. I don't have any collect statements inside the loop. I am rewriting the partitions with partition overwrite dynamic mode in a huge wide delta table - several tb. The incr...
- 0 kudos
- 11666 Views
- 10 replies
- 9 kudos
How to setup Job notifications using Microsoft Teams webhook ?
Couple of things I tried:1. I created a webhook connector in msft teams and copied it Notifications destinations via Admin page -> New destination -> from dropdown I selected Microsoft teams -> added webhook url and saved it.outcome: I don't get the ...
- 11666 Views
- 10 replies
- 9 kudos
- 9 kudos
You can set up job notifications for Databricks jobs using Microsoft Teams webhooks by following these steps:Set up a Microsoft Teams webhook:Go to the channel where you want to receive notifications in Microsoft Teams.Click on the "..." icon next to...
- 9 kudos
- 2044 Views
- 3 replies
- 0 kudos
I have a job with multiple tasks running asynchronously and I don't think its leveraging all the nodes on the cluster based on runtime.
I have a job with multiple tasks running asynchronously and I don't think its leveraging all the nodes on the cluster based on runtime. I open the Spark UI for the cluster and checkout the executors and don't see any tasks for my worker nodes. How ca...
- 2044 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Dave Hiltbrand Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 2072 Views
- 1 replies
- 0 kudos
Getting JsonParseException: Unexpected character ('<' (code 60))
I have a scheduled job that is executed using a notebook. Within one of the notebook cells, there is a check to determine if a table exists. However, even when the table does exist, it incorrectly identifies it as non-existent and proceeds to execut...
- 2072 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mahesh Chahare Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 6157 Views
- 2 replies
- 2 kudos
Resolved! AZURE_QUOTA_EXCEEDED_EXCEPTION - even with more than vCPUs than Databricks recommends
I am running this Delta Live Tables PoC from databricks-industry-solutions/industry-solutions-blueprintshttps://github.com/databricks-industry-solutions/pos-dltI have Standard_DS4_v2 with 28GB and 8 cores x 2 workers - so a total of 16 cores. This is...
- 6157 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Prasenjit Biswas We haven't heard from you since the last response from @Jose Gonzalez . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
- 2 kudos
- 1288 Views
- 1 replies
- 0 kudos
A streaming job going into infinite looping
HiBelow i am trying to read data from kafka, determine whether its fraud or not and then i need to write it back to mongodbbelow is my code read_kafka.pyfrom pyspark.sql import SparkSession from pyspark.sql.functions import * from pyspark.sql.types i...
- 1288 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Saswata,Can you remove the filter and see if it is printing output to console?kafka_df5=kafka_df4.filter(kafka_df4.status=="FRAUD")Thanks and RegardsSwetha Nandajan
- 0 kudos
- 7316 Views
- 5 replies
- 8 kudos
Ask your technical questions at Databricks Office Hours! Register here for any of our upcoming dates:May 10 - 11:00 AM - 12:00 PM PTMay 17 - 8:00 AM -...
Ask your technical questions at Databricks Office Hours! Register here for any of our upcoming dates:May 10 - 11:00 AM - 12:00 PM PTMay 17 - 8:00 AM - 9:00 AM PTMay 24 - 9:00 AM - 10:00 AM GMTDatabricks Office Hours connects you directly with experts...
- 7316 Views
- 5 replies
- 8 kudos
