- 2252 Views
- 1 replies
- 3 kudos
How to execute requests later for a number of times that exceeds the Maximum concurrent runs?
I am trying to start the same Jobs multiple times using the python sdk's "run_now" command.If the number of requests exceeds the Maximum concurrent runs, the status of the run will be Skipped and the run will not be executed.Is there any way to queue...
- 2252 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi, We do have a private preview feature which will be enabled shortly for queueing. Please tag me (@Debayan Mukherjee​ ) with your next update so that I will get notified.
- 3 kudos
- 4686 Views
- 3 replies
- 3 kudos
Databricks run-time 12.2 LTS drop function problem
I am getting an error about the `drop function of pyspark` at a cluster using 12.2 LTS. When I check the error I see spark solved that bug, see SPARK-42444. Also when I check maintenance updates page, I saw this solved issue included the Databricks R...
- 4686 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Sevval Mehder​ Elevate our community by acknowledging exceptional contributions. Your participation in marking the best answers is a testament to our collective pursuit of knowledge.
- 3 kudos
- 3149 Views
- 2 replies
- 3 kudos
Resolved! Implement & Test DR Plan
Hi,Can you direct me to any documentation on how to implement and test Disaster Recovery for Databricks (PAAS) on Azure ?Thx & RgdsRamdas
- 3149 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Ramdas Panicher​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 3 kudos
- 2465 Views
- 1 replies
- 1 kudos
- 2465 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Ayush Raj​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 1 kudos
- 7736 Views
- 3 replies
- 4 kudos
Resolved! How to checkout specific commit version via databricks UI
I have integrated gitlab with my azure databricks repo, I am able to push and pull commits from the databricks UI, I want to checkout to a specific commit version via databricks UI.Note: I am aware that via the gitlab i have checkout to specific vers...
- 7736 Views
- 3 replies
- 4 kudos
- 4 kudos
After getting more context on databricks repo in details,Currently databricks doesn't support checkout of repo to specific commit.databricks provides only limited functionality mentioned belowAdd a repo and connect remotely laterClone a repo connecte...
- 4 kudos
- 3799 Views
- 2 replies
- 2 kudos
Trigger.AvailableNow getting stuck when there is no event
Hi, I have several streaming jobs, however one of them uses the Trigger.AvailableNow. The issue is that it gets stuck when there is no events or finishes ingesting all events. The expected behavior would be the job being shutdown.I've already checked...
- 3799 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, the source is an S3 bucket using file notification with SQS.No errors or warns in the logs, the AvailableNow trigger just gets stuck.
- 2 kudos
- 3116 Views
- 1 replies
- 0 kudos
what is best practice to handle the concurrency issue in batch processing?
Normally, our ELT framework takes in batches one by one and loads the data into target tables. But if more than one batches come in at the same time, the framework will break due to the concurrency issue that multiple sources are trying to write the ...
- 3116 Views
- 1 replies
- 0 kudos
- 0 kudos
you can partition you table to avoid the changes of getting this exception.
- 0 kudos
- 2133 Views
- 1 replies
- 3 kudos
www.databricks.com
Attention Community! For a limited period, we are offering a generous 50% discount on training at the Data + AI Summit. Simply apply the code FLS4vop5ep during the registration process. Hurry, though, as this offer will expire on June 12, 2023. Don'...
- 2133 Views
- 1 replies
- 3 kudos
- 3 kudos
Thank you for sharing this @Juliet Wu​!!!
- 3 kudos
- 3555 Views
- 1 replies
- 0 kudos
A streaming job going into infinite looping
HiBelow i am trying to read data from kafka, determine whether its fraud or not and then i need to write it back to mongodbbelow is my code read_kafka.pyfrom pyspark.sql import SparkSession from pyspark.sql.functions import * from pyspark.sql.types i...
- 3555 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Saswata,Can you remove the filter and see if it is printing output to console?kafka_df5=kafka_df4.filter(kafka_df4.status=="FRAUD")Thanks and RegardsSwetha Nandajan
- 0 kudos
- 3327 Views
- 1 replies
- 0 kudos
AutoML runs fail after 5 seconds
Hi everyoneI am exploring automl, and I met a strange problem - after I launch a classification experiment on my personal newly created cluster (screenshot attached) it successfully performs data exploration, but after that, all runs fail after appro...
- 3327 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Qwetroman,we can see the following error message in the notebook - ExecutionTimeoutError: Execution timed out before any trials could be successfully run. Please increase the timeout for AutoML to run some trials.What's the size of the dataset? St...
- 0 kudos
- 3589 Views
- 1 replies
- 2 kudos
Deploy model to AWS Sagemaker: ModuleNotFoundError: No module named 'docker'
Greetings, When trying to run the following command: %sh mlflow sagemaker build-and-push-containerI get the following error:/databricks/python3/lib/python3.9/site-packages/click/core.py:2309: UserWarning: Virtualenv support is still experimental and ...
- 3589 Views
- 1 replies
- 2 kudos
- 3940 Views
- 2 replies
- 2 kudos
Table Refresh UI Error
Within the UI it is possible to "Select tables for refresh" for a specific Delta Live Tables Workflow. I often use it to make a full refresh on smaller tables during development. Unfortunately, when an error occurs during the full refresh on selected...
- 3940 Views
- 2 replies
- 2 kudos
- 2 kudos
Could you please share the full error stack trace? it will help us to narrow down the issue
- 2 kudos
- 4081 Views
- 1 replies
- 1 kudos
How to set timezone for SQL Warehouse?
Hi, I want to change the default time zone for SQL Warehoue in the SQL Persona. When I try to Edit the SQL warehouse settings in the "SQL Warehouses" section, I am not able to find any setting where I can set the time zone. I am aware that I can set ...
- 4081 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks. I am aware of the SET TIME ZONE command but I need to run this command every time I start the SQL warehouse. I am looking for a way to change the default time zone of the SQL warehouse. Something like "spark.sql.session.timeZone GMT+10" that ...
- 1 kudos
- 4064 Views
- 2 replies
- 4 kudos
Error while rendering UI editor
Hi, does anyone facing an issue related to error while rendering editor on databricks notebook? it seems like this
- 4064 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi, This looks like a browser issue. Could you please try it with some other browser? Or clear the cookies and caches of the same browser and confirm? Please tag @Debayan​ with your next comment so that I will get notified. Thank you!
- 4 kudos
- 7727 Views
- 4 replies
- 5 kudos
Resolved! DLT pipeline is unable to process struct with hyphen in nested column name
Hello,I have some nested columns with hyphen i.e. sample-1 in struct column, recently DLT pipeline has started throwing synatx error. Before May 24, 2023, this was working fine.Is this a new bug in May 2023 release?After clearing table and table's da...
- 7727 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Rishabh Tomar​ We haven't heard from you since the last response from @Kaniz Fatma​ . Kindly share the information with us, and in return, we will provide you with the necessary solution. Thanks and Regards
- 5 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
