- 19546 Views
- 3 replies
- 3 kudos
org.apache.spark.SparkException: Job aborted due to stage failure:
HiI have around 20 million records in my DF, and want to save it in HORIZINTAL SQL DB.This is error:org.apache.spark.SparkException: Job aborted due to stage failure: A shuffle map stage with indeterminate output was failed and retried. However, Spar...
- 19546 Views
- 3 replies
- 3 kudos
- 3 kudos
If there are any failures which may lead to a stage retry, but retrying the stage translates into potentially having an inconsistent result (indeterminacy) then this exception is raised. The exception is raised in newer version where the validation i...
- 3 kudos
- 2017 Views
- 0 replies
- 0 kudos
Databricks + confluent schema registry - Schema not found error
I am running a Kafka producer code on Databricks 12.2. I am testing AVRO serialization of message with help of confluent schema registry. I configured 'to_avro' function to read the schema from schema registry, but I am getting the below error> org.a...
- 2017 Views
- 0 replies
- 0 kudos
- 1359 Views
- 0 replies
- 0 kudos
Identifying the Dependency Automatically on the SQL Statements
Hi All,We have a SQL script file with lots of create, insert, and SQL statements. We need to figure out how to identify the dependencies between the statements dynamically(by using either spark/sql or using any program).For example - First, we need t...
- 1359 Views
- 0 replies
- 0 kudos
- 1759 Views
- 1 replies
- 0 kudos
Materialized view in DLT pipeline
When setting up DLT pipeline, there are 3 types of product edition, Core, Pro and Advanced. When I compare DLT Classic Core and DLT Classic Pro, the difference is that DLT Classic Pro can handle CDC. Does it means if I'm using DLT Classic Core, I hav...
- 1759 Views
- 1 replies
- 0 kudos
- 0 kudos
Also, If I'm using DLT Classic Core for DLT pipeline, the materialized view will be doing a full refresh or just update on the rows that have changes?
- 0 kudos
- 1511 Views
- 0 replies
- 0 kudos
Unzip multiple zip files in databricks
I have a zip file which in turn has multiple zip files inside it. I tried to write a code in databricks notebook to unzip all these files at once, but I ran into an error. So I started to unzip these one by one, but the code which worked in unzipping...
- 1511 Views
- 0 replies
- 0 kudos
- 2544 Views
- 4 replies
- 2 kudos
Resolved! About resuming the test - [ ref:_00D61JGc4._500Vp6jMk5:ref ]
Hello @Cert-Team I am very glad to hear from you,in response of the ticket (00481969), I have received mail for rescheduling of my test to resume. I am very thankful for your quick response in this matter.I have replied in mail about the date and tim...
- 2544 Views
- 4 replies
- 2 kudos
- 2 kudos
@Cert-Team, My issue has been clear. my test has been rescheduled to the time I have mentioned.I am very grateful for your help.Thanks and Regards@SHASHANK2
- 2 kudos
- 1484 Views
- 0 replies
- 0 kudos
Unable to unzip files recursively and copy into a different folder
I am currently trying to unzip files recursively from one folder(source folder) and copy all the unzipped files into the destination folder using databricks(pyspark). The destination path is still empty even after running this code. I tried looking f...
- 1484 Views
- 0 replies
- 0 kudos
- 2907 Views
- 1 replies
- 1 kudos
Issue with Creating and Running Databricks Jobs with new databricks cli v0.214.0
Hi Databricks Support,I'm encountering an issue with creating and running jobs on Databricks. Here are the details:Problem Description:When attempting to create and run a job using the old JSON (which was successfully used to create and run jobs usin...
- 2907 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod ,Thanks for the reference links for the solution.I found the solution mentioned in this https://github.com/databricks/databricks-sdk-go/discussions/384 GitHub. By using the get-run API, I was able to retrieve the running status of my...
- 1 kudos
- 4252 Views
- 5 replies
- 0 kudos
clusters/configuration doesn't work
Clusters configuration doesn't work, it's displaying a blank page. I need to add some libraries for the cluster
- 4252 Views
- 5 replies
- 0 kudos
- 2514 Views
- 1 replies
- 0 kudos
Microsoft PNP Azure Log Analytics
There is this com.microsoft.pnp.Util package that is used inside my Scala notebook used for Azure Databricks to Azure Log Monitoring tables push for my application logs. When I updated my Databricks Runtime from 10.4 to 14 in my clusters, it starts t...
- 2514 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @aarshps , How were you installing the library for the package com.microsoft.pnp.Util? Was it a library, whl, init script etc?
- 0 kudos
- 1806 Views
- 2 replies
- 0 kudos
REST API call throwing 403
Hi, Calling curl works just fine from my workspace ```` %sh curl --request GET "https://${DATABRICKS_HOST}/api/2.0/clusters/list" --header "Authorization: Bearer ${DATABRICKS_API_TOKEN}"```But when I transpose the same logic to a script in javascrip...
- 1806 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for sharing your findings, you can archive the ticket from Options on the right hand side.
- 0 kudos
- 1271 Views
- 0 replies
- 0 kudos
How to optimize a View ?
I have create a view from my underlying delta live table. Generally delta live table get auto optimize with vaccum in place. If thats so why do my view take 1 hour to be queried ? Any other way to optimize it ?
- 1271 Views
- 0 replies
- 0 kudos
- 1893 Views
- 2 replies
- 1 kudos
How to Determine Which CLI Methods to Use?
The Databricks CLI is considered Legacy for versions below 0.17 and is in Public Preview for versions above 0.20. I have access to documentation for both these versions separately. As I am developing a new project, I prefer not to use legacy options ...
- 1893 Views
- 2 replies
- 1 kudos
- 1 kudos
While the legacy CLI (versions 0.18 and below) is still available, it is not receiving any non-critical updates and it is recommend migrating to the new CLI as soon as possible. "Databricks recommends that you use Databricks CLI version 0.205 or abov...
- 1 kudos
- 2290 Views
- 1 replies
- 0 kudos
Community version now shows only the MLpersona
hiI have been using the community version for sometime and had saved some notebooks. While the notebooks are there in the workspace the different personas are not available as seen today. Only machine learning persona is visible. Would like to check...
- 2290 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi MSD, We removed the different personas that was observed earlier and now it is a flat system (It is not a recent change, been there for sometime). The notebooks would be saved in your workspace hence they are accessible. Irrespective of the perso...
- 0 kudos
- 1929 Views
- 0 replies
- 0 kudos
dbdemo LLAM CHATBOT RAG
i have an issue when running the below code using the default dbdemos in the advanced preparation , i have reduced the chunk_size and max_batch_size and running the code in a proper compute resources , could anyone help on that please :(spark.readStr...
- 1929 Views
- 0 replies
- 0 kudos
