I am running a Kafka producer code on Databricks 12.2. I am testing AVRO serialization of message with help of confluent schema registry. I configured 'to_avro' function to read the schema from schema registry, but I am getting the below error> org.a...
Hi All,We have a SQL script file with lots of create, insert, and SQL statements. We need to figure out how to identify the dependencies between the statements dynamically(by using either spark/sql or using any program).For example - First, we need t...
When setting up DLT pipeline, there are 3 types of product edition, Core, Pro and Advanced. When I compare DLT Classic Core and DLT Classic Pro, the difference is that DLT Classic Pro can handle CDC. Does it means if I'm using DLT Classic Core, I hav...
Also, If I'm using DLT Classic Core for DLT pipeline, the materialized view will be doing a full refresh or just update on the rows that have changes?
I have a zip file which in turn has multiple zip files inside it. I tried to write a code in databricks notebook to unzip all these files at once, but I ran into an error. So I started to unzip these one by one, but the code which worked in unzipping...
Hello @Cert-Team I am very glad to hear from you,in response of the ticket (00481969), I have received mail for rescheduling of my test to resume. I am very thankful for your quick response in this matter.I have replied in mail about the date and tim...
@Cert-Team, My issue has been clear. my test has been rescheduled to the time I have mentioned.I am very grateful for your help.Thanks and Regards@SHASHANK2
I am currently trying to unzip files recursively from one folder(source folder) and copy all the unzipped files into the destination folder using databricks(pyspark). The destination path is still empty even after running this code. I tried looking f...
Hi Databricks Support,I'm encountering an issue with creating and running jobs on Databricks. Here are the details:Problem Description:When attempting to create and run a job using the old JSON (which was successfully used to create and run jobs usin...
Hi @Retired_mod ,Thanks for the reference links for the solution.I found the solution mentioned in this https://github.com/databricks/databricks-sdk-go/discussions/384 GitHub. By using the get-run API, I was able to retrieve the running status of my...
Clusters configuration doesn't work, it's displaying a blank page. I need to add some libraries for the cluster
There is this com.microsoft.pnp.Util package that is used inside my Scala notebook used for Azure Databricks to Azure Log Monitoring tables push for my application logs. When I updated my Databricks Runtime from 10.4 to 14 in my clusters, it starts t...
Hi @aarshps , How were you installing the library for the package com.microsoft.pnp.Util? Was it a library, whl, init script etc?
Hi, Calling curl works just fine from my workspace ```` %sh curl --request GET "https://${DATABRICKS_HOST}/api/2.0/clusters/list" --header "Authorization: Bearer ${DATABRICKS_API_TOKEN}"```But when I transpose the same logic to a script in javascrip...
Thanks for sharing your findings, you can archive the ticket from Options on the right hand side.
I have create a view from my underlying delta live table. Generally delta live table get auto optimize with vaccum in place. If thats so why do my view take 1 hour to be queried ? Any other way to optimize it ?
The Databricks CLI is considered Legacy for versions below 0.17 and is in Public Preview for versions above 0.20. I have access to documentation for both these versions separately. As I am developing a new project, I prefer not to use legacy options ...
While the legacy CLI (versions 0.18 and below) is still available, it is not receiving any non-critical updates and it is recommend migrating to the new CLI as soon as possible. "Databricks recommends that you use Databricks CLI version 0.205 or abov...
hiI have been using the community version for sometime and had saved some notebooks. While the notebooks are there in the workspace the different personas are not available as seen today. Only machine learning persona is visible. Would like to check...
Hi MSD, We removed the different personas that was observed earlier and now it is a flat system (It is not a recent change, been there for sometime). The notebooks would be saved in your workspace hence they are accessible. Irrespective of the perso...
i have an issue when running the below code using the default dbdemos in the advanced preparation , i have reduced the chunk_size and max_batch_size and running the code in a proper compute resources , could anyone help on that please :(spark.readStr...
Hello,I have a quick question. If my source code call pysark collect() or any method related to rdd methods, then pytest on my local PC will report the following error. My local machine doesn't have any specific setting for pyspark and I used findspa...
Thank you very much, brockb. Probably I will try it in databricks. Thanks.
