I am currently trying to unzip files recursively from one folder(source folder) and copy all the unzipped files into the destination folder using databricks(pyspark). The destination path is still empty even after running this code. I tried looking f...
Hi Databricks Support,I'm encountering an issue with creating and running jobs on Databricks. Here are the details:Problem Description:When attempting to create and run a job using the old JSON (which was successfully used to create and run jobs usin...
Hi @Retired_mod ,Thanks for the reference links for the solution.I found the solution mentioned in this https://github.com/databricks/databricks-sdk-go/discussions/384 GitHub. By using the get-run API, I was able to retrieve the running status of my...
Clusters configuration doesn't work, it's displaying a blank page. I need to add some libraries for the cluster
There is this com.microsoft.pnp.Util package that is used inside my Scala notebook used for Azure Databricks to Azure Log Monitoring tables push for my application logs. When I updated my Databricks Runtime from 10.4 to 14 in my clusters, it starts t...
Hi @aarshps , How were you installing the library for the package com.microsoft.pnp.Util? Was it a library, whl, init script etc?
Hi, Calling curl works just fine from my workspace ```` %sh curl --request GET "https://${DATABRICKS_HOST}/api/2.0/clusters/list" --header "Authorization: Bearer ${DATABRICKS_API_TOKEN}"```But when I transpose the same logic to a script in javascrip...
Thanks for sharing your findings, you can archive the ticket from Options on the right hand side.
I have create a view from my underlying delta live table. Generally delta live table get auto optimize with vaccum in place. If thats so why do my view take 1 hour to be queried ? Any other way to optimize it ?
The Databricks CLI is considered Legacy for versions below 0.17 and is in Public Preview for versions above 0.20. I have access to documentation for both these versions separately. As I am developing a new project, I prefer not to use legacy options ...
While the legacy CLI (versions 0.18 and below) is still available, it is not receiving any non-critical updates and it is recommend migrating to the new CLI as soon as possible. "Databricks recommends that you use Databricks CLI version 0.205 or abov...
hiI have been using the community version for sometime and had saved some notebooks. While the notebooks are there in the workspace the different personas are not available as seen today. Only machine learning persona is visible. Would like to check...
Hi MSD, We removed the different personas that was observed earlier and now it is a flat system (It is not a recent change, been there for sometime). The notebooks would be saved in your workspace hence they are accessible. Irrespective of the perso...
i have an issue when running the below code using the default dbdemos in the advanced preparation , i have reduced the chunk_size and max_batch_size and running the code in a proper compute resources , could anyone help on that please :(spark.readStr...
Hello,I have a quick question. If my source code call pysark collect() or any method related to rdd methods, then pytest on my local PC will report the following error. My local machine doesn't have any specific setting for pyspark and I used findspa...
Thank you very much, brockb. Probably I will try it in databricks. Thanks.
I need to use Databricks SQL Statement Execution API w/ Javascript (see example post )For some reason, Curl Works, Python works, but Javascript fails.This works : (curl)______________________________curl --request POST \https://adb-5750xxxxxxx.azured...
Hi Team,Today (29th May 2024), I began my Databricks assessment exam, but it was abruptly suspended by the proctor without any explanation. This was my first exam and it has been a disappointing experience.I started the exam calmly, but the proctor w...
Thank you, @Cert-Team , for your quick response.I am looking forward to the resolution of my issue.
Hello,I would like to receive some support in creating a Community User Group in Romania, Cluj-Napoca. Our intention as a company is to become a partner and us to create a community: events, meetups so on.Looking for your help.Thank youNicu
I am getting an "INTERNAL_ERROR" on a databricks job submitted through the API. Which says:"Run result unavailable: run failed with error message All access to AWS S3 resource has been disabled"However, when I click on the notebook created by the job...
@Retired_mod In the s3 logs of the run, I am seeing this:24/05/29 06:39:30 WARN FileSystem: Failed to initialize fileystem dbfs:///: java.io.FileNotFoundException: Bucket user-workspace-s3-bucket does not exist24/05/29 06:39:30 ERROR DbfsHadoop3: Fai...
Hello all, I am running into a permission issue when running a simple MERGE INTO query from a Notebook: " AnalysisException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have USE SCHEMA on Schema 'system.query'."I can run the que...
Are you using an all purpose cluster or a SQL warehouse to run this query in the notebook?
