- 3377 Views
- 3 replies
- 2 kudos
DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute
Hi there, I would like to understand for DLT jobs if there is any way to get DLT jobs running in an *existing* (and currently running) All-purpose compute rather than spinning up an "ephemeral" (not yet initialized) Job Compute?
- 3377 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ChristianRRL , you cannot run a DLT pipeline in an All-purpose compute cluster. DLT Clusters are managed resources. Here are some details on how to configure a DLT cluster: https://docs.databricks.com/en/delta-live-tables/settings.html#configure-...
- 2 kudos
- 7836 Views
- 3 replies
- 0 kudos
List granted access for a group or a user
Is there any way where I can see what access a group or a user have been given to objects (Tables, views, catalogs etc.)?I noticed that we have the following information_schema tables:catalog_privilegesroutine_privilegestable_privilegesschema_privile...
- 7836 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Henrik To grant a user the privilege to query system tables, a metastore admin or another privileged user must grant USE and SELECT permissions on the system schemas. GRANT USAGE ON CATALOG system TO <user_name>; GRANT USAGE ON SCHEMA informa...
- 0 kudos
- 20969 Views
- 3 replies
- 3 kudos
org.apache.spark.SparkException: Job aborted due to stage failure:
HiI have around 20 million records in my DF, and want to save it in HORIZINTAL SQL DB.This is error:org.apache.spark.SparkException: Job aborted due to stage failure: A shuffle map stage with indeterminate output was failed and retried. However, Spar...
- 20969 Views
- 3 replies
- 3 kudos
- 3 kudos
If there are any failures which may lead to a stage retry, but retrying the stage translates into potentially having an inconsistent result (indeterminacy) then this exception is raised. The exception is raised in newer version where the validation i...
- 3 kudos
- 2786 Views
- 0 replies
- 0 kudos
Databricks + confluent schema registry - Schema not found error
I am running a Kafka producer code on Databricks 12.2. I am testing AVRO serialization of message with help of confluent schema registry. I configured 'to_avro' function to read the schema from schema registry, but I am getting the below error> org.a...
- 2786 Views
- 0 replies
- 0 kudos
- 1678 Views
- 0 replies
- 0 kudos
Identifying the Dependency Automatically on the SQL Statements
Hi All,We have a SQL script file with lots of create, insert, and SQL statements. We need to figure out how to identify the dependencies between the statements dynamically(by using either spark/sql or using any program).For example - First, we need t...
- 1678 Views
- 0 replies
- 0 kudos
- 2247 Views
- 1 replies
- 0 kudos
Materialized view in DLT pipeline
When setting up DLT pipeline, there are 3 types of product edition, Core, Pro and Advanced. When I compare DLT Classic Core and DLT Classic Pro, the difference is that DLT Classic Pro can handle CDC. Does it means if I'm using DLT Classic Core, I hav...
- 2247 Views
- 1 replies
- 0 kudos
- 0 kudos
Also, If I'm using DLT Classic Core for DLT pipeline, the materialized view will be doing a full refresh or just update on the rows that have changes?
- 0 kudos
- 1821 Views
- 0 replies
- 0 kudos
Unzip multiple zip files in databricks
I have a zip file which in turn has multiple zip files inside it. I tried to write a code in databricks notebook to unzip all these files at once, but I ran into an error. So I started to unzip these one by one, but the code which worked in unzipping...
- 1821 Views
- 0 replies
- 0 kudos
- 3275 Views
- 4 replies
- 2 kudos
Resolved! About resuming the test - [ ref:_00D61JGc4._500Vp6jMk5:ref ]
Hello @Cert-Team I am very glad to hear from you,in response of the ticket (00481969), I have received mail for rescheduling of my test to resume. I am very thankful for your quick response in this matter.I have replied in mail about the date and tim...
- 3275 Views
- 4 replies
- 2 kudos
- 2 kudos
@Cert-Team, My issue has been clear. my test has been rescheduled to the time I have mentioned.I am very grateful for your help.Thanks and Regards@SHASHANK2
- 2 kudos
- 2019 Views
- 0 replies
- 0 kudos
Unable to unzip files recursively and copy into a different folder
I am currently trying to unzip files recursively from one folder(source folder) and copy all the unzipped files into the destination folder using databricks(pyspark). The destination path is still empty even after running this code. I tried looking f...
- 2019 Views
- 0 replies
- 0 kudos
- 3474 Views
- 1 replies
- 1 kudos
Issue with Creating and Running Databricks Jobs with new databricks cli v0.214.0
Hi Databricks Support,I'm encountering an issue with creating and running jobs on Databricks. Here are the details:Problem Description:When attempting to create and run a job using the old JSON (which was successfully used to create and run jobs usin...
- 3474 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod ,Thanks for the reference links for the solution.I found the solution mentioned in this https://github.com/databricks/databricks-sdk-go/discussions/384 GitHub. By using the get-run API, I was able to retrieve the running status of my...
- 1 kudos
- 5103 Views
- 5 replies
- 0 kudos
clusters/configuration doesn't work
Clusters configuration doesn't work, it's displaying a blank page. I need to add some libraries for the cluster
- 5103 Views
- 5 replies
- 0 kudos
- 2879 Views
- 1 replies
- 0 kudos
Microsoft PNP Azure Log Analytics
There is this com.microsoft.pnp.Util package that is used inside my Scala notebook used for Azure Databricks to Azure Log Monitoring tables push for my application logs. When I updated my Databricks Runtime from 10.4 to 14 in my clusters, it starts t...
- 2879 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @aarshps , How were you installing the library for the package com.microsoft.pnp.Util? Was it a library, whl, init script etc?
- 0 kudos
- 2387 Views
- 2 replies
- 0 kudos
REST API call throwing 403
Hi, Calling curl works just fine from my workspace ```` %sh curl --request GET "https://${DATABRICKS_HOST}/api/2.0/clusters/list" --header "Authorization: Bearer ${DATABRICKS_API_TOKEN}"```But when I transpose the same logic to a script in javascrip...
- 2387 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for sharing your findings, you can archive the ticket from Options on the right hand side.
- 0 kudos
- 1556 Views
- 0 replies
- 0 kudos
How to optimize a View ?
I have create a view from my underlying delta live table. Generally delta live table get auto optimize with vaccum in place. If thats so why do my view take 1 hour to be queried ? Any other way to optimize it ?
- 1556 Views
- 0 replies
- 0 kudos
- 2834 Views
- 2 replies
- 1 kudos
How to Determine Which CLI Methods to Use?
The Databricks CLI is considered Legacy for versions below 0.17 and is in Public Preview for versions above 0.20. I have access to documentation for both these versions separately. As I am developing a new project, I prefer not to use legacy options ...
- 2834 Views
- 2 replies
- 1 kudos
- 1 kudos
While the legacy CLI (versions 0.18 and below) is still available, it is not receiving any non-critical updates and it is recommend migrating to the new CLI as soon as possible. "Databricks recommends that you use Databricks CLI version 0.205 or abov...
- 1 kudos
