- 1429 Views
- 1 replies
- 0 kudos
Resolved! Limitation in Managed Volumes Recovery — UNDROP Should Be Supported
Hello Databricks Community,While reviewing the Databricks official documentation and performing a POC on managed volumes, I observed that volumes cannot be recovered using the UNDROP command if accidentally deleted — unlike managed tables.Technically...
- 1429 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you for highlighting this issue!Databricks is already working on implementing this in the future.
- 0 kudos
- 2975 Views
- 3 replies
- 0 kudos
TypeError: 'JavaPackage' object is not callable
from pyspark.sql import *if __name__ == "__main__": spark = SparkSession.builder \ .appName("hello Spark") \ .master("local[2]") \ .getOrCreate() data_list = [("Ravi",28), ("David",45), ("Abd...
- 2975 Views
- 3 replies
- 0 kudos
- 0 kudos
@farazahmad372 May I know the DBR version and type of cluster?Are you using serverless?
- 0 kudos
- 4054 Views
- 5 replies
- 2 kudos
Serverless job error - spark.rpc.message.maxSize
Hello, I am facing this error when moving a Workflow to serverless modeERROR : SparkException: Job aborted due to stage failure: Serialized task 482:0 was 269355219 bytes, which exceeds max allowed: spark.rpc.message.maxSize (268435456 bytes). Consid...
- 4054 Views
- 5 replies
- 2 kudos
- 2 kudos
Hello PiotrMi,We found out that the problem was caused by a collect() and managed to fix it by changing some codeThanks for your quick repliesBest regards,Antoine
- 2 kudos
- 2993 Views
- 1 replies
- 0 kudos
Unable to run python script from Azure DevOps git repo in Databricks Workflow job
Hi, I'm getting an issue while running a python script from Azure DevOps git repo in Databricks Workflow job task. The error stating internal commit path issue. But I referred the Source as Azure DevOps Services and I gave the branch name when settin...
- 2993 Views
- 1 replies
- 0 kudos
- 0 kudos
@SakthiGanesh This is a known type of issue when running Databricks Workflows with Azure DevOps Git-backed repos.Did you try, Workspace Path Instead of Internal Git Path?If possible, use a .ipynb notebook-based task rather than a raw .py script, note...
- 0 kudos
- 2705 Views
- 8 replies
- 1 kudos
Resolved! Is it possible for Store Procedure to be in Unity Catalog Dataricks
I got information that the latest release of Unity Catalog already supports Store Procedure, but I have searched from several sources that Unity catalog does not support Store Procedure, according to the following post: https://community.databricks.c...
- 2705 Views
- 8 replies
- 1 kudos
- 1 kudos
Yes, you can attend virtually and it is free. What I don't know is what is free. I believe the keynotes are free and some sessions. You should definitely register and check it out.
- 1 kudos
- 1813 Views
- 4 replies
- 1 kudos
Databricks DLT execution issue
I am having an issue when trying to do a full refresh of a DLT pipeline. I am getting the following error below: com.databricks.sql.managedcatalog.UnityCatalogServiceException: [RequestId=97d4fe52-b185-4757-b0b1-113cb96ae0bb ErrorClass=TABLE_ALREADY_...
- 1813 Views
- 4 replies
- 1 kudos
- 1 kudos
Are you facing the same issue, If you give a different name in the dlt decorator for the table?
- 1 kudos
- 4876 Views
- 3 replies
- 0 kudos
SQL - Dynamic overwrite + overwrite schema
Hello,Let say we have an empty table S that represents the schema we want to keepABCDEWe have another table T partionned by column A with a schema that depends on the file we have load into. Say :ABCF1b1c1f12b2c2f2Now to make T having the same schema...
- 4876 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, thanks for the reply. I've already looked at the documentation on this point, which actually states that dynamic overwrite doesn't work with schema overwrite, while the instructions described above seem to indicate the opposite.
- 0 kudos
- 725 Views
- 1 replies
- 0 kudos
Question about which tags appear in drop down
Hi there, I have a question regarding the appearance of tags in the drop down when adding a tag to a resource (catalog, schema, table, column - level). When does a tag get populated in a drop down? I noticed when I created a column level tag, and wan...
- 725 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @andreapeterson Yes, your understanding of Databricks tag behavior is correct. In Databricks Unity Catalog, tags follow a hierarchical inheritance pattern:Downward inheritance: Tags applied at higher levels (catalog → schema → table) become ava...
- 0 kudos
- 2448 Views
- 3 replies
- 2 kudos
Resolved! Creating Cluster configuration with library dependency using DABS
I am trying to create a cluster configuration using DABS and defining library dependencies.My yaml file looks like this: resources: clusters: project_Job_Cluster: cluster_name: "Project Cluster" spark_version: "16.3.x-cpu-ml-scala2.12" node_type_id: ...
- 2448 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @sparklez You're encountering this issue because the libraries field is not valid in the cluster configuration.Libraries need to be specified at the job level, not the cluster level.Option 1: Job-Level Libraries (Recommended)Move the libraries sec...
- 2 kudos
- 4162 Views
- 5 replies
- 7 kudos
Resolved! Migrating From Azure to Databricks
Hi Techie,May someone please help me with Pros and Cons from migrating my Realtime streaming solution from Azure to Databricks.which component I can replaced with Databricks and what benefit I can get out of it.Current Architecture:- Many Thanks
- 4162 Views
- 5 replies
- 7 kudos
- 7 kudos
I completely agree with @lingareddy_Alva on the costing part. One small point I would like to mention is We should only enable SPOT instances (60-90% cost savings) in Development/non-critical(PROD) environment. This option works great and is indeed c...
- 7 kudos
- 2094 Views
- 2 replies
- 0 kudos
Slow running Spark job issue - due to the unknown spark stages created by Databircks Compute cluster
Hi Team,Recently we migrated the spark jobs from self hosted spark(YARN) Cluster to Databricks.Currently we are using the Databricks workflows with Job_Compute clusters and the Job Type - Spark JAR type execution, so when we run the job in databricks...
- 2094 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Brahma,Thanks for your reply. As a first step I will disable AQE config and test it. We are using the node pools with job_compute cluster type so that its not spinning up a new cluster for each Job. I'm configuring the below two configs also, do ...
- 0 kudos
- 1489 Views
- 1 replies
- 0 kudos
Legacy Autoscaling(workflow) VS Enhanced Autoscaling(DLT)
I conducted a proof of concept (POC) to compare the performance of the DLT pipeline and Databricks Workflow using the same workload, task, code, and cluster configuration. Both configurations were set with autoscaling enabled, with a minimum of 1 wor...
- 1489 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi chsoni12,How are you doing today?, As per my understanding, That's a great observation, and it's awesome that you're testing performance and cost between DLT and regular workflows. The key difference here lies in how autoscaling works. DLT pipelin...
- 0 kudos
- 3646 Views
- 4 replies
- 0 kudos
i can list out the file using dbutils but can not able to read files in databricks
I can list out the file using dbutils but can not able to read files in databricks. PFB in screenshot. I can able to see the file using dbutils.fs.ls but when i try to read this file using read_excel then it is showing me an error like "FileNotFound...
- 3646 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello @MohammadWasi did you finally figure out what the problem was? I am encountering the exact same issue
- 0 kudos
- 752 Views
- 1 replies
- 0 kudos
Folders in Workflows/Jobs
Would be great if we could "group" Workflows/Jobs in Databricks using folders.This way, the Workflows list won't be too cluttered with all Workflows/Jobs in the same root-level.
- 752 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @BMex! You can submit this as a feature request through the Databricks Ideas Portal. This helps the product team consider it for future improvements
- 0 kudos
- 1765 Views
- 5 replies
- 1 kudos
Resolved! Fails to use unity catalog in All purpose cluster
Hey there,today we cannot load/read data from Unity Catalog with the same cluster as we did yesterday successfully (no changes in clsuter configuration).The error, which persists, according to the cluster logs is:com.databricks.common.client.Databric...
- 1765 Views
- 5 replies
- 1 kudos
- 1 kudos
@Advika the issue is gone.Now without any change all-purpose has access again to unity catalog.Who knows what happened...Thanks again for your interest
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
