- 5006 Views
- 2 replies
- 1 kudos
Resolved! Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error
Job is scheduled on interactive cluster, and it failed with below error and in the next scheduled run it ran fine. I want to why this error occurred and how can I prevent from occurring this again.How to debug these types of error? com.databricks.b...
- 5006 Views
- 2 replies
- 1 kudos
- 1 kudos
@JKR Could you try setting the configurations below at the cluster level and retry the job?spark.databricks.python.defaultPythonRepl pythonshellspark.databricks.pyspark.py4j.pinnedThread false
- 1 kudos
- 2690 Views
- 2 replies
- 0 kudos
Mount Workspace to Docker container
Is there a way to mount Workspace folder (WSFS) to the Docker container if I'm using the Databricks Container Services ofr running a general purpose cluster?If I create a cluster without a Docker image, the `!ls` command in Databricks notebook return...
- 2690 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello:Thanks for contacting Databricks Support! I'm afraid that mounting the WSFS directly into a Docker container isn't directly supported. The Databricks workspace is a specialized environment and isn't directly analogous to a regular filesystem. W...
- 0 kudos
- 5007 Views
- 9 replies
- 3 kudos
Databricks Certified Associate Developer for Apache Spark 3.0
Databricks Certified Associate Developer for Apache Spark 3.0
- 5007 Views
- 9 replies
- 3 kudos
- 3 kudos
Hey I am looking for sample papers for the above exam other than the one provided by databricks do any one have any idea about it
- 3 kudos
- 11975 Views
- 1 replies
- 0 kudos
Resolved! Azure Shared Clusters - P4J Security Exception on non-whitelisted classes
Hi allHaving some fun trying to run a notebook on a shared UC-aware, shared cluster - I keep on running into this error:py4j.security.Py4JSecurityException: Method public static org.apache.spark.sql.SparkSession org.apache.sedona.spark.SedonaContext....
- 11975 Views
- 1 replies
- 0 kudos
- 10220 Views
- 4 replies
- 1 kudos
Internal error. Attach your notebook to a different cluster or restart the current cluster.
Started getting this error while running all the scripts. All the scripts were running fine before. I tried de-attaching and also restart nothing seems to work.Internal error. Attach your notebook to a different cluster or restart the current cluste...
- 10220 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @210573 (Customer)​ ,I got the same error, tried to restart and create a new cluster but the solution does not work. What I did to fix the issue: Instead of putting in function, break the code out to run line by line. I just want to see where the ...
- 1 kudos
- 9239 Views
- 1 replies
- 2 kudos
Pytest imports of sibling modules when using Databricks for VSCode
Hello all, I am following the Databrick's documentation on unit testing found here: Run tests with pytest for the Databricks extension for Visual Studio Code - Azure Databricks | Microsoft LearnHowever, when taking it a step further I get an ImportEr...
- 9239 Views
- 1 replies
- 2 kudos
- 2 kudos
HelloImport errors happen often with Pytest. To Debug this error you can add this in your "test_myfunction_test.py":import sys # printing all directories for # interpreter to search sys.pathsys.path is a built-in variable within the sys module. I...
- 2 kudos
- 4223 Views
- 4 replies
- 1 kudos
Resolved! Error inviting user to workspace "Failed to add user: A user with email ... or username ... in different cases already exist in the account"
We have 3 workspaces - 1 old version in one AWS account, 2 latest versions in another.We are PAYG full edition, not using SSO.Our admins (existing DBX users in the `admins` group) can invite new users via the Admin Console from the 1 old and 1 new wo...
- 4223 Views
- 4 replies
- 1 kudos
- 1 kudos
We are facing same issue, We are on azure. @AndyAtINX you mean if user exist in workspace with abc@gmail.com we should add the user in workspace2 with abc@gmail.com not ABC@GMAIL.COM. if this the case we tried this and its not working for us.
- 1 kudos
- 4608 Views
- 6 replies
- 0 kudos
databricks-connector: Error: Cluster MASKED is in unexpected state Pending.
Is there a way to make databricks-connector wait for cluster to be running?Details:databricks-connector==13.1.0 and the python minor version of cluster and environment are both 3.10If the cluster is not running this will start it, but any commands af...
- 4608 Views
- 6 replies
- 0 kudos
- 0 kudos
FYI for anyone that finds this: This seems to be resolved in databricks-connector 14+
- 0 kudos
- 2303 Views
- 0 replies
- 0 kudos
What is the local-ssd used for in databricks?
What is the use-case for local-ssd's in databricks clusters? I noticed some clusters have many Tb's worth and some have no local ssd's.What are the pro's and con's of changing the disk size bigger and smaller? According to the docs:> The disk cache i...
- 2303 Views
- 0 replies
- 0 kudos
- 7998 Views
- 1 replies
- 0 kudos
DUPLICATE: Missing 'DBAcademy DLT' as a Cluster Policy when creating Delta Live Tables pipeline
Good afternoon,I'm currently going through Module 4 of the Data Engineering Associate pathway, specifically lesson 4.1 - DLT UI Walkthrough. We are instructed to specify the Cluster Policy as 'DBAcademy DLT' when configuring the pipeline. However, th...
- 7998 Views
- 1 replies
- 0 kudos
- 0 kudos
The policy will be available after running the notebook /Users/<YOUR USER NAME>/Data Engineering with Databricks - v3.1.4/Includes/Workspace-Setup
- 0 kudos
- 9923 Views
- 2 replies
- 0 kudos
tracing the history of a workflow
Hi!I use Databricks in Azure and I find it inconvenient not knowing the last modified user and modified time. How can I trace the history of modified time and user details? Would it be possible to deploy the workflows into higher environments?Thanks!
- 9923 Views
- 2 replies
- 0 kudos
- 0 kudos
Sorry! I added another issue at the end without mentioning it was a new issue I encountered. I had challenges in changing the owner of a workflow when I created a workflow. I ended up seeking help from another user with admin privileges to change the...
- 0 kudos
- 10125 Views
- 2 replies
- 1 kudos
Default Catalog Name setting doesn't work
I've recently started using Unity Catalog and I'm trying to set the default catalog name to something else than the hive_metastore for some of my workspaces.According to the documentation (Update an assignment | Metastores API | REST API reference | ...
- 10125 Views
- 2 replies
- 1 kudos
- 1 kudos
I found that setting the default catalog in the workspace "Admin Settings" works for Sql warehouse, spark cluster and compute polices. Consult this documentation : https://docs.databricks.com/en/data-governance/unity-catalog/create-catalogs.html#view...
- 1 kudos
- 5812 Views
- 3 replies
- 2 kudos
TPC -DS test On databricks
If I want to run TPC-DS test on databricks what are the steps involved, do we have already daya available on databricks file system or I have to download or create from somewhere.
- 5812 Views
- 3 replies
- 2 kudos
- 2 kudos
See the spark-sql-perf repo for details on how to run benchmark tests using TPC-DS - https://github.com/databricks/spark-sql-perf
- 2 kudos
- 1024 Views
- 1 replies
- 0 kudos
Bug - data profile internal code
Hi I am not sure how to post a potential bug but I stumble upon the following issue on DBR 13.2.The same code 'sometimes' works on DBR 12.2 LTS. But if I do it on a real table, this issue always occurs.
- 1024 Views
- 1 replies
- 0 kudos
- 0 kudos
Tried reproducing the issue on DBR 13.2, but unable to. find attached the screenshot How intermittently is the issue occurring ?
- 0 kudos
- 3924 Views
- 1 replies
- 0 kudos
Resolved! Merge error in streaming case
I have a streaming case, where i stream from 2 sources: source1 and source2. I write to seperate streams to pick the data up from the landing area (step1). then i write 2 extra streams to apply some tranformations in order to give them the same schem...
- 3924 Views
- 1 replies
- 0 kudos
- 0 kudos
Solved the problem by changing the cluster settings. The whole thing works when disabling Photon Acceleration...
- 0 kudos
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
