- 2775 Views
- 2 replies
- 1 kudos
Databricks notebook issue
Hi,I'm trying to run ADF pipeline.However, it is getting fail at Notebook activity with below error.Error :NoSuchMethodError: com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(Lcom/microsoft/sqlserver/jdbc/ISQLServerBulkRecord;)V I think i...
- 2775 Views
- 2 replies
- 1 kudos
- 1 kudos
@shan_chandra Thanks for your reply as per your suggetion changed Databricks version from 9.1LTS to 12.2LTSBut after change this when i check library which you provided(i.e com.microsoft.azure:spark-mssql-connector_2.12:1.3.0) under Maven it is not...
- 1 kudos
- 3834 Views
- 6 replies
- 0 kudos
How to close gke databricks cluster
I have on gke databricks cluster with multiple nodes running. I want to close nodes when not in use. To reduce cost usage. And Is there any way to pause gke cluster on-demand.
- 3834 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @DeveshKaushik ,If this is the case then create a support ticket with databricks
- 0 kudos
- 4058 Views
- 3 replies
- 0 kudos
unable to install pymqi in azure databricks
Hi,I am trying to install pymqi via below command:pip install pymqi However, I am getting below error message:Python interpreter will be restarted. Collecting pymqi Using cached pymqi-1.12.10.tar.gz (91 kB) Installing build dependencies: started Inst...
- 4058 Views
- 3 replies
- 0 kudos
- 0 kudos
I don't think so, because it won't be specific to Databricks - this is all a property of the third party packages. And, there are billions of possible library conflicts. But this is not an example of a package conflict. It's an example of not complet...
- 0 kudos
- 5784 Views
- 1 replies
- 1 kudos
Resolved! Configure job to use one cluster instance to multiple jobs
Hi! I have several tiny jobs that run in parallel and I want them to run on the same cluster:- Tasks type Python Script: I send the parameters this way to run the pyspark scripts.- Job compute cluster created as (copied JSON from Databricks Job UI)Ho...
- 5784 Views
- 1 replies
- 1 kudos
- 1 kudos
Unfortunately, running multiple jobs in parallel using a single job cluster is not supported (yet). New in databricks is the possibility to create a job that orchestrates multiple jobs. These jobs will however still use their own cluster (configurati...
- 1 kudos
- 1352 Views
- 1 replies
- 1 kudos
Is there a solution that we can display the worker types based on spark version selection using api?
Is there a solution that allows us to display the worker types or driver types based on the selection of Spark version using an api?
- 1352 Views
- 1 replies
- 1 kudos
- 1 kudos
Can you clarify what you mean? Worker and driver types are not related to Spark version.
- 1 kudos
- 2939 Views
- 2 replies
- 2 kudos
Resolved! Reduce EBS Default Volumes
By default Databricks creates 2 volumes: one with 30GB and the other one with 150GB. We have a lot of nodes in our pools and so a los of Terabytes of Volumes, but we are not making any use of them in the jobs. Is there any way to reduce the volumes? ...
- 2939 Views
- 2 replies
- 2 kudos
- 2 kudos
Yes, EBS vols are essential for shuffle spill for example. You are probably using them!
- 2 kudos
- 7322 Views
- 1 replies
- 0 kudos
Uninstalling a preinstalled python package from Databricks
[Datasets](https://pypi.org/project/datasets/) python package comes preinstalled on databricks clusters. I want to uninstall it or completely prevent it's installation when I create/start a cluster.I couldn't find any solution on stackoverflow.And I ...
- 7322 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod note that you can't actually uninstall packages in the runtime with pip.
- 0 kudos
- 13262 Views
- 1 replies
- 0 kudos
Databricks cluster launch time
Hi Team,We have an @adf pipeline which will run some set of activities before #Azure databricks notebooks get called.As and when the notebooks are called our pipeline will launch a new cluster for every job with job compute as Standard F4 with a sing...
- 13262 Views
- 1 replies
- 0 kudos
- 2983 Views
- 1 replies
- 0 kudos
The job run failed because task dependency types are temporarily disabled
I am trying the recently released conditional tasks (https://docs.databricks.com/en/workflows/jobs/conditional-tasks.html). I have created a workflow where the leaf task depends on multiple tasks and its run_if property is set as AT_LEAST_ONE_SUCCESS...
- 2983 Views
- 1 replies
- 0 kudos
- 2807 Views
- 0 replies
- 0 kudos
change cloud provider from AWS to GOOGLE
I registered a Databricks account and selected using AWS as cloud provider, may I know how to change it to using Google? Thanks!
- 2807 Views
- 0 replies
- 0 kudos
- 5229 Views
- 2 replies
- 2 kudos
Resolved! com.databricks.NotebookExecutionException: FAILED
I am running the comparisons but I get an error, I am working from a databricks notebook.Could someone help me to solve the following error:com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED: Notebook not found: /user...
- 5229 Views
- 2 replies
- 2 kudos
- 2 kudos
two things that come to mind:1. the notebook resides on another path than '/users/cuenta_user/user/Tests'2. the notebook is not saved as a notebook but rather as an ordinary python file
- 2 kudos
- 2172 Views
- 0 replies
- 0 kudos
Databricks Assistant HIPPA? Future Cost?
With the Public Preview of Databricks Assistant, I have a few questions. 1) If the Azure Tenet is HIPPA compliant does that compliance also include the Databricks Assistant features? 2) Right now the product is free but what will the cost be? Will we...
- 2172 Views
- 0 replies
- 0 kudos
- 3195 Views
- 3 replies
- 1 kudos
Liquid Clustering
Hi Team,Could you please help us understand,1)Performance benchmarks of liquid clustering compared to z-order and partition.2)How much cost it incurs/saves compared to z-order and partitionRegards,Phanindra
- 3195 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Phani1 ,You can find performance related benchmarking here : https://www.databricks.com/blog/announcing-delta-lake-30-new-universal-format-and-liquid-clustering
- 1 kudos
- 1976 Views
- 0 replies
- 0 kudos
Intermittent (cert) failure when connecting to AWS RDS
I've just upgraded a bunch of jobs to 12.2 LTS runtime and now getting intermittent failures with the following message:```java.sql.SQLException: [Amazon](600000) Error setting/closing connection: PKIX path building failed: sun.security.provider.cert...
- 1976 Views
- 0 replies
- 0 kudos
- 11196 Views
- 0 replies
- 0 kudos
what are the different types life_cycle_state in databricks for job cluster
We are trying to get cluster life_cycle_state using API and we are able to get various values as belowRUNNINGPENDINGTERMINATEDINTERNAL_ERRORIs there any other values apart from above values it would be a great help.
- 11196 Views
- 0 replies
- 0 kudos
