- 5028 Views
- 5 replies
- 4 kudos
Is there a way to clone job cluster or edit cluster using JSON?
I've created workflow job (let say job A) and set up job cluster configuration for it.Now I want to create another workflow job (job B) but use almost the same settings for job cluster.I can see cluster settings in JSON (for both jobs) but I can't ed...
- 5028 Views
- 5 replies
- 4 kudos
- 4 kudos
Also you can use terraform exporter with -match flag to get a .tf definition for a job A. Once initialized, you can define job B.Another option is to use dbx
- 4 kudos
- 18639 Views
- 6 replies
- 15 kudos
Job cluster vs All purpose cluster
Environment: AzureI've a workflow that takes approximately a minute to execute and I want to run the job every 2 minutes.. All purpose cluster:On attaching all purpose cluster to the job, it takes approx. 60 seconds to execute.Using job cluster:On at...
- 18639 Views
- 6 replies
- 15 kudos
- 6247 Views
- 4 replies
- 6 kudos
Resolved! Error When Starting the Cluster
I am having this error when running my cluster, any idea why?
- 6247 Views
- 4 replies
- 6 kudos
- 6 kudos
@Werner Stinckens​ , I checked again, you cannot change them after your workspace is deployed. The only way right now is to recreate the workspace and migrate. It’s not possible to update CIDR range right now without migration.
- 6 kudos
- 1751 Views
- 2 replies
- 0 kudos
Not able to merge Databricks cluster delta tables(location Abfs) with PowerBi for analytics Purpose
Hi everyone .. i have an issue while connecting azure Databricks cluster delta tables(location Abfs) with PowerBi for analytics Purpose. but after succesfully linking cluster using Server Hostname and HTTP Path with powerbi is showing an error:-Data...
- 1751 Views
- 2 replies
- 0 kudos
- 12583 Views
- 2 replies
- 2 kudos
Is it possible to have Cluster with pre-installed dependencies?
I run some jobs in the Databricks environment where some resources need authentication. I do this (and I need to) through the vault-cli in the init-script.However, every time in the init-script I need to install vault-cli and other libraries. Is ther...
- 12583 Views
- 2 replies
- 2 kudos
- 2 kudos
@João Victor Albuquerque​ :Yes, there are a few ways to pre-install libraries and tools in the Databricks environment:Cluster-scoped init scripts: You can specify a shell script to be run when a cluster is created or restarted. This script can includ...
- 2 kudos
- 11586 Views
- 1 replies
- 0 kudos
Azure Databricks cluster driver config
Hi there,I am trying to set up databricks storage account access in Global init script. according to Azure Databricks document on creating cluster with driver config for all clusters (https://learn.microsoft.com/en-us/azure/databricks/archive/compute...
- 11586 Views
- 1 replies
- 0 kudos
- 1286 Views
- 2 replies
- 2 kudos
workflow cluster was create error
I set the workflow to run at 12:00 every day in the workflow, but the workflow failed with the error message below, and I don't know why.Run result unavailable: run failed with error message Unexpected failure while waiting for the cluster (0506-0233...
- 1286 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @Sangwoo Lee​ ,As mentioned by vignesh, it seems like an infra related issue. > Does the user (which executes the job) has access to start a cluster?> Incase if it is not an access issue and Incase if you are starting a lot of workflow jobs tog...
- 2 kudos
- 6039 Views
- 9 replies
- 1 kudos
Unable to launch notebook
I have created a 12.2 LTS cluster and trying to launch notebook attached to this cluster. But unable to launch, it is not giving any error instead it is still showing the same home page.
- 6039 Views
- 9 replies
- 1 kudos
- 1 kudos
Hi @Muskan Bansal​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 1 kudos
- 4218 Views
- 3 replies
- 1 kudos
databricks cluster creation error
Hi Databricks Community,My Databricks workspace created on Azure pay-as-go subscription.I am facing two sided challenges.First>> I am not able to create Delta-Live-table pipeline or any other all-purpose multi-node cluster as it is throwing below e...
- 4218 Views
- 3 replies
- 1 kudos
- 1 kudos
HI All, Thanks for your reply. Just to update you, I am now able to to create DLT pipeline as well as all purpose multi-node cluster with minimum resources. This is due to Quota limit and I was able to increase it. But observation is, if I try to us...
- 1 kudos
- 2670 Views
- 3 replies
- 1 kudos
Create a Dockerfile from Cluster
Is there a way to create a Dockerfile from Workspace A's cluster configurations and deploy that on a different different cluster in Workspace B?
- 2670 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Kevin Kim​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
- 1 kudos
- 3639 Views
- 2 replies
- 0 kudos
pip install in cluster using web UI and extra index
In an init script or a notebook, we can:pip install --index-url=<our private pypi url> --extra-index-url=https://pypi.org/simple <a module>In the cluster web UI (libraries -> install library), we can give only the url of our private repository, but n...
- 3639 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Philippe CRAVE​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 0 kudos
- 7795 Views
- 8 replies
- 0 kudos
Not able to connect to On-Prem Oracle from Databricks cluster
Hi Everyone,I was trying to connect to Oracle Instance from Databricks cluster and it is giving below error:java.sql.SQLTimeoutException: ORA-12170: Cannot connect. TCP connect timeout of 30000ms for host xx.x.x.*** port 1521. (CONNECTION_ID=CgM7V7UB...
- 7795 Views
- 8 replies
- 0 kudos
- 0 kudos
@Satya89:The error message you received indicates that the TCP connection to the Oracle database timed out. This could be caused by a number of factors such as network issues, firewall restrictions, or the database being overloaded.Here are a few ste...
- 0 kudos
- 1682 Views
- 2 replies
- 4 kudos
Specifying cluster on running a job
Hi,Let's say that I am starting jobs with different parameters at a certain time each day in the following manner:response = requests.post( "https://%s/api/2.0/jobs/run-now" % (DOMAIN), headers={"Authorization": "Bearer %s" % TOKEN}, json={ ...
- 1682 Views
- 2 replies
- 4 kudos
- 4 kudos
@Tjadi Peeters​ You can select option Autoscaling/Enhanced Scaling in workflows which will scale based on workload
- 4 kudos
- 3392 Views
- 3 replies
- 0 kudos
Resolved! Why does my Single Node cluster automatically resize num_workers?
Hi community,We have setup a Databricks cluster as Single node with num_workers=0 . Sometimes the cluster automatically resizes to e.g. 10 workers. When I edit the cluster subsequently it gives an error that num_workers is not allowed for Single node...
- 3392 Views
- 3 replies
- 0 kudos
- 0 kudos
I think the issue is solved, at least it didn't occur in the last month. We monitored this via Azure Log Analytics. Maybe it was solved due to some patch/update, thanks anyway!
- 0 kudos
- 3192 Views
- 4 replies
- 1 kudos
Resolved! How can I cluster-install a c-Python library (pyRFC)?
If possible, how can one go about installing a Python library with SDK dependencies like pyRFC? (https://github.com/SAP/PyRFC)The SDK dependencies depend on the type of OS, and since we're running Databricks out of AWS, I assume one would have to mat...
- 3192 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Wonseok Choi​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...
- 1 kudos