i just can't seem to make a cluster without error

[hmcdowelle]

I have been trying to create a cluster and this is my first time using databricks. I have tried across multiple resources and am getting frustrated. Each time the cluster comes up with an error. I have no idea what I am doing wrong. I use default settings to create the clusters. These are the errors I have received:

Cloud Provider Launch Failure: A cloud provider error was encountered while setting up the cluster. See theDatabricks guide for more information. Azure error code: AuthorizationFailed

Azure error message: The client '5e9a4129-c335-4dcb-84d0-488531e7b026' with object id '5e9a4129-c335-4dcb-84d0-488531e7b026' does not have authorization to perform action 'Microsoft.Network/publicIPAddresses/write' over scope '/subscriptions/58d9f65a-cfc8-4803-85a8-efc99715623c/resourceGroups/databricks-rg-LearnDatabricks-2zwx6ksnott2g/providers/Microsoft.Network/publicIPAddresses/9bf9900fd1d24a829e94d61a0e705d39-publicIP'.

Cloud Provider Launch Failure: A cloud provider error was encountered while setting up the cluster. See theDatabricks guide for more information.

Azure error code: OperationNotAllowed

Azure error message: Operation results in exceeding quota limits of Core. Maximum allowed: 4, Current in use: 4, Additional requested: 4. Please read more about quota increase at http://aka.ms/corequotaincrea

[MichalRawluk]

You have put some limits on top of your Azure Subscription. Therefore Bricks cannot create a new cluster cause simply Azure won't let it. Check resource group that was created alongside provision, and remove some restrains on that one.

Be vary to remove cluster when not used so it won't generate costs.

[hmcdowelle]

Thank you! But just to clarify, how do I put limits on the entire Azure Subscription? I created both the resource group and cluster with the default options and no provisions so I didn't really have any restraints on the cluster or resource group.

[evogelpohl]

@Michal Rawluk - can you be more specific as to the setting in Azure we need to change to allow DataBricks to create the cluster? I can't find the limit-setting you're referring to in my Azure portal/res-group.

[MichalRawluk]

I think your issue is using trial credits.

They have some limits beside value I reckon.To create a Databricks cluster you need at last 8 Cores (4 per driver, 4 per worker).You can contact Azure support:https://azure.microsoft.com/en-us/support/options/And have them increase Core limit on trial subscription.Also you may try changing regions. There are different limits on different region.

[evogelpohl]

Get the same error; using Azure w/ $200 initial trial-credits (have $115 available).

[JosieMundi]

Are you using a trial license?

As suggested you can submit a request for a higher quota.

Alternatively, you could try choosing a lower spec cluster. If you try removing the auto-scaling option and scaling down the workers to just 1 workers. Then choose something like the standard F4s worker Type.

[planks]

I am using an Azure Free Trial subscription and a Databricks Free Trial license. I did see the suggestion about submitting a request for a higher quota: when I attempted this the Azure service informed me that I cannot reques tthe increase without upgrading to "Pay-As-You-Go".

The lower-spec cluster doesn't work either, unfortunately. Even with autoscaling disabled there is still a Databricks requirement for 1 worker and 1 driver: the lowest spec type for each is 4 cores, 8 cores total, exceeding the 4 core quota on Azure for Free Trial subscriptions.

[rmariboe]

Requesting quota changes is not possible with Azure Trial subscription. We need 2 core cluster nodes and driver in order to run Databricks trial and complete Databricks training within the hard 4 core Azure Trial limit.

@Databricks_Support How can we force a cluster with 2 core node and driver? - This cannot be a hard problem to solve..?

[rmariboe]

Ok, got it to work using a single node "cluster" that doesn't require a driver. This does not, however, conform to the training material, and it will of course not enable users to test multiple node processing.

[rmariboe]

Huh, Single Node "cluster" did not work, but creating a "cluster" with 0 to 1 nodes apparently creates a driver and no nodes(?). I also defined to following to make sure only one core is being requested:

spark.executor.cores 1

spark.executor.memory 6g

Now the "cluster" is running, and I am getting some requests through.......

[rmariboe]

Ok, Single Node didn't work because of Azure's laggy backend.

Single Node works, also with Spark; creating a zero node "cluster" will not execute Spark functions (as is mentioned on Create Cluster page).

[leonarbe]

@Databricks_Support How can you follow any training, if it's nearly impossible to do it by using these two options:

1. Azure Standard (one month trial with $260 credit)

2. Databricks 14 day trial.

I tried creating a cluster using both options with the following:

* disabled autoscaling

* only one worker.

I atalaye got the same error message:

"Azure error code: OperationNotAllowed Azure error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westus2, Current Limit: 4, Current Usage: 4, Additional Required: 4, (Minimum) New Limit Required: 8. Submit a request for Quota increase at "

This is ridiculous, how can you even test this? Do you have to pay up front? Why you guys bother by offering a 14-day trial which is useless?

[PaulienLb]

Using free Azure subscription and trial tier for databricks I got the same error while doing this module https://docs.microsoft.com/en-us/learn/modules/describe-azure-databricks/

When creating the cluster I modified the cluster mode from 'standard' to 'none', problem solved; I could run the python notebook.

[artmy_mx]

What do you mean 'none'? I can see only the options: 'High Concurrency', 'Standard' and 'Single Node'