topic Re: Delta Live Table Pipeline - Azure cluster fail in Data Engineering

Delta Live Table Pipeline - Azure cluster fail

Aaron1234567 — Tue, 02 Aug 2022 17:22:32 GMT

Failed to launch pipeline cluster 0802-171503-4m02lexd: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results in...

I've also noticed that it is not deploying in the same region as my Resource Group or the Databricks instance which is in Southcentral US. I've also specifying the specs of the VM/cluster as stated in the top solution in this forum.

{

"id": "faa79e00-1286-11ed-82b7-ea83de10642f",

"sequence": {

"control_plane_seq_no": 1658882570762832

"origin": {

"cloud": "Azure",

"region": "westus",

"org_id": 5389340109879859,

"pipeline_id": "24007e7f-15b9-45e8-9628-500ba30c826c",

"pipeline_name": "DLT-Demo-81-aaron@clevelanalytics.com",

"update_id": "23fff1f7-a765-46dc-82bc-43f04ef451c1"

"timestamp": "2022-08-02T17:17:25.856Z",

"message": "Update 23fff1 is FAILED.",

"level": "ERROR",

"error": {

"exceptions": [

{

"message": "Failed to launch pipeline cluster 0802-171503-4m02lexd: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results in..."

}

"fatal": true

"details": {

"update_progress": {

"state": "FAILED"

}

"event_type": "update_progress"

}

Re: Delta Live Table Pipeline - Azure cluster fail

Hubert-Dudek — Tue, 02 Aug 2022 19:33:39 GMT

You will receive this error when you exceed the limit of cores for a region. To fix it, you need to raise a support ticket to increase the boundary of the number of cores for a West Europe region.

Re: Delta Live Table Pipeline - Azure cluster fail

Aaron1234567 — Tue, 02 Aug 2022 19:35:31 GMT

Yeah I checked that and it looks good so far. What I did find is if I flip it to Production it works. Then it seems I can flip to to Development and it works fine after that. Weird? Admittedly just learning this stuff so I don't really know what I'm doing yet.

Re: Delta Live Table Pipeline - Azure cluster fail

Aaron1234567 — Tue, 02 Aug 2022 20:09:48 GMT

So I think you were correct but I think the quota data in my subscription was stale the first time I checked it or I just screwed up. I just checked and it was at the limit. Got that increased but now I'm getting another error which is related to quotas for Public IP address limits. I was able to update the quota for the cores without a ticket in Azure but this one requires a support request. Logging this for others.

Re: Delta Live Table Pipeline - Azure cluster fail

Aaron1234567 — Tue, 02 Aug 2022 20:11:07 GMT

Here is the error I'm getting now:

{

"reason": {

"code": "CLOUD_PROVIDER_LAUNCH_FAILURE",

"type": "CLOUD_FAILURE",

"parameters": {

"azure_error_code": "PublicIPCountLimitReached",

"azure_error_message": "Cannot create more than 10 public IP addresses for this subscription in this region. (Azure request Id: 3f096c68-2d5d-4bcf-bb19-91b04ecd041e)",

"databricks_error_message": "Error code: PublicIPCountLimitReached, error message: Cannot create more than 10 public IP addresses for this subscription in this region. (Azure request Id: 3f096c68-2d5d-4bcf-bb19-91b04ecd041e)"

}

Re: Delta Live Table Pipeline - Azure cluster fail

Aaron1234567 — Tue, 02 Aug 2022 22:10:45 GMT

So I stood up a fresh instance of Databricks and now I'm back to this original error. All the quotas I can see are good to go except for maybe a NetworkWatcher. This is confusing.

{

"id": "d4246d80-12ae-11ed-82b7-ea83de10642f",

"sequence": {

"control_plane_seq_no": 1658882570791530

"origin": {

"cloud": "Azure",

"region": "westus",

"org_id": 2417202028545588,

"pipeline_id": "fd140f96-5ab6-4557-b9fe-ece2d1ff8e2c",

"pipeline_name": "LiveTablePipelineTest",

"update_id": "2b8b7f5b-2424-423d-bc3b-ed9257b37a5d"

"timestamp": "2022-08-02T22:02:41.112Z",

"message": "Update 2b8b7f is FAILED.",

"level": "ERROR",

"error": {

"exceptions": [

{

"message": "Failed to launch pipeline cluster 0802-220150-qaojk6yb: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results in..."

}

"fatal": false

"details": {

"update_progress": {

"state": "FAILED"

}

"event_type": "update_progress"

}

----------------------------------------------------------

{

"id": "fd140f96-5ab6-4557-b9fe-ece2d1ff8e2c",

"clusters": [

{

"label": "default",

"num_workers": 1

}

"development": true,

"continuous": false,

"channel": "CURRENT",

"edition": "ADVANCED",

"photon": false,

"libraries": [

{

"notebook": {

"path": "/Users/aaron@clevelanalytics.com/DeltaLiveTableTest"

}

"name": "LiveTablePipelineTest",

"storage": "dbfs:/mnt/land",

"target": "Netsuite"

}

Re: Delta Live Table Pipeline - Azure cluster fail

Aaron1234567 — Tue, 02 Aug 2022 22:12:28 GMT

What I still don't understand is why it is trying to create the cluster in the "westus" region. The resource group and DataBricks instance are in "southcentral us". I have no idea how I got it working earlier when flipping back and forth between Development and Production.

Re: Delta Live Table Pipeline - Azure cluster fail

Aaron1234567 — Tue, 02 Aug 2022 22:14:54 GMT

Southcentral US Region

Re: Delta Live Table Pipeline - Azure cluster fail

Aaron1234567 — Tue, 02 Aug 2022 22:34:37 GMT

So it started working again. I'm not seeing a rhyme or reason to this. When it does work it picks the correct region.

Re: Delta Live Table Pipeline - Azure cluster fail

Vidula — Wed, 07 Sep 2022 07:16:12 GMT

Hi there @Aaron LeBato

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!