cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Live Table Pipeline - Azure cluster fail

Aaron1234567
New Contributor III

Failed to launch pipeline cluster 0802-171503-4m02lexd: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results in...

I've also noticed that it is not deploying in the same region as my Resource Group or the Databricks instance which is in Southcentral US. I've also specifying the specs of the VM/cluster as stated in the top solution in this forum.

{

  "id": "faa79e00-1286-11ed-82b7-ea83de10642f",

  "sequence": {

    "control_plane_seq_no": 1658882570762832

  },

  "origin": {

    "cloud": "Azure",

    "region": "westus",

    "org_id": 5389340109879859,

    "pipeline_id": "24007e7f-15b9-45e8-9628-500ba30c826c",

    "pipeline_name": "DLT-Demo-81-aaron@clevelanalytics.com",

    "update_id": "23fff1f7-a765-46dc-82bc-43f04ef451c1"

  },

  "timestamp": "2022-08-02T17:17:25.856Z",

  "message": "Update 23fff1 is FAILED.",

  "level": "ERROR",

  "error": {

    "exceptions": [

      {

        "message": "Failed to launch pipeline cluster 0802-171503-4m02lexd: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results in..."

      }

    ],

    "fatal": true

  },

  "details": {

    "update_progress": {

      "state": "FAILED"

    }

  },

  "event_type": "update_progress"

}

9 REPLIES 9

Hubert-Dudek
Esteemed Contributor III

You will receive this error when you exceed the limit of cores for a region. To fix it, you need to raise a support ticket to increase the boundary of the number of cores for a West Europe region.

47369-image

Yeah I checked that and it looks good so far. What I did find is if I flip it to Production it works. Then it seems I can flip to to Development and it works fine after that. Weird? Admittedly just learning this stuff so I don't really know what I'm doing yet.

So I think you were correct but I think the quota data in my subscription was stale the first time I checked it or I just screwed up. I just checked and it was at the limit. Got that increased but now I'm getting another error which is related to quotas for Public IP address limits. I was able to update the quota for the cores without a ticket in Azure but this one requires a support request. Logging this for others.

Here is the error I'm getting now:

{

"reason": {

"code": "CLOUD_PROVIDER_LAUNCH_FAILURE",

"type": "CLOUD_FAILURE",

"parameters": {

"azure_error_code": "PublicIPCountLimitReached",

"azure_error_message": "Cannot create more than 10 public IP addresses for this subscription in this region. (Azure request Id: 3f096c68-2d5d-4bcf-bb19-91b04ecd041e)",

"databricks_error_message": "Error code: PublicIPCountLimitReached, error message: Cannot create more than 10 public IP addresses for this subscription in this region. (Azure request Id: 3f096c68-2d5d-4bcf-bb19-91b04ecd041e)"

}

}

}

Aaron1234567
New Contributor III

So I stood up a fresh instance of Databricks and now I'm back to this original error. All the quotas I can see are good to go except for maybe a NetworkWatcher. This is confusing.

{

  "id": "d4246d80-12ae-11ed-82b7-ea83de10642f",

  "sequence": {

    "control_plane_seq_no": 1658882570791530

  },

  "origin": {

    "cloud": "Azure",

    "region": "westus",

    "org_id": 2417202028545588,

    "pipeline_id": "fd140f96-5ab6-4557-b9fe-ece2d1ff8e2c",

    "pipeline_name": "LiveTablePipelineTest",

    "update_id": "2b8b7f5b-2424-423d-bc3b-ed9257b37a5d"

  },

  "timestamp": "2022-08-02T22:02:41.112Z",

  "message": "Update 2b8b7f is FAILED.",

  "level": "ERROR",

  "error": {

    "exceptions": [

      {

        "message": "Failed to launch pipeline cluster 0802-220150-qaojk6yb: The operation could not be performed on your account with the following error message: azure_error_code: OperationNotAllowed, azure_error_message: Operation could not be completed as it results in..."

      }

    ],

    "fatal": false

  },

  "details": {

    "update_progress": {

      "state": "FAILED"

    }

  },

  "event_type": "update_progress"

}

----------------------------------------------------------

{

  "id": "fd140f96-5ab6-4557-b9fe-ece2d1ff8e2c",

  "clusters": [

    {

      "label": "default",

      "num_workers": 1

    }

  ],

  "development": true,

  "continuous": false,

  "channel": "CURRENT",

  "edition": "ADVANCED",

  "photon": false,

  "libraries": [

    {

      "notebook": {

        "path": "/Users/aaron@clevelanalytics.com/DeltaLiveTableTest"

      }

    }

  ],

  "name": "LiveTablePipelineTest",

  "storage": "dbfs:/mnt/land",

  "target": "Netsuite"

}

Aaron1234567
New Contributor III

What I still don't understand is why it is trying to create the cluster in the "westus" region. The resource group and DataBricks instance are in "southcentral us". I have no idea how I got it working earlier when flipping back and forth between Development and Production.

Aaron1234567
New Contributor III

Southcentral US Region

image

Aaron1234567
New Contributor III

So it started working again. I'm not seeing a rhyme or reason to this. When it does work it picks the correct region.

Vidula
Honored Contributor

Hi there @Aaron LeBato​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!