Delta Live Table Pipeline Load Error

[abhay_sud]

I am getting this error when running Delta Live Table Pipelines pulling from a source in our sandbox's local dbfs folder. It says the user is not authorized to perform this operation, whereas I am able to see the data when I run a simple select statement. Any idea why this might be happening?

Able to query the data from dbfs folder 

But the DLT pipeline notebook which has the load statement to pull the same file does not work.

Error Message:

 

Full JSON error message:

{

  "id": "6bdfbea0-d307-11ec-a13a-b20c225d589b",

  "sequence": {

    "control_plane_seq_no": 1651599564260008

  },

  "origin": {

    "cloud": "AWS",

    "region": "us-east-2",

    "org_id": 8628124014979244,

    "pipeline_id": "08d0a244-a1ee-4488-8960-14e56ad81535",

    "pipeline_name": "DLT_Test",

    "update_id": "549ffb7a-5fd5-49f8-813c-0eaa52725249"

  },

  "timestamp": "2022-05-13T21:55:36.970Z",

  "message": "Update 549ffb is FAILED.",

  "level": "ERROR",

  "error": {

    "exceptions": [

      {

        "message": "Failed to launch pipeline cluster 0513-215521-4w03rki9: Could not launch cluster due to cloud provider failures. aws_api_error_code: UnauthorizedOperation, aws_error_message: You are not authorized to perform this operation. Encoded authorization failur..."

      }

    ],

    "fatal": true

  },

  "details": {

    "update_progress": {

      "state": "FAILED"

    }

  },

  "event_type": "update_progress"

}

[Noopur_Nigam]

Hi @Abhay Sudhakaran​ The job is failing at the cluster start state, hence actual DLT pipeline has not been started yet and the job failed before that.

This seems to be an issue with authorization. You may want to check the cluster policy and other authorizations to start and work with the cluster.

[Kaniz]

Hi @Abhay Sudhakaran​ , We haven’t heard from you on the last response from @Noopur Nigam​, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.

[abhay_sud]

Thank @Kaniz Fatma​ , I am working with our AWS team to see how we can overcome the EC2 creation restriction. Hopefully that resolves the problem.

[Kaniz]

Hi @Abhay Sudhakaran​, Let us know how it goes.

[Anonymous]

Hey there @Abhay Sudhakaran​ 

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution and mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!