topic Re: Delta Live Table Pipeline Load Error in Data Engineering

Delta Live Table Pipeline Load Error

abhay_sud — Fri, 13 May 2022 23:14:51 GMT

I am getting this error when running Delta Live Table Pipelines pulling from a source in our sandbox's local dbfs folder. It says the user is not authorized to perform this operation, whereas I am able to see the data when I run a simple select statement. Any idea why this might be happening?

Able to query the data from dbfs folder

But the DLT pipeline notebook which has the load statement to pull the same file does not work.

Error Message:

Full JSON error message:

{

"id": "6bdfbea0-d307-11ec-a13a-b20c225d589b",

"sequence": {

"control_plane_seq_no": 1651599564260008

"origin": {

"cloud": "AWS",

"region": "us-east-2",

"org_id": 8628124014979244,

"pipeline_id": "08d0a244-a1ee-4488-8960-14e56ad81535",

"pipeline_name": "DLT_Test",

"update_id": "549ffb7a-5fd5-49f8-813c-0eaa52725249"

"timestamp": "2022-05-13T21:55:36.970Z",

"message": "Update 549ffb is FAILED.",

"level": "ERROR",

"error": {

"exceptions": [

{

"message": "Failed to launch pipeline cluster 0513-215521-4w03rki9: Could not launch cluster due to cloud provider failures. aws_api_error_code: UnauthorizedOperation, aws_error_message: You are not authorized to perform this operation. Encoded authorization failur..."

}

"fatal": true

"details": {

"update_progress": {

"state": "FAILED"

}

"event_type": "update_progress"

}

Re: Delta Live Table Pipeline Load Error

Noopur_Nigam — Thu, 02 Jun 2022 02:17:01 GMT

Hi @Abhay Sudhakaran The job is failing at the cluster start state, hence actual DLT pipeline has not been started yet and the job failed before that.

This seems to be an issue with authorization. You may want to check the cluster policy and other authorizations to start and work with the cluster.

Re: Delta Live Table Pipeline Load Error

abhay_sud — Thu, 16 Jun 2022 13:44:41 GMT

Thank @Kaniz Fatma , I am working with our AWS team to see how we can overcome the EC2 creation restriction. Hopefully that resolves the problem.

Re: Delta Live Table Pipeline Load Error

Anonymous — Mon, 11 Jul 2022 18:07:38 GMT

Hey there @Abhay Sudhakaran

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution and mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!