Delta Live Table Pipeline Load Error
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-13-2022 04:14 PM
I am getting this error when running Delta Live Table Pipelines pulling from a source in our sandbox's local dbfs folder. It says the user is not authorized to perform this operation, whereas I am able to see the data when I run a simple select statement. Any idea why this might be happening?
Able to query the data from dbfs folder
But the DLT pipeline notebook which has the load statement to pull the same file does not work.
Error Message:
Full JSON error message:
{
"id": "6bdfbea0-d307-11ec-a13a-b20c225d589b",
"sequence": {
"control_plane_seq_no": 1651599564260008
},
"origin": {
"cloud": "AWS",
"region": "us-east-2",
"org_id": 8628124014979244,
"pipeline_id": "08d0a244-a1ee-4488-8960-14e56ad81535",
"pipeline_name": "DLT_Test",
"update_id": "549ffb7a-5fd5-49f8-813c-0eaa52725249"
},
"timestamp": "2022-05-13T21:55:36.970Z",
"message": "Update 549ffb is FAILED.",
"level": "ERROR",
"error": {
"exceptions": [
{
"message": "Failed to launch pipeline cluster 0513-215521-4w03rki9: Could not launch cluster due to cloud provider failures. aws_api_error_code: UnauthorizedOperation, aws_error_message: You are not authorized to perform this operation. Encoded authorization failur..."
}
],
"fatal": true
},
"details": {
"update_progress": {
"state": "FAILED"
}
},
"event_type": "update_progress"
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-01-2022 07:17 PM
Hi @Abhay Sudhakaran The job is failing at the cluster start state, hence actual DLT pipeline has not been started yet and the job failed before that.
This seems to be an issue with authorization. You may want to check the cluster policy and other authorizations to start and work with the cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-16-2022 06:44 AM
Thank @Kaniz Fatma , I am working with our AWS team to see how we can overcome the EC2 creation restriction. Hopefully that resolves the problem.
![](/skins/images/1C7D039E274DA4E433FB1B1A3EAE173A/responsive_peak/images/icon_anonymous_profile.png)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2022 11:07 AM
Hey there @Abhay Sudhakaran
Hope all is well!
Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution and mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)