I am getting this error when running Delta Live Table Pipelines pulling from a source in our sandbox's local dbfs folder. It says the user is not authorized to perform this operation, whereas I am able to see the data when I run a simple select statement. Any idea why this might be happening?
Able to query the data from dbfs folder
But the DLT pipeline notebook which has the load statement to pull the same file does not work.
Full JSON error message:
{
"id": "6bdfbea0-d307-11ec-a13a-b20c225d589b",
"sequence": {
"control_plane_seq_no": 1651599564260008
},
"origin": {
"cloud": "AWS",
"region": "us-east-2",
"org_id": 8628124014979244,
"pipeline_id": "08d0a244-a1ee-4488-8960-14e56ad81535",
"pipeline_name": "DLT_Test",
"update_id": "549ffb7a-5fd5-49f8-813c-0eaa52725249"
},
"timestamp": "2022-05-13T21:55:36.970Z",
"message": "Update 549ffb is FAILED.",
"level": "ERROR",
"error": {
"exceptions": [
{
"message": "Failed to launch pipeline cluster 0513-215521-4w03rki9: Could not launch cluster due to cloud provider failures. aws_api_error_code: UnauthorizedOperation, aws_error_message: You are not authorized to perform this operation. Encoded authorization failur..."
}
],
"fatal": true
},
"details": {
"update_progress": {
"state": "FAILED"
}
},
"event_type": "update_progress"
}
