DLT Pipeline Issue with New Version of Advanced Data Engineering with Databricks training

[Artman23]

I am currently taking the new Advanced Data Engineering with Databricks training. I am stuck in the Demo: Auto Load to Bronze section of Streaming ETL Patterns with DLT. Looks to be an access issue to AWS as the pipeline errors out with the following:

DataPlaneException: [DLT ERROR CODE: CLUSTER_LAUNCH_FAILURE.CLIENT_ERROR] Failed to launch pipeline cluster 0115-031151-am1n1gme with termination code AWS_AUTHORIZATION_FAILURE and termination type CLIENT_ERROR: Could not launch cluster due to cloud provider authorization failure. databricks_error_message: Failure happened when talking to AWS, AWS API error code: UnauthorizedOperation AWS error message: Yo...

Has anyone had this issue with the new version of Advanced Data Engineering with Databricks training?

[Kaniz_Fatma]

Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).

[Kaniz_Fatma]

Hi, Could you please paste the error stack here?



 

[AlvaroCM]

Hello Kaniz Fatma,

I'm new at databricks and I have the exact same error after I created an External Location and attached my Notebook script to my Delta Live Tables pipeline. When I press "validate," it throws that error at the "waiting for resources" step in the DLT graph.

This is my DLT status workflow:

This is the msg log error:
DataPlaneException: [DLT ERROR CODE: CLUSTER_LAUNCH_FAILURE.CLIENT_ERROR] Failed to launch pipeline cluster aaaaa-bbbbb-cccc with termination code AWS_AUTHORIZATION_FAILURE and termination type CLIENT_ERROR: Could not launch cluster due to cloud provider authorization failure. databricks_error_message: Failure happened when talking to AWS, AWS API error code: AccessDenied AWS error message: User: arn:a...

My guess is that I'm missing some additional cluster authorization in AWS (IAM) for DLT that I couldn't find in the Databricks documentation.

Thanks,

Alvaro