Hello,
I am attempting to configure Autoloader in File Notification mode with Delta Live Tables. I configured an instance profile, but it is not working because I immediately get AWS access denied errors. This is the same issue that is referenced here
After a bunch of debugging, I've determined that the issue is caused by the "Access Mode" being set to "Shared".
To test this, I created a new compute cluster with the correct instance profile and Access Mode set to "No Isolation Shared". I then connected to that computer cluster in a Python Notebook and I had no problem making AWS API calls through boto3.
I then ran this same experiment with another compute cluster, but this time with the Access Mode set to "Shared". I double checked that the instance profile was configured properly. This time when I connected to the compute cluster via a Python Notebook, all of my AWS commands failed with "invalid credentials" errors.
The only difference between the two consecutive tests was the Access Mode.
Therefore, it appears that Delta Live Tables need to be configured with a Cluster Policy containing an Access Mode of "No isolation Shared". However, that does not seem possible.
I've tried to update the `data_security_mode` JSON property when creating a new cluster policy. However, I then am faced with a validation error when refreshing my Delta Live Table which says that the `data_security_mode` key is not supported.
Is there a way to give the Delta Live Tables access to instance profiles?
Thanks so much for your time!
-Jared