Hello @tonykun_sg,

It looks like ADLS Gen2 might be restricting access to the data through an ACL, which is why Databricks allows access but the underlying files remain protected. Could you check with your team to temporarily enable access for testing?

Another option to consider is creating a Service Principal with a token that has access to a Databricks SQL Warehouse. The user could authenticate with the Service Principal, send a request to the warehouse, and retrieve the data—all without needing direct access to the storage.

Additionally, permissions can be refined (Unity Catalog) to allow access only to the selected tables, ensuring that the user gets access strictly to the necessary datasets.

This approach wouldn’t even require a Python script; the user could do everything through DBeaver or a similar SQL client. Let me know if this sounds like a viable workaround for your case!

🙂