pandas issue

[gtyhchang]

We identify a potential bug in either DBFS or Pandas that when writting a dataframe using Pandas `to_csv`, `to_parquet`, `to_pickle` etc to a mounted ADLS location with read-only service principle didn't throw permission deny exceptions. However, methods like `to_html`, `to_latex`, `to_excel` Spark API and Python API correctly throw the error. See attached notebook screenshot. I'm guessing the issue is more likely on the Pandas IO implementation that interact with DBFS but would like to notify Databricks since how DBFS underlying implementation with cloud object is not clear.

[Anonymous]

@Yung-Hang Chang​ :

Thank you for bringing this to our attention. It does seem like there could be a potential issue with the way Pandas interacts with DBFS, especially when writing to a mounted ADLS location with a read-only service principal.

We suggest that you file a support ticket with Databricks support team with the details of the issue, along with the notebook screenshot, so that our team can investigate and provide a resolution. You can do this by clicking on the Help icon in the Databricks workspace and selecting "Contact Databricks Support".

In the meantime, you could consider using the methods that correctly throw the permission denied error or using a different storage location with the necessary write permissions until the issue is resolved.

[Anonymous]

Hi @Yung-Hang Chang​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!