Yes, Pandas uses a different access mechanism compared to Spark when accessing S3. While Spark can leverage the "external location" configuration in Databricks to access S3 without explicitly specifying credentials, Pandas requires explicit AWS credentials to access S3.

You can configure credentials as follows:

• Instance Profiles: Attach an instance profile to your Databricks cluster that has the necessary permissions to access the S3 bucket. This way, the credentials are managed by AWS and are available to all libraries, including Pandas.
• Databricks Secrets: Store your AWS credentials in Databricks Secrets and configure your notebooks to use these secrets. This approach keeps your credentials secure and avoids hardcoding them in your notebooks.