Trying to create table in minio bucket using databricks.

spark.sql("create database if not exists minio_db_1 managed location 's3a://my-bucket/minio_db_1'");

I am passing the s3 configurations using spark context.

access_key = 'XXXX'

secret_key = 'XXXXXXX'

sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", access_key)

sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", secret_key)

sc._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "http://my-ip:9000")

With this config I am able to write data in s3 using

df.write.format("parquet").save("s3a://my-bucket/file-path");

But it's throwing exception when I m trying to create table/database;

spark.sql("create database if not exists minio_db_1 managed location 's3a://my-bucket/minio_db_1'");

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.nio.file.AccessDeniedException s3a://my-bucket/my-database: getFileStatus on s3a://test2/minio_db_1: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://test2.s3.us-east-1.amazonaws.com minio_db_1 {} Hadoop 3.3.4, aws-sdk-java/1.12.189 Linux/5.4.0-1093-aws OpenJDK_64-Bit_Server_VM/25.345-b01 java/1.8.0_345 scala/2.12.14 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: 6YBEAZY59EYGAEVB, Extended Request ID: o+h6YBGczQmWsnFMW8kLGi+llJ+v3ysqoz05fnNYTH901+ACgmi5x50dE2ekXbNrr3qQf81uOx8=, Cloud Provider: AWS, Instance ID: i-072d1969af3c17cb6 (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 6YBEAZY59EYGAEVB; S3 Extended Request ID: o+h6YBGczQmWsnFMW8kLGi+llJ+v3ysqoz05fnNYTH901+ACgmi5x50dE2ekXbNrr3qQf81uOx8=; Proxy: null), S3 Extended Request ID: o+h6YBGczQmWsnFMW8kLGi+llJ+v3ysqoz05fnNYTH901+ACgmi5x50dE2ekXbNrr3qQf81uOx8=:403 Forbidden)

The request should routed to the s3a endpoint, but it's routing to the generic s3 endpoint. Somehow spar.sql not honouring the spark context configurations.

Can anyone please point out the configs lacking here for table creation?