Hi @shan-databricks,
This is expected behaviour on Unity Catalog shared or standard compute. The mongodb Spark data source path can fail there with UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION / SQLSTATE 0AKUC. The public docs for that are UC_COMMAND_NOT_SUPPORTED and Standard compute requirements and limitations.
If the goal is to keep using the MongoDB Spark connector, the simplest fix is to move the workload to dedicated compute.
df = (
spark.read.format("mongodb")
.option("connection.uri", MONGO_URI)
.option("database", MONGO_DB)
.option("collection", MONGO_COLLECTION)
.load()
)
display(df)
If the requirement is to stay on Unity Catalog compute, the alternative is a JDBC connection, which supports Spark read/write across Unity Catalog compute types once the JDBC driver and connection object are configured.
df = (
spark.read.format("jdbc")
.option("databricks.connection", "my_jdbc_connection")
.option("query", "select * from my_table")
.load()
)
display(df)
If your flow needs both MongoDB IO and UC-governed table operations, you can also consider splitting it into two stages. MongoDB read/write on dedicated compute, and the UC-governed transformations on the compute type best suited for those policies.
Hope this helps.
If this answer resolves your question, could you mark it as โAccept as Solutionโ? That helps other users quickly find the correct fix.
Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***