Mongodb connection in GCP Databricks

[xavier_db]

I am trying to connect with Mongodb from databricks which is UC enabled, and both the mongodb and databricks are in same VPC, I am using the below code, df = ( spark.read.format("mongodb") .option( "connection.uri", f'''mongodb://{username}:{password}@{cluster_uri}:27017/{database}?authSource={database}&directConnection=true''' ) .option("database", database) .option("collection", table_name) .load() ) 1. when trying with shared cluster, I am getting this error "[UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): Data source v2 are not supported in Unity Catalog. SQLSTATE: 0AKUC". 2. when trying with legacy cluster, I am getting different error, 3. when trying with pandas with shared cluster it's working, but for larger datasets it's failing, 4. when trying with single user dedicated cluster the same code works. What is the difference between shared cluster and single user dedicated cluster, why it is not working with shared cluster where uc enabled, and why it is working with single user dedicated cluster with uc enabled.

[szymon_dybczak]

Hi @xavier_db ,

Standard access mode has more limitations compared to dedicate access mode. For example, look at the limitations list of standard access mode:

Standard compute requirements and limitations | Databricks on AWS

Now, compare it to dedicated access mode:

Dedicated compute requirements and limitations | Databricks on AWS

As you can see, in dedicated access mode you can do much more things. Probably mongodb connector requires access to some API which is blocked in standard access mode (maybe RDD API), but is allowed in dedicated access mode. 

That's why it works on dedicated access mode.