Although the PySpark documentation states that DataFrame.foreachPartition() is a shorthand for df.rdd. foreachPartition(), there is an important difference when running on Databricks shared clusters (especially with Unity Catalog and Spark Connect).

• Direct access to df.rdd is restricted on shared clusters, so df.rdd.foreachPartition() will typically fail.
• However, DataFrame.foreachPartition() is a supported DataFrame API. Spark Connect handles the execution on the server side, so the underlying RDD operations are not exposed to your notebook code.
• This is why DataFrame.foreachPartition() can work even when direct RDD access is blocked.

Can you continue using DataFrame.foreachPartition()?

Yes. If it is working in your shared cluster environment, it is a supported DataFrame API and there is no immediate need to replace it solely because it may use RDDs internally.

Should you migrate to mapInPandas() or mapInArrow()?

Consider migrating only if:

• You need to transform and return data.
• You want to leverage Arrow-based optimizations.
• Your use case aligns better with Pandas or Arrow processing.

For simple side-effect operations (such as writing data to an external system per partition), DataFrame.foreachPartition() remains a valid and commonly used approach.

In short, the fact that df.rdd is blocked does not automatically mean DataFrame.foreachPartition() is unsupported. If your workload only needs partition-level processing without returning a DataFrame, continuing to use DataFrame.foreachPartition() is perfectly reasonable.