Internal error: com.databricks.rpc.RPCResponseTooLarge, when attempting to use mosaic's st_intersects
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2023 10:31 AM
I get an exception when attempting to run the following line of code, which filters a spark DataFrame based on the geometry.
df_tx = df_zip.filter(st_intersects(st_aswkt("zip_code_geom"), tx_poly))
df_tx.show()where, `tx_poly` is,
`tx_poly = shapely.wkt.dumps(tx_poly2)`
Exception:
Internal error. Attach your notebook to a different compute or restart the current compute.
com.databricks.rpc.RPCResponseTooLarge: rpc response (of 20982872 bytes) exceeds limit of 20971520 bytes
at com.databricks.rpc.Jetty9Client$$anon$1.onContent(Jetty9Client.scala:657)
at shaded.v9_4.org.eclipse.jetty.client.api.Response$Listener.onContent(Response.java:294)
Labels:
- Labels:
-
Internal error
Anonymous
Not applicable
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-27-2023 08:20 AM
I am not familiar with st_intersects, but it seems that it runs solely on the driver (as python code, not spark).
Does mosaic work in pyspark?
If not: try to use a larger driver.