Internal error: com.databricks.rpc.RPCResponseTooLarge, when attempting to use mosaic's st_intersects

kll — Fri, 23 Jun 2023 17:31:13 GMT

I get an exception when attempting to run the following line of code, which filters a spark DataFrame based on the geometry.

df_tx = df_zip.filter(st_intersects(st_aswkt("zip_code_geom"), tx_poly))
 
df_tx.show()

where, `tx_poly` is,

`tx_poly = shapely.wkt.dumps(tx_poly2)`

Exception:

Internal error. Attach your notebook to a different compute or restart the current compute.
 
com.databricks.rpc.RPCResponseTooLarge: rpc response (of 20982872 bytes) exceeds limit of 20971520 bytes
 
	at com.databricks.rpc.Jetty9Client$$anon$1.onContent(Jetty9Client.scala:657)
 
	at shaded.v9_4.org.eclipse.jetty.client.api.Response$Listener.onContent(Response.java:294)

Re: Internal error: com.databricks.rpc.RPCResponseTooLarge, when attempting to use mosaic's st_inter

Anonymous — Tue, 27 Jun 2023 15:20:27 GMT

I am not familiar with st_intersects, but it seems that it runs solely on the driver (as python code, not spark).
Does mosaic work in pyspark?

If not: try to use a larger driver.

topic Internal error: com.databricks.rpc.RPCResponseTooLarge, when attempting to use mosaic's st_intersects in Machine Learning

Internal error: com.databricks.rpc.RPCResponseTooLarge, when attempting to use mosaic's st_intersects

Re: Internal error: com.databricks.rpc.RPCResponseTooLarge, when attempting to use mosaic's st_inter