cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Internal error: com.databricks.rpc.RPCResponseTooLarge, when attempting to use mosaic's st_intersects

kll
New Contributor III

I get an exception when attempting to run the following line of code, which filters a spark DataFrame based on the geometry.

df_tx = df_zip.filter(st_intersects(st_aswkt("zip_code_geom"), tx_poly))
 
df_tx.show()

where, `tx_poly` is,

`tx_poly = shapely.wkt.dumps(tx_poly2)`

Exception:

Internal error. Attach your notebook to a different compute or restart the current compute.
 
com.databricks.rpc.RPCResponseTooLarge: rpc response (of 20982872 bytes) exceeds limit of 20971520 bytes
 
	at com.databricks.rpc.Jetty9Client$$anon$1.onContent(Jetty9Client.scala:657)
 
	at shaded.v9_4.org.eclipse.jetty.client.api.Response$Listener.onContent(Response.java:294)

1 REPLY 1

Anonymous
Not applicable

I am not familiar with st_intersects, but it seems that it runs solely on the driver (as python code, not spark).
Does mosaic work in pyspark?

If not: try to use a larger driver.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group