I get an exception when attempting to run the following line of code, which filters a spark DataFrame based on the geometry.
df_tx = df_zip.filter(st_intersects(st_aswkt("zip_code_geom"), tx_poly))
df_tx.show()
where, `tx_poly` is,
`tx_poly = shapely.wkt.dumps(tx_poly2)`
Exception:
Internal error. Attach your notebook to a different compute or restart the current compute.
com.databricks.rpc.RPCResponseTooLarge: rpc response (of 20982872 bytes) exceeds limit of 20971520 bytes
at com.databricks.rpc.Jetty9Client$$anon$1.onContent(Jetty9Client.scala:657)
at shaded.v9_4.org.eclipse.jetty.client.api.Response$Listener.onContent(Response.java:294)