topic Does Databricks support writing Geography/Geometry data into BigQuery? in Data Engineering

Does Databricks support writing Geography/Geometry data into BigQuery?

109005 — Fri, 02 Sep 2022 07:07:38 GMT

Re: Does Databricks support writing Geography/Geometry data into BigQuery?

Prabakar — Fri, 02 Sep 2022 07:54:54 GMT

Hi @Ayushi Pandey are you looking for Geomesa library support? If so please refer this notebook.

https://www.databricks.com/notebooks/geomesa-h3-notebook.html

Re: Does Databricks support writing Geography/Geometry data into BigQuery?

109005 — Fri, 02 Sep 2022 09:15:07 GMT

No, I want to know if Databricks supports writing Geography/Geometry data back into BigQuery? When I try to write it to BQ, it shows as incorrect BYTES data.

I performed the below statement and it ran successfully in Databricks:

val wktDF = wktDFText.withColumn("bq_geography", st_geomFromWKT(col("bq_geography"))).cache

But when loading this wktDF to BigQuery, data is seen like this:

Re: Does Databricks support writing Geography/Geometry data into BigQuery?

109005 — Fri, 02 Sep 2022 09:19:01 GMT

Do let me know if this is an expected behaviour since Spark does not natively support Geography/Geometry types.

Re: Does Databricks support writing Geography/Geometry data into BigQuery?

jnb2387 — Mon, 19 Sep 2022 19:15:14 GMT

The only method I have been able to successfully write in geography data was to have the table schema already defined with a geography column and then 'APPEND' the data to it. Anytime I would try with OVERWRITE it seemingly changed the data type to string.

Re: Does Databricks support writing Geography/Geometry data into BigQuery?

109005 — Tue, 20 Sep 2022 10:37:40 GMT

Thanks a lot! That worked 🙂