topic Re: Auto Loader for Shape File in Data Engineering

Auto Loader for Shape File

JD2 — Tue, 20 Jul 2021 02:16:43 GMT

Hello:

As you can see from below link, that it support 7 file formats. I am dealing with GeoSpatial Shape files and I want to know if Auto Loader can support Shape Files ???

Any help on this is greatly appreciated.

Thanks.

https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loader-gen2#common-options

avro: Avro file

binaryFile: Binary file

csv: CSV file

json: JSON file

orc: ORC file

parquet: Parquet file

* text: Text file

Re: Auto Loader for Shape File

dbkent — Mon, 27 Sep 2021 18:46:44 GMT

Hi @Jay DAVE ,

Currently, shapefiles are not a supported file-type when using auto-loader. Would you be willing to share more about your use case? I am the Product Manager responsible for Geospatial in Databricks, and I need help from customers like you to better understand what you are doing with spatial data, how often you refresh the data, how big it is, etc. Any information you can share would be great!

Kent Marten

PM Databricks

Re: Auto Loader for Shape File

-werners- — Wed, 29 Sep 2021 08:56:25 GMT

You could try to use the binary file type. But the disadvantage of this is that the content of the shape files will be put into a column, that might not be what you want.

If you absolutely want to use the autoloader, maybe some thinking outside the box can help.

What if you convert the shape files to geojson or topojson (this is not hard to do) and then use the json file format for autoloader? I have not tried this but it might just work, and in the mean time Kent can do his thing 🙂

Re: Auto Loader for Shape File

JD2 — Thu, 30 Sep 2021 04:50:32 GMT

Hello Kent:

Thanks for your reply. we receive .shape files from Satellites, LiDAR, Drones etc... consist of Geometry data along with other data points. this helps to get data insights based on geo location of assets where we do further EDA in Machine Learning.

It depends on requirements but data will be refresh weekly.

FYI ..... if you read this link, you will get better idea about all Oil & Gas companies are going to do with this .shape files.

https://scottpeters.house.gov/media-center/in-the-news/biden-signs-methane-reduction-resolution-by-rep-scott-peters

Thanks

Re: Auto Loader for Shape File

JD2 — Thu, 30 Sep 2021 04:55:48 GMT

Hello Werners:

Thanks for your reply, i agree to an extent but shape files are best way to handle geometry\geography data. converting\translating is a careful consideration keeping data integrity & corruption in a mind.

For now I am using Azure event grid & function to automate processing of shape files.

Thanks

Re: Auto Loader for Shape File

dbkent — Thu, 30 Sep 2021 16:19:45 GMT

@Jay DAVE ,

How are you solving your analysis with geo-data today?

Are you using a GIS or spatial-ETL tool?

If you could upload shapefiles, what else would you want to do with that data? Run spatial operations against it -- like spatially joining your point datasets to your boundaries?

Can I email you and ask more questions 🙂

Kent Marten

PM Databricks

Re: Auto Loader for Shape File

Ramana — Wed, 28 May 2025 17:48:41 GMT

Kent, we have the same use case. Probably, I will reach out to Bryan and ask him to schedule a call to go over our requirements.