- 3757 Views
- 4 replies
- 7 kudos
Hi,I need to ingest 60 millions json files from S3 and have create a Delta Live Tables to ingest these data to delta table with Auto Loader. However the input rate in my DLT is always around 8 records/second no matter how many worker I add to the DLT...
- 3757 Views
- 4 replies
- 7 kudos
Latest Reply
Please consider the following:consider having driver 2 times bigger than worker,check is S3 in the same region, is communicating via the private gateway (local IPs),enable S3 transfer acceleration,in ingestion please user autoloader as described here...
3 More Replies
- 4777 Views
- 5 replies
- 4 kudos
Hello:
As you can see from below link, that it support 7 file formats. I am dealing with GeoSpatial Shape files and I want to know if Auto Loader can support Shape Files ???Any help on this is greatly appreciated.
Thanks.
https://docs.microsoft.com/...
- 4777 Views
- 5 replies
- 4 kudos
Latest Reply
You could try to use the binary file type. But the disadvantage of this is that the content of the shape files will be put into a column, that might not be what you want.If you absolutely want to use the autoloader, maybe some thinking outside the b...
4 More Replies
- 1203 Views
- 1 replies
- 0 kudos
Hello,
As you can see from below link, that it support 7 file formats. I am dealing with GeoSpatial Shape files and I want to know if Auto Loader can support Shape Files ???Any help on this is greatly appreciated.
avro: Avro filebinaryFile: Binary f...
- 1203 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Yuv Saha ,Currently, shapefiles are not a supported file-type when using auto-loader. Would you be willing to share more about your use case? I am the Product Manager responsible for Geospatial in Databricks, and I need help from customers like ...
