Databricks Community

nikhilmb · 03-06-2024

Hi,We are trying to ingest zip files into Azure Databricks delta lake using COPY INTO command. There are 100+ zip files with average size of ~300MB each.Cluster configuration:1 driver: 56GB, 16 cores2-8 workers: 32GB, 8 cores (each). Autoscaling enab...

nikhilmb · 04-10-2024

Although we were able to copy the zip files onto the DB volume, we were not able to share them with any system outside of the Databricks environment. Guess delta sharing does not support sharing files that are on UC volumes.

nikhilmb · 03-14-2024

Just in the hope that this might benefit other users, we have decided to go for the good-old way of mounting cloud object store onto DBFS and then ingesting data from mounted drive into Unity Catalog-managed volume. Tried this for the 500+ zip files ...

nikhilmb · 03-11-2024

Thanks for the response.We tried all the suggestions in the post. It's still failing.I think Spark tries to unzip files during ingestion and that's where it goes out of memory. May be ingesting zip files is not supported yet. We are now exploring the...

Databricks Community

User Stats

User Activity

Error ingesting zip files: ExecutorLostFailure Reason: Command exited with code 50

Re: Error ingesting zip files: ExecutorLostFailure Reason: Command exited with code 50

Re: Error ingesting zip files: ExecutorLostFailure Reason: Command exited with code 50

Re: Error ingesting zip files: ExecutorLostFailure Reason: Command exited with code 50