topic Re: Ingesting and Transforming NetCDF Data in Delta Table on Databricks Cluster in Get Started Discussions

Ingesting and Transforming NetCDF Data in Delta Table on Databricks Cluster

OlehSemeniuk — Tue, 14 Jan 2025 08:20:25 GMT

Hi,

I need to ingest and transform historical climate data into a Delta table. The data is stored in .nc format (NetCDF). To work with this format, specific C libraries for Python are required, along with particular versions of Python libraries (e.g., numpy).

On my local machine, I resolved this using Anaconda, which installed the necessary libraries (xarray, netCDF4) and handled all dependencies seamlessly.

However, I'm encountering issues when trying to achieve the same on a Databricks cluster:

Upgrading certain libraries (e.g., numpy) causes dependency conflicts, breaking the cluster's functionality.

I came across the Databricks Container Service, which seems to allow customization by using custom containers.

https://docs.databricks.com/en/compute/custom-containers.html#enable

Is this the only way to install xarray, netCDF4, and upgrade pre-installed libraries? Are there alternative approaches to handle this without compromising the cluster's stability?

Any help or guidance would be much appreciated!

Thanks!

Re: Ingesting and Transforming NetCDF Data in Delta Table on Databricks Cluster

Walter_C — Tue, 14 Jan 2025 10:14:53 GMT

Using custom containers is generally the most stable and flexible approach to ensure all dependencies are correctly managed and do not interfere with the cluster's functionality.

Re: Ingesting and Transforming NetCDF Data in Delta Table on Databricks Cluster

OlehSemeniuk — Tue, 14 Jan 2025 10:46:38 GMT

Thanks! Will proceed with custom containers then.

Re: Ingesting and Transforming NetCDF Data in Delta Table on Databricks Cluster

Walter_C — Tue, 14 Jan 2025 12:38:46 GMT

Great, please let us know in case any assistance is needed