cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Ingesting and Transforming NetCDF Data in Delta Table on Databricks Cluster

OlehSemeniuk
New Contributor

Hi,

I need to ingest and transform historical climate data into a Delta table. The data is stored in .nc format (NetCDF). To work with this format, specific C libraries for Python are required, along with particular versions of Python libraries (e.g., numpy).

On my local machine, I resolved this using Anaconda, which installed the necessary libraries (xarray, netCDF4) and handled all dependencies seamlessly.

However, I'm encountering issues when trying to achieve the same on a Databricks cluster:

  1. Upgrading certain libraries (e.g., numpy) causes dependency conflicts, breaking the cluster's functionality.

I came across the Databricks Container Service, which seems to allow customization by using custom containers.

https://docs.databricks.com/en/compute/custom-containers.html#enable

Is this the only way to install xarray, netCDF4, and upgrade pre-installed libraries? Are there alternative approaches to handle this without compromising the cluster's stability?

Any help or guidance would be much appreciated!

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Walter_C
Databricks Employee
Databricks Employee

Using custom containers is generally the most stable and flexible approach to ensure all dependencies are correctly managed and do not interfere with the cluster's functionality.

View solution in original post

3 REPLIES 3

Walter_C
Databricks Employee
Databricks Employee

Using custom containers is generally the most stable and flexible approach to ensure all dependencies are correctly managed and do not interfere with the cluster's functionality.

OlehSemeniuk
New Contributor

Thanks! Will proceed with custom containers then.

Walter_C
Databricks Employee
Databricks Employee

Great, please let us know in case any assistance is needed

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group