cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Issues with using Databricks-Connect and Petastorm

YSF
New Contributor III

Has anyone successfully used Petastorm + Databricks-Connect + Delta Lake?

The use case is being able to use DeltaLake as a data store regardless of whether I want to use the databricks workspace or not for my training tasks.

I'm using a cloud-hosted jupyterlab environment(in Paperspace), and trying to use Petastorm + Databricks Connect.

What I'm trying to do:

  • Connect to cluster via databricks-connect
  • Read in data from delta lake using a databricks spark cluster
  • Use Petastorm to convert the dataframe into a pytorch ready object

The exact same code, on the same cluster works when using the databricks notebook environment. But when running the `make_spark_converter()` function in my hosted jupyterlab environment it throws me a "Unable to infer schema" error. Even though if I check the `.schema` attribute of the dataframe I'm giving it, it shows me a spark compatible schema.

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

I would not definitely use Databricks-Connect in production.

View solution in original post

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

I would not definitely use Databricks-Connect in production.

YSF
New Contributor III

because its janky or why? I don't need it for customer facing production. More so for if I'm using my own HPC or local workstation, but I want to access data from delta lake. Figured it was easier/preferable to setting up my own spark environment locally. I'm paying for databricks might as well get the benefits of the runtime.

Can you elaborate on your answer?

Kaniz
Community Manager
Community Manager

Hi @Yusuf Khan​ , Please go through the link here.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!