cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to access databricks feature store outside databricks?

Nithin
New Contributor II

We are building the feature store using databricks API. Few of the machine learning engineers are using Jupyter notebooks. Is it possible to use feature store outside databricks?

1 ACCEPTED SOLUTION

Accepted Solutions

Kaniz
Community Manager
Community Manager

Hi @Ariel Cedola​ , I repeat - The Databricks Feature Store library is available only on Databricks Runtime for Machine Learning and is accessible through notebooks and jobs.

Source

View solution in original post

14 REPLIES 14

Kaniz
Community Manager
Community Manager

Hi @ Nithin! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

Kaniz
Community Manager
Community Manager

Hi @Nithin Anil​ , The Databricks Feature Store library is available only on Databricks Runtime for Machine Learning and is accessible through notebooks and jobs.

Source

Atanu
Esteemed Contributor
Esteemed Contributor

Thanks true.

CHANDAN_NANDY
New Contributor III

I see databricks plugin in Visual Studio. Is that of any use? I am interested because if I get the things in Visual Studio I will be able to use GitCopilot there. That's amazing.

Kaniz
Community Manager
Community Manager

Hi @CHANDAN NANDY​ , Databricks Connect allows you to connect your favourite IDE (Eclipse, IntelliJ, PyCharm, RStudio, Visual Studio Code), notebook server (Jupyter Notebook, Zeppelin), and other custom applications to Azure Databricks clusters.

This article explains how Databricks Connect works, walks you through the steps to get started with Databricks Connect, explains how to troubleshoot issues that may arise when using Databricks Connect, and the differences between running using Databricks Connect versus running in an Azure Databricks notebook.

Screenshot 2022-01-31 at 2.11.07 PMScreenshot 2022-01-31 at 2.11.30 PM

akankshii
New Contributor II

from databricks import feature_store

from databricks.feature_store import FeatureStoreClient,feature_table

we used to use them in databricks notebooks trying to replicate the same in pycharm however it doesn't seem to work . some say u can patch code using below link

Tips and Tricks for using Python with Databricks Connect - DEV Community

what's your suggestion on this and i read your above response it says accessible through jobs what does that mean ... did u mean trigger it through job (adf) and databricks notebook's code run on databricks cluster . How about running a job through IDE .

please explain

Kaniz
Community Manager
Community Manager

Hi @akanksha pandya​ ,

Note:-

Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time. Also, be aware of the limitations of Databricks Connect.

Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.

The Databricks Connect configuration script automatically adds the package to your project configuration.

Python 3 clusters

  • When you create a PyCharm project, select Existing Interpreter. From the drop-down menu, select the Conda environment you created (see Requirements).

image

  • Go to  Run > Edit Configurations.
  • Add PYSPARK_PYTHON=python3 as an environment variable.

image 

Kaniz
Community Manager
Community Manager

Hi @akanksha pandya​ , Did you get the solution for your proposed question, or do you need help?

Elroczy
New Contributor II

@Kaniz Fatma​ I have a question regarding accessing feature tables created in databricks via DBFS.

I see that the data has to reside in dbfs to read and run the feature store on top of it. In my environment, dbfs is locked for security reasons. Is there a workaround or another way to write and read access the feature table maybe from an azure blob container?

Kaniz
Community Manager
Community Manager

Hi @Uc A​ , Please go through the doc. This might be helpful to you. Please let me know if this helps you.

Hi @Uc A​ ,

Just a friendly follow-up. Do you still need help or @Kaniz Fatma​ 's response help you to find the solution? Please let us know.

still need help

datariel
New Contributor II

Hi @Kaniz Fatma​  and @Jose Gonzalez​ ,

turning back to the original question, and considering that one of the main benefits of the Feature Store is the removal of the online/offline skew, how could I access to the features from a client application like a web app for instance, external to Databricks? Is there any way?

Thanks!

Kaniz
Community Manager
Community Manager

Hi @Ariel Cedola​ , I repeat - The Databricks Feature Store library is available only on Databricks Runtime for Machine Learning and is accessible through notebooks and jobs.

Source

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!