How to access databricks feature store outside databricks?

[Nithin]

We are building the feature store using databricks API. Few of the machine learning engineers are using Jupyter notebooks. Is it possible to use feature store outside databricks?

[Kaniz]

Hi @ Nithin! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Kaniz]

Hi @Nithin Anil​ , The Databricks Feature Store library is available only on Databricks Runtime for Machine Learning and is accessible through notebooks and jobs.

Source

[Atanu]

Thanks true.

[CHANDAN_NANDY]

I see databricks plugin in Visual Studio. Is that of any use? I am interested because if I get the things in Visual Studio I will be able to use GitCopilot there. That's amazing.

[Kaniz]

Hi @CHANDAN NANDY​ , Databricks Connect allows you to connect your favourite IDE (Eclipse, IntelliJ, PyCharm, RStudio, Visual Studio Code), notebook server (Jupyter Notebook, Zeppelin), and other custom applications to Azure Databricks clusters.

This article explains how Databricks Connect works, walks you through the steps to get started with Databricks Connect, explains how to troubleshoot issues that may arise when using Databricks Connect, and the differences between running using Databricks Connect versus running in an Azure Databricks notebook.

[akankshii]

from databricks import feature_store

from databricks.feature_store import FeatureStoreClient,feature_table

we used to use them in databricks notebooks trying to replicate the same in pycharm however it doesn't seem to work . some say u can patch code using below link

Tips and Tricks for using Python with Databricks Connect - DEV Community

what's your suggestion on this and i read your above response it says accessible through jobs what does that mean ... did u mean trigger it through job (adf) and databricks notebook's code run on databricks cluster . How about running a job through IDE .

please explain

[Kaniz]

Hi @akanksha pandya​ ,

Note:-

Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time. Also, be aware of the limitations of Databricks Connect.

Before you begin to use Databricks Connect, you must meet the requirements and set up the client for Databricks Connect.

The Databricks Connect configuration script automatically adds the package to your project configuration.

Python 3 clusters

• When you create a PyCharm project, select Existing Interpreter. From the drop-down menu, select the Conda environment you created (see Requirements).

• Go to  Run > Edit Configurations.
• Add PYSPARK_PYTHON=python3 as an environment variable.

 

[Kaniz]

Hi @akanksha pandya​ , Did you get the solution for your proposed question, or do you need help?

[Elroczy]

@Kaniz Fatma​ I have a question regarding accessing feature tables created in databricks via DBFS.

I see that the data has to reside in dbfs to read and run the feature store on top of it. In my environment, dbfs is locked for security reasons. Is there a workaround or another way to write and read access the feature table maybe from an azure blob container?

[Kaniz]

Hi @Uc A​ , Please go through the doc. This might be helpful to you. Please let me know if this helps you.

[jose_gonzalez]

Hi @Uc A​ ,

Just a friendly follow-up. Do you still need help or @Kaniz Fatma​ 's response help you to find the solution? Please let us know.

[Elroczy]

still need help

[datariel]

Hi @Kaniz Fatma​  and @Jose Gonzalez​ ,

turning back to the original question, and considering that one of the main benefits of the Feature Store is the removal of the online/offline skew, how could I access to the features from a client application like a web app for instance, external to Databricks? Is there any way?

Thanks!

[Kaniz · Accepted solution]

Hi @Ariel Cedola​ , I repeat - The Databricks Feature Store library is available only on Databricks Runtime for Machine Learning and is accessible through notebooks and jobs.

Source