cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Feature store feature table location

Direo
Contributor

Can Databricks feature tables be stored outside of DBFS?

3 REPLIES 3

Priyag1
Honored Contributor II

@Direo Direoโ€‹ Feature tables are delta tables it can shared among different workspaces . But you are asking outside of dbfs , what exactly is requirement and where you want to store

Anonymous
Not applicable

Hi @Direo Direoโ€‹ 

Does @Priyadarshini Gโ€‹  answer help? If it does, would you be happy to mark it as best? If it doesn't, please tell us so we can help you.

the-sab
New Contributor II

Yes, Databricks feature tables can be stored outside of Databricks File System (DBFS). You can store your feature tables in external storage systems such as Amazon S3, Azure Blob Storage, Azure Data Lake Storage, or Hadoop Distributed File System (HDFS).

To store your feature tables in external storage, you need to configure the storage system and provide the appropriate connection information when creating your Delta table. For example, when using Amazon S3, you would specify the S3 bucket path when creating the table.

Here's an example of how to create a Delta table in an Amazon S3 bucket using PySpark:

```python
from pyspark.sql import SparkSession

# Start a Spark session
spark = SparkSession.builder \
.appName("Databricks Feature Table on S3") \
.getOrCreate()

# Define a sample DataFrame
data = [("Alice", 34), ("Bob", 45), ("Cathy", 29)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)

# Write the DataFrame to a Delta table in S3
delta_table_path = "s3a://your-bucket-name/your-delta-table-path/"
df.write.format("delta").mode("overwrite").save(delta_table_path)
```

Replace `your-bucket-name` and `your-delta-table-path` with the appropriate values for your Amazon S3 bucket and desired path. Note that you need to configure your S3 authentication and ensure that you have the necessary permissions to read and write to the specified bucket.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group