Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-03-2025 12:57 PM
Using the provided CONTEXT, it can be concluded that:
-
Creating Databricks Feature Tables using the
create_table()function is well-documented for use with PySpark DataFrames. However, passing a PySpark DataFrame generated in R usingsparklyrto thecreate_table()function via reticulate is not directly documented or supported. -
The primary challenge is compatibility between the SparkR or
sparklyrDataFrame and the PySpark DataFrame expected by the Databricks Feature Store API. This process is not explicitly described in the available documentation. -
To work around this limitation, consider creating the feature table directly within PySpark after exporting the relevant data from R. Another approach is to save the DataFrame from R using the Delta table format and load it into a PySpark DataFrame in Python before invoking the
create_table()function in the Feature Store API.
Hope this helps, Lou.