'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'Too many sources. It cannot be more than 100'

[yorabhir]

I am getting the following error while saving a delta table in the feature store

WARNING databricks.feature_store._catalog_client_helper: Failed to record data sources in the catalog. Exception: {'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'Too many sources. It cannot be more than 100'}

The code:

from databricks.feature_store import FeatureStoreClient

fs = FeatureStoreClient()

feature_table = fs.create_table(

  name='feature_db.feature_delta_df',

  df= pydf,

  primary_keys=['key_1','key_2','key_3'],

  schema=pydf.schema,

  description='BB customer analytics record'

)

[Kaniz_Fatma · Accepted solution]

Hi @yorabhir, 

• Verify how many sources you’re trying to record in the catalog. If it exceeds 100, you’ll need to reduce the number of sources.
• Ensure that the feature table creation process is correctly configured. In your code snippet, you’re creating a feature table named 'feature_db.feature_delta_df'.
• Confirm that the provided pydf (presumably a DataFrame) contains the necessary data and schema for the feature table.
• Consider using partition columns when creating the feature table. Partitioning can facilitate writing and loading data. You can find more information in the Databricks documentation.
• Ensure that your Databricks Runtime version is compatible with the Feature Store. Feature Engineering in Unity Catalog requires Databricks Runtime 13.2 ML or above.
• Verify that you have the necessary privileges for creating catalogs, schemas, and feature tables. Refer to the documentation for details on creating catalogs and schemas.
  
  Investigate the points mentioned above to resolve the issue. If you need further assistance, feel free to ask! 🚀