cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'Too many sources. It cannot be more than 100'

yorabhir
New Contributor III

I am getting the following error while saving a delta table in the feature store

WARNING databricks.feature_store._catalog_client_helper: Failed to record data sources in the catalog. Exception: {'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'Too many sources. It cannot be more than 100'}

The code:

from databricks.feature_store import FeatureStoreClient
fs = FeatureStoreClient()
feature_table = fs.create_table(
  name='feature_db.feature_delta_df',
  df= pydf,
  primary_keys=['key_1','key_2','key_3'],
  schema=pydf.schema,
  description='BB customer analytics record'
)
1 ACCEPTED SOLUTION

Accepted Solutions

Kaniz_Fatma
Community Manager
Community Manager

Hi @yorabhir

  • Verify how many sources you’re trying to record in the catalog. If it exceeds 100, you’ll need to reduce the number of sources.
  • Ensure that the feature table creation process is correctly configured. In your code snippet, you’re creating a feature table named 'feature_db.feature_delta_df'.
  • Confirm that the provided pydf (presumably a DataFrame) contains the necessary data and schema for the feature table.
  • Consider using partition columns when creating the feature table. Partitioning can facilitate writing and loading data. You can find more information in the Databricks documentation.
  • Ensure that your Databricks Runtime version is compatible with the Feature Store. Feature Engineering in Unity Catalog requires Databricks Runtime 13.2 ML or above.
  • Verify that you have the necessary privileges for creating catalogs, schemas, and feature tables. Refer to the documentation for details on creating catalogs and schemas.

    Investigate the points mentioned above to resolve the issue. If you need further assistance, feel free to ask! 🚀

View solution in original post

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @yorabhir

  • Verify how many sources you’re trying to record in the catalog. If it exceeds 100, you’ll need to reduce the number of sources.
  • Ensure that the feature table creation process is correctly configured. In your code snippet, you’re creating a feature table named 'feature_db.feature_delta_df'.
  • Confirm that the provided pydf (presumably a DataFrame) contains the necessary data and schema for the feature table.
  • Consider using partition columns when creating the feature table. Partitioning can facilitate writing and loading data. You can find more information in the Databricks documentation.
  • Ensure that your Databricks Runtime version is compatible with the Feature Store. Feature Engineering in Unity Catalog requires Databricks Runtime 13.2 ML or above.
  • Verify that you have the necessary privileges for creating catalogs, schemas, and feature tables. Refer to the documentation for details on creating catalogs and schemas.

    Investigate the points mentioned above to resolve the issue. If you need further assistance, feel free to ask! 🚀

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group