- 6003 Views
- 6 replies
- 7 kudos
How to save model produce by distributed training?
I am trying to save model after distributed training via the following codeimport sys from spark_tensorflow_distributor import MirroredStrategyRunner import mlflow.keras mlflow.keras.autolog() mlflow.log_param("learning_rate", 0.001) import...
- 6003 Views
- 6 replies
- 7 kudos
- 7 kudos
I think I finally worked this out.Here is the extra code to save out the model only once and from the 1st node:context = pyspark.BarrierTaskContext.get() if context.partitionId() == 0: mlflow.keras.log_model(model, "mymodel")
- 7 kudos
- 2615 Views
- 0 replies
- 0 kudos
'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'Too many sources. It cannot be more than 100'
I am getting the following error while saving a delta table in the feature storeWARNING databricks.feature_store._catalog_client_helper: Failed to record data sources in the catalog. Exception: {'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'To...
- 2615 Views
- 0 replies
- 0 kudos
- 3274 Views
- 2 replies
- 1 kudos
AutoMl Dataset too large
Hello community,i have the following problem: I am using automl to solve a regression model, but in the preprocessing my dataset is sampled to ~30% of the original amount.I am using runtime 14.2 ML Driver: Standard_DS4_v2 28GB Memory 8 coresWorker: S...
- 3274 Views
- 2 replies
- 1 kudos
- 1 kudos
I am pretty sure that i know what the problem was. I had a timestamp column (with second precision) as a feature. If they get one hot encoded, the dataset can get pretty large.
- 1 kudos
- 2374 Views
- 2 replies
- 0 kudos
Error: batch scoring with mlflow.keras flavor model
I am logging a trained keras model using the following: fe.log_model( model=model, artifact_path="wine_quality_prediction", flavor= mlflow.keras, training_set=training_set, registered_model_name=model_name )And when I call the following:predictions_...
- 2374 Views
- 2 replies
- 0 kudos
- 1465 Views
- 1 replies
- 0 kudos
BAD_REQUEST: ExperimentIds cannot be empty when checking ACLs in bulk
I encountered the error when using Databricks CE to log experiments from mlflow. It worked perfectly fine before, but now I cannot open any of my experiments. I tried clean the cookies, change the browser, and create a new account to manually create ...
- 1465 Views
- 1 replies
- 0 kudos
- 3010 Views
- 0 replies
- 0 kudos
MLFlow connection pool warning
Hi,I have a transformer model from Hugging Face I have logged to MLFlow.When I load in using mlflow.transformers.load_model I receive a bunch of warnings: WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: xxxx. Connection...
- 3010 Views
- 0 replies
- 0 kudos
- 1179 Views
- 0 replies
- 0 kudos
ApplyInPandas failing at a particular grouped item
Hello,I have a code that performs a forecast for 21k items in parallel. It looks like this: def forward_forecast(data): model = ETSModel(window_data, error='add', trend='add', seasonal=None) fitted_model = model.fit(disp=0) ...
- 1179 Views
- 0 replies
- 0 kudos
- 2518 Views
- 1 replies
- 0 kudos
Feature Store Log Model and Score Batch - env_manager
Hi Everyone. I have a couple of questions about the feature store log model and score batch. After you log a model with the feature store then use fs.score_batch is it possible to pass the env_manager to predict with the same env as training as descr...
- 2518 Views
- 1 replies
- 0 kudos
- 0 kudos
I also like to know if that works.
- 0 kudos
- 1922 Views
- 2 replies
- 1 kudos
Model Lineage with Feature Engineering is missing tables and notebooks
I am trying to track the lineage of model and tables using the FeatureEngineeringClient. The table lineage shows the relevant tables and notebooks but the model lineage shows only the model. No notebook and tables. here is my code fe = FeatureEngine...
- 1922 Views
- 2 replies
- 1 kudos
- 1 kudos
ok I realized something else. That although I used FeatureEngineeringCient, MLflow model artifact suggest I used FeatureStoreClient. Please see attachment.
- 1 kudos
- 10811 Views
- 6 replies
- 6 kudos
Displaying graphviz images in a notebook
Hi,I'm experimenting with process mining in a Databricks notebook using the OSS library PM4PY. I've been working through some tutorials and the notebook they provide on Github:https://github.com/pm4py/pm4py-core/blob/release/notebooks/3_process_disco...
- 10811 Views
- 6 replies
- 6 kudos
- 6 kudos
@Rob_S i am also in the same situation the code cell executes but no visualization how did you tackle this problem?
- 6 kudos
- 3977 Views
- 1 replies
- 0 kudos
parallel execution with applyinpandas on partitioned table
Hi,I have a job that uses df.groupby(“Country”).applyInPandas(..) to run pandas-based hyperparameter tuning in parallel for 6 countries.It runs on a cluster with 4 workers (Chosen like this because the countries’ datasets are of different sizes – so ...
- 3977 Views
- 1 replies
- 0 kudos
- 7828 Views
- 5 replies
- 2 kudos
Serving API endpoint failing
Hi Team,I registered my ML model in databricks but while trying to serve an API endpoint for the model it is failing with the following error logs.Service logs: There are currently no replicas in a running state.Build logs :Build never started - chec...
- 7828 Views
- 5 replies
- 2 kudos
- 2 kudos
@ombhuyan We currently only upload logs during the build phase to the user (i.e where we install the pip dependencies) but we don't upload logs during the pre-build phase (i.e where we download the model). That's why you may not see clear error messa...
- 2 kudos
- 9529 Views
- 6 replies
- 3 kudos
Databricks Notebook Rendering Issue: IPython.lib.display.IFrame
Similar issue here: https://stackoverflow.com/questions/71336374/randomforestclassifier-explainer-dashboard-output-in-databricks-notebook-is-notActual output – Databricks Notebook Expected Output – Jupyter Notebook Reproducible Code Example#pip insta...
- 9529 Views
- 6 replies
- 3 kudos
- 3 kudos
Hi Abhishek, I followed your steps, I am having in identifying the dashboard link. How do I figure out the first two words dbc-dp- for my cluster?
- 3 kudos
- 1394 Views
- 1 replies
- 0 kudos
Identity Resolution
Looking for best solutions for identity resolution. I already have deterministic matching. Exploring probabilistic solutions. Any advice for me?
- 1394 Views
- 1 replies
- 0 kudos
- 0 kudos
Recommend checking out Amperity. Listed on Databricks marketplace, support delta sharing and unity catalog. Patented AI approach to ID resolution https://docs.amperity.com/stitch.html
- 0 kudos
- 2471 Views
- 1 replies
- 0 kudos
Resolved! Github Datasets/Labs for Large Language Models: Application through Production is not working
I've signed up for the module for certification on Large Language Models: Application through Production.Follow the Github instructions and install the notebooks provided.Unfortunately none of the workbooks are working due to the- Badly setup file pa...
- 2471 Views
- 1 replies
- 0 kudos
- 0 kudos
No further instructions on the Read-me here: https://github.com/databricks-academy/large-language-models/tree/publishedFollowed all the setup steps, but the file paths in /include are not working fine.Why does not Databricks provide the direct links ...
- 0 kudos
