- 710 Views
- 1 replies
- 0 kudos
- 710 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @SJ5 ,Probably you're using cluster with standard access mode (previously called shared).Try to use cluster with dedicated mode (previously called single user)
- 0 kudos
- 1141 Views
- 1 replies
- 0 kudos
Resolved! Auto ML training - Early Stopping (training time) / Data Split
Greetings dear community,I am using AutoML for the first time ands was wondering whether it is possible to have early stopping or incorporate any approach in my code to make the training of a model stop when the performance plateaus. Early stopping i...
- 1141 Views
- 1 replies
- 0 kudos
- 0 kudos
First question: See here for what is possible. https://docs.databricks.com/aws/en/machine-learning/automl/classification Second question: See here for what is possible. https://docs.databricks.com/gcp/en/machine-learning/automl/classification-data-pr...
- 0 kudos
- 8025 Views
- 14 replies
- 12 kudos
Resolved! ML experiment giving error - RESOURCE_DOES_NOT_EXIST
Followed the below documentation to create a ML experiment - https://docs.databricks.com/aws/en/mlflow/experimentsI created an experiment using the databricks console, then tried running the below code but getting error - getting error - RESOURCE_DOE...
- 8025 Views
- 14 replies
- 12 kudos
- 12 kudos
can you mark your own post as a solution as well @dbuser24? (would be useful for the additional steps)Appreciate you feeding back your findings.Congrats on getting it working.All the best,BS
- 12 kudos
- 1252 Views
- 1 replies
- 2 kudos
Resolved! AutoML: 403 Error error_code:"PERMISSION_DENIED"
Hello Community,I am using AutoML on my AutoML enabled cluster and have a service principle. Here is the code features_for_split = df.drop(columns=['cae_type', 'id']).select_dtypes(include=[np.number]) target_for_split = df['cae_type'] # Use same ra...
- 1252 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @spearitchmeta ,When you're using following code, you're interacting wiht dbfs and there are several limitations that apply to dbutils when it comes to interacting with Workspace files.. dbutils.fs.mkdirs(experiment_dir) It's confusing but you ...
- 2 kudos
- 2135 Views
- 2 replies
- 3 kudos
Resolved! Issue with FeatureEngineeringClient().log_model()
I am receiving a weird error when trying to log an xgboost model using feature engineering api.I was able to log the model correctly with classic mlflow.xgboost.log_model() without any issues but when I switched to feature store recommended approach ...
- 2135 Views
- 2 replies
- 3 kudos
- 3 kudos
There is a typo in the libraries versions: I was using databricks-feature-engineering version 0.13, by downgrading to databricks-feature-engineering==0.12.1 (current stable version as of today: 4th August 2025) the code above functions as expected.
- 3 kudos
- 11490 Views
- 5 replies
- 1 kudos
What's the recommended way to scale XGBoost/LGBM to datasets that don't fit in memory ?
I'm looking to scale xgboost to large datasets which won't fit in memory on a single large EC2 instance (billions to tens of billions of rows scale). I also require many of the bells & whistles of regular in-memory xgboost slash lightgbm including:Th...
- 11490 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello,I saw this post earlier this year as I was stuck something similar. I have recently managed to train XGBoost models with approximately 120 features and up to 1 200 000 000 rows using GPUs (took around 7min, with 50 boosting rounds, using 6 H100...
- 1 kudos
- 852 Views
- 1 replies
- 0 kudos
Forecasting serverless can write predicitons, compute cluster cannot ???
Hi! I have something I don't understand.... I used automl forecasting (serverless) to train a model and marked my schema edw_forecasting as output database where it saved the predictions of my best model. Awesome.However, when I try to do automl fore...
- 852 Views
- 1 replies
- 0 kudos
- 0 kudos
Did you contact your account team? @elisabethfalck Also as per the error: can you make 5 max worker nodes?
- 0 kudos
- 5573 Views
- 2 replies
- 2 kudos
Resolved! Databricks Free Edition serverless
I am using the databricks free edition and want to learn how to use ML projects in databricks. However, when I try to connect to serverless, it does not allow me to do so. The only option I have is SOL compute. Is there a way to connect to serverless...
- 5573 Views
- 2 replies
- 2 kudos
- 2 kudos
@rc2 apologies, left you hanging on the last post. Was traveling back from the library.I imported this notebook from this resource: https://docs.databricks.com/aws/en/mlflow/end-to-end-example If you look at the navigation bar on the left hand side o...
- 2 kudos
- 2690 Views
- 2 replies
- 3 kudos
Resolved! Serving Endpoint: Container image creation
Hi TeamWhenever I try to create an endpoint from a model in Databricks, the process often gets stuck at the 'Container Image Creation' step. I've tried to understand what happens during this step, but couldn't find any detailed or helpful information...
- 2690 Views
- 2 replies
- 3 kudos
- 4486 Views
- 5 replies
- 3 kudos
Resolved! This API is disabled for users without the databricks-sql-access
Running a deply on github: Run databricks bundle deploydatabricks bundle deployshell: /usr/bin/bash -e {0}env:DATABRICKS_HOST: {{HOST}}DATABRICKS_CLIENT_ID: {{ID}}DATABRICKS_CLIENT_SECRET: ***DATABRICKS_BUNDLE_ENV: prodError: This API is disabled for...
- 4486 Views
- 5 replies
- 3 kudos
- 3 kudos
Got it working, yes I see it was a little confusing at first, the workspace displayed at the top right is the account information whereas the profile icon is where you can access the workspace settings. For anyone that got as confused as I did. Thank...
- 3 kudos
- 2224 Views
- 1 replies
- 1 kudos
The inference table is not updated
Hi, I am deploying a model with following code: w = WorkspaceClient()model_cfg = {"entity_name": uc_model,"entity_version": str(version),"workload_type": "CPU","workload_size": "Small","scale_to_zero_enabled": True}ai_gateway_config = AiGatewayConfig...
- 2224 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Dharma, As mentioned in the documentation, Inference table log delivery is currently best effort, but logs are usually available within 1 hour of a request.Please try to query the inference tables after waiting for an hour. There are certain scena...
- 1 kudos
- 2723 Views
- 3 replies
- 5 kudos
Resolved! Not Able to run AutoML - RESOURCE DOES NOT EXIST ERROR
Hello,I'm new to both ML and Databricks. I'm running a Classification Experiment and getting a RESOURCE DOES NOT EXIST ERROR. It says the experiment_id does not exist. Can you help me point where to fix the error? I tried the Diagnose Error option, b...
- 2723 Views
- 3 replies
- 5 kudos
- 5 kudos
Hello Nt2good day!!If you view a stack trace and it looks similar to the following:RestException Traceback (most recent call last)File <command-XXXXXXXXXXXX>:72 mlflow.sklearn.autolog()...File /databricks/python/lib/python3.9/site-packages/mlflow/tra...
- 5 kudos
- 1327 Views
- 1 replies
- 1 kudos
Resolved! Model Inferencing
Any links, pointers to host a model in real time (similar to sagemaker endpoint in aws) - how can we host a model in DBX in real time - any documentation please?
- 1327 Views
- 1 replies
- 1 kudos
- 1 kudos
@Sachin_Amin you can find an example in our docs here: https://docs.databricks.com/aws/en/machine-learning/model-serving/model-serving-intro We also have free training courses on realtime model deployment for both classical ML (https://www.databricks...
- 1 kudos
- 1749 Views
- 1 replies
- 2 kudos
Resolved! How to choose legacy MLflow to upgrade Unity Catalog models
Hi there,I'm trying to upgrade MLflow models to Unity Catalog from legacy Models.I'm referencing this document. https://docs.databricks.com/aws/en/machine-learning/manage-model-lifecycle/upgrade-modelsBut I'm facing the error when I run `workspace_cl...
- 1749 Views
- 1 replies
- 2 kudos
- 2 kudos
I'm sorry, I figured it out myself.I have to write like this, workspace_client = MlflowClient(registry_uri="databricks")as the document: https://docs.databricks.com/gcp/en/machine-learning/manage-model-lifecycle/workspace-model-registry
- 2 kudos
- 4378 Views
- 5 replies
- 2 kudos
Python Debugger won't stop
I'm trying to use the debugger on python scripts that I'm running in my databricks workspace. The first few times i used it, it worked and stopped at my breakpoints. Since then, it just won't stop. I'm sure they are valid lines for breakpoints, but i...
- 4378 Views
- 5 replies
- 2 kudos
- 2 kudos
Thanks for the tips! This is a new cluster with DB rutime 16.4 LTS, access mode is "Dedicated". I have stopped and restarted the cluster several times and the same behavior persists. Does restarting the cluster not restart the Python kernel? If not t...
- 2 kudos
