Databricks Community

Khaled75 · ‎03-31-2024

I discovered recently mlflow managed by Databricks so I'm very new to this and I need some help.

Can someone explain for me clearly the steps to do to be able to track my runs into the Databricks API.

Here are the steps I followed :

1/ Installing Databricks CLI

2/ I sewed up authentication between Databricks CLI and my Databricks workspaces according to instructions here text

I checked the file cat ~/.databrickscfg and everything is fine

3/ I'm using Pycharm and I'm writing a python script including mlflow and I want to track the runs on my Databricks workspace Here is a part of my code :

mlflow.autolog(
        log_input_examples=True,
        log_model_signatures=True,
        log_models=True,
        disable=False,
        exclusive=False,
        disable_for_unsupported_versions=True,
        silent=False
    )

    #mlflow.login()
    mlflow.set_tracking_uri("databricks")
    
    mlflow.set_experiment("mlflowAUS")
    with mlflow.start_run() as run:
        bestModel.fit(X_train, y_train)
        y_pred = bestModel.predict(X_test)
        runId = run.info.run_id
        mlflow.set_tag('mlflow.runName', datetime.datetime.now().strftime("%Y%m%d_%H%M%S"))
        mlflow.log_param("model_name", str(bestModel)[: str(bestModel).index("(")])

and if I add to my code

mlflow.login()

I got another time the same error but before the error I got : (I hided my the link with xxxx )

2024/03/30 01:40:44 INFO mlflow.utils.credentials: Successfully connected to MLflow hosted tracking server! Host: https://dbc-xxxxxx-xxxx.cloud.databricks.com.

What I don't understand why it says that I'm successfully connected after adding mlflow.login() but I still got the same error and I can't track my runs ? Please help me and I thank you in advance for the support

this code return error :

Traceback (most recent call last):
  File "/Users/kevin/PycharmProjects/Projects2024/nov23_continu_mlops_meteo/src/models/bestModel.py", line 184, in <module>
    mlflow.set_experiment("mlflowAUS")
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/tracking/fluent.py", line 142, in set_experiment
    experiment = client.get_experiment_by_name(experiment_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/tracking/client.py", line 539, in get_experiment_by_name
    return self._tracking_client.get_experiment_by_name(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 236, in get_experiment_by_name
    return self.store.get_experiment_by_name(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 323, in get_experiment_by_name
    response_proto = self._call_endpoint(GetExperimentByName, req_body)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 60, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 220, in call_endpoint
    response = verify_rest_response(response, endpoint)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 170, in verify_rest_response
    raise MlflowException(f"{base_msg}. Response body: '{response.text}'")
mlflow.exceptions.MlflowException: API request to endpoint was successful but the response body was not in a valid JSON format. Response body: '<!doctype html>

You can see that if I use mlflow ui everything works without any issues

Kaniz_Fatma · ‎04-02-2024

Hi @Khaled75,

The specific error message you provided is related to fetching the experiment by name. It’s essential to understand the exact error message. Can you share the complete error text?
Confirm that your Databricks authentication is working correctly. You mentioned that you’ve successfully connected to the MLflow hosted tracking server, but let’s double-check.
Make sure your Databricks account has the necessary permissions to create and manage experiments.
Confirm that MLflow is correctly installed on your Databricks Runtime ML clusters. If not, install the mlflow library following the instructions provided in the Databricks documentation.
Make sure you’ve installed the correct packages for Python, R, or Scala, depending on your use case.