Connect databricks

Khaled75 — Sun, 31 Mar 2024 14:57:24 GMT

I discovered recently mlflow managed by Databricks so I'm very new to this and I need some help.

Can someone explain for me clearly the steps to do to be able to track my runs into the Databricks API.

Here are the steps I followed :

1/ Installing Databricks CLI

2/ I sewed up authentication between Databricks CLI and my Databricks workspaces according to instructions here text

I checked the file cat ~/.databrickscfg and everything is fine

3/ I'm using Pycharm and I'm writing a python script including mlflow and I want to track the runs on my Databricks workspace Here is a part of my code :

mlflow.autolog(
        log_input_examples=True,
        log_model_signatures=True,
        log_models=True,
        disable=False,
        exclusive=False,
        disable_for_unsupported_versions=True,
        silent=False
    )

    #mlflow.login()
    mlflow.set_tracking_uri("databricks")
    
    mlflow.set_experiment("mlflowAUS")
    with mlflow.start_run() as run:
        bestModel.fit(X_train, y_train)
        y_pred = bestModel.predict(X_test)
        runId = run.info.run_id
        mlflow.set_tag('mlflow.runName', datetime.datetime.now().strftime("%Y%m%d_%H%M%S"))
        mlflow.log_param("model_name", str(bestModel)[: str(bestModel).index("(")])

and if I add to my code

mlflow.login()

I got another time the same error but before the error I got : (I hided my the link with xxxx )

2024/03/30 01:40:44 INFO mlflow.utils.credentials: Successfully connected to MLflow hosted tracking server! Host: https://dbc-xxxxxx-xxxx.cloud.databricks.com.

What I don't understand why it says that I'm successfully connected after adding mlflow.login() but I still got the same error and I can't track my runs ? Please help me and I thank you in advance for the support

this code return error :

Traceback (most recent call last):
  File "/Users/kevin/PycharmProjects/Projects2024/nov23_continu_mlops_meteo/src/models/bestModel.py", line 184, in <module>
    mlflow.set_experiment("mlflowAUS")
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/tracking/fluent.py", line 142, in set_experiment
    experiment = client.get_experiment_by_name(experiment_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/tracking/client.py", line 539, in get_experiment_by_name
    return self._tracking_client.get_experiment_by_name(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 236, in get_experiment_by_name
    return self.store.get_experiment_by_name(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 323, in get_experiment_by_name
    response_proto = self._call_endpoint(GetExperimentByName, req_body)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 60, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 220, in call_endpoint
    response = verify_rest_response(response, endpoint)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kevin/opt/anaconda3/envs/env_2023/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 170, in verify_rest_response
    raise MlflowException(f"{base_msg}. Response body: '{response.text}'")
mlflow.exceptions.MlflowException: API request to endpoint was successful but the response body was not in a valid JSON format. Response body: '<!doctype html>

You can see that if I use mlflow ui everything works without any issues

topic Connect databricks in Data Engineering

Connect databricks