Databricks Community

rahuja

HelloCurrently we have multiple ML models running in Production which are logging metrics and other meta-data on mlflow. I wanted to ask is it possible somehow to build Databricks dashboards on top of this data and also can this data be somehow avail...

rahuja

HelloI have written a python script that uses Databricks Rest API(s). I am trying to clone/ update an Azure Devops Repository inside databricks using Azure Service Principal. I am able to retrieve the credential_id for the service principal I am usin...

rahuja · 05-31-2024

Hello EveryoneWe are integrating Unity Catalog in our Organisation's Databricks. In our case we are planning to move our inference from Databricks to Kubernetes. In order to make the inference code use the latest registered model we need to query the...

rahuja · 05-30-2024

Hello EveryoneWe are trying to create an ML pipeline on Databricks using the famous Databricks workflows. Currently our pipeline includes having 3 major components: Data Ingestion, Model Training and Model Testing. My question is whether it is possib...

rahuja · 05-02-2024

Hello I am trying to run the SparkXGBoostRegressor and I am getting the following error:SpoilerPy4JError: An error occurred while calling o992.resourceProfileManager. Trace: py4j.security.Py4JSecurityException: Method public org.apache.spark.resource...

rahuja

Hi @ranged_coop Yes, we are using the same job compute for using different workflows. But I think different tasks are like different docker containers so that is why it becomes an issue. It would be nice if you can explain a bit about the approach yo...

rahuja

Hello @Kaniz_Fatma and @erigaud I am also having the same issue. We have multiple tasks inside databricks jobs which require to share dictionaries of dataframes amongst them. Is there any way we can do this? Initially we thought TaskValues might hel...

rahuja

Hello @Kaniz_Fatma Thanks for responding. I think you are talking about using the Python API. But we don't want that is it possible since MLFlow also uses an sql table to store metrics. To expose those tables as a part of our meta-store and build da...

rahuja

Hello @mhiltner Currently we are contemplating whether to do this or not. In my organisation this decision is based due to cost. We already have a managed Kubernetes cluster that we are paying for.

rahuja · 06-13-2024

In my case the problem was that we were trying to use SparkXGBoostRegressor and in the docs it says that it does not work on clusters with autoscaling enabled. So we just disabled autoscaling for the interactive cluster where we were testing the mode...

Databricks Community

User Stats

User Activity

Create Databricks Dashboards on MLFlow Metrics

Cloning Git Repository in Databricks via Rest API Endpoint using Azure Service principal

Accessing Unity Catalog's MLFlow model registry from outside Databricks

Sharing Output between different tasks for MLOps pipeline as a Databricks Jobs

Py4JError: An error occurred while calling o992.resourceProfileManager

Re: Pass Dataframe to child job in "Run Job" task

Re: Pass Dataframe to child job in "Run Job" task

Re: Create Databricks Dashboards on MLFlow Metrics

Re: Accessing Unity Catalog's MLFlow model registry from outside Databricks

Re: Cannot use RDD and cannot set "spark.databricks.pyspark.enablePy4JSecurity false" for