Databricks Community

kunduruanil

Hi everyone,I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).Does autologging behave differently depending on wh...

kunduruanil

@AdamIH123 Did you try this?from Synapse. ml.lightgbm import LightGBMClassifierConvert a Spark Pandas DataFrame to a standard PySpark DataFrame.spark_df = spark_pandas_df.to_spark()Train distributed LightGBMlgbm = LightGBMClassifier(learningRate=0.1,...

kunduruanil

@VibinRoy_C Great thought process!Regarding your second question on Native Patterns:I highly recommend building an ML model for anomaly detection based on previous runs. You can leverage Databricks MLflow to train a model that predicts pipeline fail...

kunduruanil

@Akash_Wadhankar amazing work, kudos to you. "THE BRICK LEARNING"

kunduruanil

@infinitylearnin, the above podcast link was changed; it's not working !!

kunduruanil

Great end-to-end project! To take it to the next level, consider exploring these native Databricks capabilities.Lakehouse Monitoring: Track data drift and automate model retraining when performance drops.Model Serving: Deploy your models behind serve...

Databricks Community

User Stats

User Activity

Difference between Workspace and Unity Catalog experiments when using MLflow autologging?

Re: Tuning with Optuna and MlflowSparkStudy

Re: Databricks-Native AI Agent for Job Incident Detection, RCA & Safe Remediation

Re: Learn Databricks AI medium article series for fellow learners.

Re: Data practitioner in AI Era

Re: End-to-End Streaming NLP Pipeline with GDELT, Azure Data Factory, ADLS Gen2 and Databricks