I am following the guide for tuning a model with Optuna and MlflowSparkStudy. My compute is configured with autoscaling enabled, with 1–2 Spark workers, each with 8 cores and 32 GB of memory. I set n_jobs=2 and trials=100, in mlflow_study.optimize()....
I have a job with the following base parameters. I use dbutils.widgets.get to access the parameter.What is the best practice for developing with dbutils, especially when working locally in vs code or during CI? For example, IntelliSense and linting t...
What is the best way to aggregate a map across rows? In the below, The agg results would be red: 4, green 7, blue: 10. This can be achieved using explode wondering if there is a better way. %sql
with cte as (
select
1 as id
, map('red', 1, 'green...
Hi Everyone. I have a couple of questions about the feature store log model and score batch. After you log a model with the feature store then use fs.score_batch is it possible to pass the env_manager to predict with the same env as training as descr...
