- 10493 Views
- 7 replies
- 6 kudos
Databricks runtime version Error
Hello,I'm following courses on the Databricks academy and using for that purpose the Databricks Community edition using a runtime 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) and I believe it can't be changedI'm following the Data engineering c...
- 10493 Views
- 7 replies
- 6 kudos
- 6 kudos
I was facing the same error. This could be resolved by adding the version that you are currently working with in the config function present in '_common' notebook in the "Includes' folder. (This was the case of my folder structure that I downloaded f...
- 6 kudos
- 2651 Views
- 3 replies
- 4 kudos
Are UDFs necessary for applying models from ML libraries at scale ?
Hello,I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method. Spark MLlib models don't need this k...
- 2651 Views
- 3 replies
- 4 kudos
- 4 kudos
MlLib is in the maintenance model and udf is not used by creating model in most cases
- 4 kudos
- 19305 Views
- 5 replies
- 6 kudos
Resolved! Access the environment variable from the custom container base cluster
Hi Databricks Community, I want to set environment variables for all clusters in my workspace. The goal is to the have environment variable, available in all notebooks executed on the cluster.The environment variable is generated in global init scrip...
- 19305 Views
- 5 replies
- 6 kudos
- 6 kudos
Thanks @Lukasz Lu​ - that worked for me as well. When I used the following script:#!/bin/bash echo MY_TEST_VAR=value1 | tee -a /etc/environment >> /databricks/spark/conf/spark-env.shfor non-docker clusters, MY_TEST_VAR shows up twice in ` /databrick...
- 6 kudos
- 687 Views
- 0 replies
- 1 kudos
docs.azure.cn
https://docs.azure.cn/en-us/databricks/_static/notebooks/mlflow/mlflow-end-to-end-example-azure.htmlI imported the above notebook and try in Databricks community, but those subplots for Box plots are giving me errors as below:AttributeError ...
- 687 Views
- 0 replies
- 1 kudos
- 1582 Views
- 2 replies
- 0 kudos
API limit on mlflow.tracking.client.MlflowClient.list_run_infos method?
I'm trying out managed MLflow on Databricks Community edition, with tracking data saved on Databricks and artifacts saved on my own AWS S3 bucket. I created one experiment and logged 768 runs in the experiment. When I try to get the list of the runs ...
- 1582 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @jae baak​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos