- 6319 Views
- 2 replies
- 2 kudos
why does the client need to have git installed for auto-logging to an mlflow server running in "--serve-artifacts" mode?
I have an mlflow server with `--serve-artifacts` and with postgres as `--backend-store-uri`. The machine(container with base image python:3.9-bullseye) running the server has git installed which is available on path. I am logging from jupyter-noteboo...
- 6319 Views
- 2 replies
- 2 kudos
- 27695 Views
- 4 replies
- 1 kudos
Resolved! Set default "spark.driver.maxResultSize" from the notebook
Hello,I would like to set the default "spark.driver.maxResultSize" from the notebook on my cluster. I know I can do that in the cluster settings, but is there a way to set it by code?I also know how to do it when I start a spark session, but in my ca...
- 27695 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Maximilian Hansinger​ Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark the answer as best? If not, please tell us so we can help you.Thanks!
- 1 kudos
- 2432 Views
- 0 replies
- 0 kudos
Pymc3 on Databricks: Progress bar
Hello everybody..I am trying to run pymc3 models on Databricks (runtime 9.1) and when I start the sampling process, the progress bar is not showing. It is a bit annoying since this way I do not have any information on when the process is going to end...
- 2432 Views
- 0 replies
- 0 kudos
- 6071 Views
- 4 replies
- 2 kudos
Resolved! Cluster setup for ML work for Pandas in Spark, and vanilla Python.
My setup:Worker type: Standard_D32d_v4, 128 GB Memory, 32 Cores, Min Workers: 2, Max Workers: 8Driver type: Standard_D32ds_v4, 128 GB Memory, 32 CoresDatabricks Runtime Version: 10.2 ML (includes Apache Spark 3.2.0, Scala 2.12)I ran a snowflake quer...
- 6071 Views
- 4 replies
- 2 kudos
- 2 kudos
Hey there @Vivek Ranjan​ Checking in. If Joseph's answer helped, would you let us know and mark the answer as best? It would be really helpful for the other members to find the solution more quickly.Thanks!
- 2 kudos
- 5938 Views
- 1 replies
- 2 kudos
Resolved! How to deploy or create mlflow model as docker image with REST api endpoint within databricks?
Is it possible to create mlflow model as a docker image with REST api endpoint and use it for inferencing within databricks or hosting the image in azure container instances?
- 5938 Views
- 1 replies
- 2 kudos
- 2 kudos
@Vijeth Moudgalya​ , Hey there, we are definitely interested in making model serving easier and simpler on Databricks. There are some useful product features coming down the line - contact me at bilal dot aslam at databricks dot com if you are intere...
- 2 kudos
- 13582 Views
- 4 replies
- 2 kudos
Resolved! In Databricks, the Python kafka consumer app in notebook to Confluent Cloud having the issue captured in the Body of question: SASL/PLAIN authentication being used
kafkashaded.org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at kafkashaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:823) at kafkashaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>...
- 13582 Views
- 4 replies
- 2 kudos
- 2 kudos
@Kaniz Fatma​ I am having the same issue.%python import pyspark.sql.functions as fn from pyspark.sql.types import StringType binary_to_string = fn.udf(lambda x: str(int.from_bytes(x, byteorder='big')), StringType()) df = spark.readStream.format("...
- 2 kudos
- 4983 Views
- 3 replies
- 0 kudos
Community Edition - MLflow RestException: PERMISSION_DENIED: Model Registry is not enabled for organization
I am trying the databricks community edition . When i try to register my model it is throwing following exception.RestException: PERMISSION_DENIED: Model Registry is not enabled for organizationHow to enable the model registry ... I have tried in set...
- 4983 Views
- 3 replies
- 0 kudos
- 1560 Views
- 0 replies
- 1 kudos
Can display() plots be controlled programmatically?
Is is possible to pass parameters to the display() function to tell it to create a specific type of plots? There are scenarios, such as running notebooks in Databricks Jobs, where it is not possible to use the display() UI to select the plot type and...
- 1560 Views
- 0 replies
- 1 kudos
- 5696 Views
- 4 replies
- 2 kudos
Resolved! Feature Store : for sklearn flavored models, are timestamps fully supported?
I have created a feature table (Databricks runtime ML 10.2) that includes a timestamp column as a primary key, that is not used as a feature but as a column to join on.I have then created a model that trains from this feature table and some additiona...
- 5696 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, it did not, but at least I know they are not fully supported so a workaround is to avoid timestamps, so I suppose you can mark this as resolved
- 2 kudos
- 4051 Views
- 1 replies
- 0 kudos
Resolved! Cluster terminated in Databricks Community Edition
I've tried to start a single cluster 4 times on Databricks Community Edition today (13 March 2022). It's failed every time. Here's the first part of the output summary.```Time2022-03-13 13:59:14 EDTMessageCluster terminated.Reason:Unexpected launch f...
- 4051 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Noel Jameson​ We have some internal service interruptions due to which we had this issue. Our engineering has applied the fix and the cluster startup works as expected. Sincerely apologies for the inconvenience caused here.Regards,Darshan
- 0 kudos
- 5448 Views
- 1 replies
- 32 kudos
Databricks Roadmap Azure There are a lot of excitement new features coming in 2022. I tried to put them all on one list: Unity catalog (seems that it ...
Databricks Roadmap AzureThere are a lot of excitement new features coming in 2022. I tried to put them all on one list:Unity catalog (seems that it will exists next to hive metastore and it will be possible to migrate)Control metastore, unity creatio...
- 5448 Views
- 1 replies
- 32 kudos
- 3066 Views
- 2 replies
- 2 kudos
Resolved! DBFS for AutoML
Hi, for AutoML, I see that the data has to reside in dbfs to read and run AutoML on top of it. In my environment, dbfs is locked for security reasons. Is there a workaround or another way to access data or maybe from S3 bucket?
- 3066 Views
- 2 replies
- 2 kudos
- 2 kudos
@Silky Sharad Shah​ please look into the doc https://docs.databricks.com/data/data-sources/aws/amazon-s3.html?&_ga=2.228395418.684786035.1646666830-480220406.1638459894#access-s3-buckets-directly . this might help you.
- 2 kudos
- 3813 Views
- 2 replies
- 6 kudos
Resolved! Spark nlp on Databricks - looking for known issues/best practices
I m currently looking for information on whether Spark NLP can run fine on Databricks platform.Can someone please share - known issues/bugs encountered- any fixes or config settings required in environment- best practices to follow
- 3813 Views
- 2 replies
- 6 kudos
- 12452 Views
- 4 replies
- 0 kudos
Resolved! I am saving a new feature table to the Databricks feature store, and it won't write the data sources of the tables used to create the feature table, because they are Hive tables that point to Azure Data Lake Storage Gen1 Delta tables
My notebook is pulling in Hive tables from DBFS, that point to ADLS Gen1 file locations for their data (Delta tables), creating the feature table as a data frame within the notebook, then calling on the feature store client to save down the feature t...
- 12452 Views
- 4 replies
- 0 kudos
- 0 kudos
@Jack Watson​ Could you please confirm the write is succeeding ? If yes, as per my understanding This is a warning for some validation that we will be removing shortly. We’ll likely remove the validation which save the data source.Thanks.
- 0 kudos
- 31964 Views
- 2 replies
- 4 kudos
Resolved! Problem with spinning up a cluster on a new workspace
Error: Please check network connectivity from the data plane to the control plane.{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-0457092c), status: INSTANCE_INITIALIZING, workerEnvId:...
- 31964 Views
- 2 replies
- 4 kudos
- 4 kudos
Can you please get the system logs from AWS EC2 console as soon the cluster fails - System Logs for the failed instance will be accessible from the AWS console up to an hour after the shutdown.AWS console clears the references of terminated clusters ...
- 4 kudos
