- 2300 Views
- 2 replies
- 3 kudos
- 2300 Views
- 2 replies
- 3 kudos
- 3 kudos
No error, just seeing the EXPAND DISK in cluster event logs. This is just a regular spark application. I am not sure if the cloud storage matters - a spark application uses it as input and output.
- 3 kudos
- 2826 Views
- 1 replies
- 0 kudos
Databricks online store - Login to Azure SQL Database with Service Principal
I want to use Databricks Online Store with Azure SQL Database, however I am unable to autenthicate through Databricks Feature Store API. I need to use Service Principal credentials.I tried using Application ID as username and Secret as password, but ...
- 2826 Views
- 1 replies
- 0 kudos
- 0 kudos
no one?
- 0 kudos
- 4631 Views
- 2 replies
- 2 kudos
why does the client need to have git installed for auto-logging to an mlflow server running in "--serve-artifacts" mode?
I have an mlflow server with `--serve-artifacts` and with postgres as `--backend-store-uri`. The machine(container with base image python:3.9-bullseye) running the server has git installed which is available on path. I am logging from jupyter-noteboo...
- 4631 Views
- 2 replies
- 2 kudos
- 23239 Views
- 4 replies
- 1 kudos
Resolved! Set default "spark.driver.maxResultSize" from the notebook
Hello,I would like to set the default "spark.driver.maxResultSize" from the notebook on my cluster. I know I can do that in the cluster settings, but is there a way to set it by code?I also know how to do it when I start a spark session, but in my ca...
- 23239 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Maximilian Hansinger​ Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark the answer as best? If not, please tell us so we can help you.Thanks!
- 1 kudos
- 1974 Views
- 0 replies
- 0 kudos
Pymc3 on Databricks: Progress bar
Hello everybody..I am trying to run pymc3 models on Databricks (runtime 9.1) and when I start the sampling process, the progress bar is not showing. It is a bit annoying since this way I do not have any information on when the process is going to end...
- 1974 Views
- 0 replies
- 0 kudos
- 4820 Views
- 4 replies
- 2 kudos
Resolved! Cluster setup for ML work for Pandas in Spark, and vanilla Python.
My setup:Worker type: Standard_D32d_v4, 128 GB Memory, 32 Cores, Min Workers: 2, Max Workers: 8Driver type: Standard_D32ds_v4, 128 GB Memory, 32 CoresDatabricks Runtime Version: 10.2 ML (includes Apache Spark 3.2.0, Scala 2.12)I ran a snowflake quer...
- 4820 Views
- 4 replies
- 2 kudos
- 2 kudos
Hey there @Vivek Ranjan​ Checking in. If Joseph's answer helped, would you let us know and mark the answer as best? It would be really helpful for the other members to find the solution more quickly.Thanks!
- 2 kudos
- 4752 Views
- 1 replies
- 2 kudos
Resolved! How to deploy or create mlflow model as docker image with REST api endpoint within databricks?
Is it possible to create mlflow model as a docker image with REST api endpoint and use it for inferencing within databricks or hosting the image in azure container instances?
- 4752 Views
- 1 replies
- 2 kudos
- 2 kudos
@Vijeth Moudgalya​ , Hey there, we are definitely interested in making model serving easier and simpler on Databricks. There are some useful product features coming down the line - contact me at bilal dot aslam at databricks dot com if you are intere...
- 2 kudos
- 10768 Views
- 4 replies
- 2 kudos
Resolved! In Databricks, the Python kafka consumer app in notebook to Confluent Cloud having the issue captured in the Body of question: SASL/PLAIN authentication being used
kafkashaded.org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at kafkashaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:823) at kafkashaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>...
- 10768 Views
- 4 replies
- 2 kudos
- 2 kudos
@Kaniz Fatma​ I am having the same issue.%python import pyspark.sql.functions as fn from pyspark.sql.types import StringType binary_to_string = fn.udf(lambda x: str(int.from_bytes(x, byteorder='big')), StringType()) df = spark.readStream.format("...
- 2 kudos
- 3922 Views
- 3 replies
- 0 kudos
Community Edition - MLflow RestException: PERMISSION_DENIED: Model Registry is not enabled for organization
I am trying the databricks community edition . When i try to register my model it is throwing following exception.RestException: PERMISSION_DENIED: Model Registry is not enabled for organizationHow to enable the model registry ... I have tried in set...
- 3922 Views
- 3 replies
- 0 kudos
- 1192 Views
- 0 replies
- 0 kudos
Can display() plots be controlled programmatically?
Is is possible to pass parameters to the display() function to tell it to create a specific type of plots? There are scenarios, such as running notebooks in Databricks Jobs, where it is not possible to use the display() UI to select the plot type and...
- 1192 Views
- 0 replies
- 0 kudos
- 4184 Views
- 4 replies
- 2 kudos
Resolved! Feature Store : for sklearn flavored models, are timestamps fully supported?
I have created a feature table (Databricks runtime ML 10.2) that includes a timestamp column as a primary key, that is not used as a feature but as a column to join on.I have then created a model that trains from this feature table and some additiona...
- 4184 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, it did not, but at least I know they are not fully supported so a workaround is to avoid timestamps, so I suppose you can mark this as resolved
- 2 kudos
- 3130 Views
- 1 replies
- 0 kudos
Resolved! Cluster terminated in Databricks Community Edition
I've tried to start a single cluster 4 times on Databricks Community Edition today (13 March 2022). It's failed every time. Here's the first part of the output summary.```Time2022-03-13 13:59:14 EDTMessageCluster terminated.Reason:Unexpected launch f...
- 3130 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Noel Jameson​ We have some internal service interruptions due to which we had this issue. Our engineering has applied the fix and the cluster startup works as expected. Sincerely apologies for the inconvenience caused here.Regards,Darshan
- 0 kudos
- 4290 Views
- 1 replies
- 32 kudos
Databricks Roadmap Azure There are a lot of excitement new features coming in 2022. I tried to put them all on one list: Unity catalog (seems that it ...
Databricks Roadmap AzureThere are a lot of excitement new features coming in 2022. I tried to put them all on one list:Unity catalog (seems that it will exists next to hive metastore and it will be possible to migrate)Control metastore, unity creatio...
- 4290 Views
- 1 replies
- 32 kudos
- 2217 Views
- 2 replies
- 2 kudos
Resolved! DBFS for AutoML
Hi, for AutoML, I see that the data has to reside in dbfs to read and run AutoML on top of it. In my environment, dbfs is locked for security reasons. Is there a workaround or another way to access data or maybe from S3 bucket?
- 2217 Views
- 2 replies
- 2 kudos
- 2 kudos
@Silky Sharad Shah​ please look into the doc https://docs.databricks.com/data/data-sources/aws/amazon-s3.html?&_ga=2.228395418.684786035.1646666830-480220406.1638459894#access-s3-buckets-directly . this might help you.
- 2 kudos
- 2719 Views
- 2 replies
- 6 kudos
Resolved! Spark nlp on Databricks - looking for known issues/best practices
I m currently looking for information on whether Spark NLP can run fine on Databricks platform.Can someone please share - known issues/bugs encountered- any fixes or config settings required in environment- best practices to follow
- 2719 Views
- 2 replies
- 6 kudos
