Databricks Community

Unable to install SynapseML on clusters

I would like to run a distributed training using LightGBM but I cannot install SynapseML. I have tried doing so on a few different clusters (note: our clusters are running on AWS, not sure if that matters. Also, I am running the Databricks ML Runtime...

databricks-connect error when executing sparkml

I use databricks-connect, and spark jobs related spark dataframe works good. But, when I trigger spark ml code, I am getting errors.For example, after executing in the code: https://docs.databricks.com/_static/notebooks/gbt-regression.htmlpipelineMod...

Serverless Inference Setup Error

Hello, I am using Azure Databricks Premium and am an Admin on the Workspace. I am trying to create a Serving Endpoint for a registered model created with MLFlow. I can make a traditional endpoint without issues, but when I try to make a serverless en...

DBFS REST API - unable to access or upload experiment artifacts - permission denied

Hello,we are trying to achieve artifacts upload to MLflow experiments via REST API. (We have an edge case when we need to do that)But if we try to use DBFS API to upload an artifact, we are not allowed. Always ends up with:`PERMISSION_DENIED: No oper...

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outs...

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outside of Databricks. While serving the model via REST API is ideal for external use cases, it is recom...

Pushing SparkNLP Model on Mlflow

Hello Everyone, I am trying to load a SparkNLP (link for more details about the model if required) from Mlflow Registry. To this end, I have followed one tutorial and implemented below codes:import mlflow.pyfunc   class LangDetectionModel(mlflow.pyfu...

Can I use Feature tables in feature store to augment the original input dataset for AutoML forecasting problems

How far does model size and lag impact distributed inference ?

Hello !I was wondering how impactful a model's size of inference lag was in a distributed manner.With tools like Pandas Iterator UDFs or mlflow.pyfunc.spark_udf() we can make it so models are loaded only once per worker, so I would tend to say that m...

Enable programmatically writing to files

Hi everyone. I'm training a time series forecasting model in Azure Databricks. When I try to parallelize, it gives me this error:I have Contributor permission on the Databricks service, on the Azure Portal, and I'm an admin inside the Databricks work...

Don’t miss out! Data + AI Summit early bird pricing ends soon Register by February 28 to take advantage of our early bird discount. Join thousands of ...

Don’t miss out! Data + AI Summit early bird pricing ends soonRegister by February 28 to take advantage of our early bird discount. Join thousands of data engineers, data scientists and data analysts from around the world at this year’s Data + AI Summ...

Access Denied 403 error when trying to access data in S3 with dlt pipeline using configured and working instance profile and mounted bucket

I can read all of my s3 data without any issues after configuring my cluster with an instance profile however when I try to run the following dlt decorator it gives me an access denied error. Are there some other IAM tweaks I need to make for delta? ...

Model serving with Serverless Real-Time Inference - How could I call the endpoint with json file consisted of raw text that need to be transformed and get the prediction?

Hi!I want to call the generated endpoint with a json file consisted of texts directly, could this endpoint take the raw texts, transform the texts into vectors and then output the prediction?Is there a way to support so?Thanks in advance!!!

INVALID_STATE: Databricks could not access keyvault

Hi Team,Update: We are using Unity Catalog workspace. Also we are using RBAC model.I am able to create a secret scope and able to list the scope in a notebook usingdbutils.secrets.list("<scopename>")But when I try get the secret value using value = d...

Are UDFs necessary for applying models from ML libraries at scale ?

Hello,I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method. Spark MLlib models don't need this k...