- 4069 Views
- 2 replies
- 1 kudos
How to fix this runtime error in this Databricks distributed training tutorial workbook
I am following along with this notebook found from this article. I am attempting to fine tune the model with a single node and multiple GPUs, so I run everything up to the "Run Local Training" section, but from there I skip to "Run distributed traini...
- 4069 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi AChang, have you eventually resolved the error? I've also having the same error.
- 1 kudos
- 2006 Views
- 2 replies
- 1 kudos
Resolved! import ml.dmlc.xgboost4j.scala.spark.{XGBoostEstimator, XGBoostClassificationModel}
how i can import : import com.microsoft.ml.spark.{LightGBMClassifier,LightGBMClassificationModel}import ml.dmlc.xgboost4j.scala.spark.{XGBoostEstimator, XGBoostClassificationModel} projet spark & scala in databricks
- 2006 Views
- 2 replies
- 1 kudos
- 1 kudos
XGBoostEstimator is not a member of package ml.dmlc.xgboost4j.scala.spark ?How can I resolve this error?with maven : ml.dmlc:xgboost4j-spark_2.12:2.0.3
- 1 kudos
- 723 Views
- 0 replies
- 0 kudos
XGBoostEstimator is not a member of package ml.dmlc.xgboost4j.scala.spark ?
XGBoostEstimator is not a member of package ml.dmlc.xgboost4j.scala.spark ?How can I resolve this error?
- 723 Views
- 0 replies
- 0 kudos
- 1001 Views
- 0 replies
- 0 kudos
Extract calculations naive bayes model
I have a naive Bayes ML model that takes call attributes and predicts if the caller is going to abandon the call while they are on hold waiting to speak to an agent. The model lives in Databricks ML flow, I have it registered. What I need to do is ex...
- 1001 Views
- 0 replies
- 0 kudos
- 1430 Views
- 2 replies
- 0 kudos
Unity Catalog table management with multiple teams members
Hi! How are you guys managing large teams working on the same project. Each member has their own data to save in Unity Catalog.Based on my understanding there is only two ways to manage this:1) Create an individual member schea so they can store thei...
- 1430 Views
- 2 replies
- 0 kudos
- 688 Views
- 0 replies
- 0 kudos
404 Page Not Found Error on Features page
We are facing this issue when accessing Features page. Our workspace is on AWS, ap-southeast-1.I think this is related to new feature for online tables and serverless. Is it because of online tables are not available yet in our region? If it not avai...
- 688 Views
- 0 replies
- 0 kudos
- 3726 Views
- 5 replies
- 1 kudos
Resolved! Endpoint performance questions
Hi! Had really interesting results from some endpoint performance tests I did. I set up the non-optimized endpoint with zero-cluster scaling and optimized had this feature disabled.1) Why does the non-optimized endpoint have variable response time fo...
- 3726 Views
- 5 replies
- 1 kudos
- 1 kudos
Answering Q1: 1) The variable response time is due to the first endpoint response time requiring ~180 seconds to scale to 1 cluster from 02) Can i change zero scale time from the preset 30 min?
- 1 kudos
- 1134 Views
- 0 replies
- 0 kudos
Serving a custom transformer class via a pyfunc wrapper for a pyspark recommendation model
I am trying to serve an ALS pyspark model with a custom transformer(for generating user-specific recommendations) via a pyfunc wrapper. Although I can successfully score the logged model, the serving endpoint is throwing the following error.URI '/mod...
- 1134 Views
- 0 replies
- 0 kudos
- 27677 Views
- 25 replies
- 0 kudos
Problem when serving a langchain model on Databricks
I´m trying to model serving a LLM LangChain Model and every time it fails with this messsage:[6b6448zjll] [2024-02-06 14:09:55 +0000] [1146] [INFO] Booting worker with pid: 1146[6b6448zjll] An error occurred while loading the model. You haven't confi...
- 27677 Views
- 25 replies
- 0 kudos
- 0 kudos
Hi @DataWrangler and Team.I got to solve the initial problem from some tips you gave. I used your code as base and did some modifications adapted to what I have, I mean , No UC enabled and not able to use DatabricksEmbeddings, DatabricksVectorSearch ...
- 0 kudos
- 8752 Views
- 2 replies
- 2 kudos
Resolved! Running multiple linear regressions in parallel (speeding up for loop)
Hi, I am running several linear regressions on my dataframe, in which I run a regression for every unique value in the column "item" , apply the model to a new dataset (vector_new), and at the end union the results as the loop runs. The problem is th...
- 8752 Views
- 2 replies
- 2 kudos
- 2 kudos
@Marcela Bejarano​ :One approach to speed up the process is to avoid using a loop and instead use Spark's groupBy and map functions. Here is an example:from pyspark.ml import Pipeline from pyspark.ml.feature import VectorAssembler from pyspark.ml.reg...
- 2 kudos
- 1353 Views
- 1 replies
- 0 kudos
Copy Into command to copy into delta table with predefined schema and csv file has no headers
How do i use copy into command to load 200+ tables with 50+ columns into a delta lake table with predefined schema. I am looking for a more generic approach to be handled in pyspark code.I am aware that we can pass the column expression into the sele...
- 1353 Views
- 1 replies
- 0 kudos
- 0 kudos
Does your source data have same number of columns as your target Delta tables? In that case, you can do it this way:COPY INTO my_pipe_dataFROM 's3://my-bucket/pipeData'FILEFORMAT = CSVFORMAT_OPTIONS ('mergeSchema' = 'true','delimiter' = '|','header' ...
- 0 kudos
- 4131 Views
- 2 replies
- 0 kudos
Resolved! How do I use the Copy Into command to copy data into a Delta Table? Looking for examples where you want to have a pre-defined schema
I've reviewed the COPY INTO docs here - https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-copy-into.html#examples but there's only one simple example. Looking for some additional examples that show loading data from CSV - with ...
- 4131 Views
- 2 replies
- 0 kudos
- 0 kudos
Here's an example for predefined schemaUsing COPY INTO with a predefined table schema – Trick here is to CAST the CSV dataset into your desired schema in the select statement of COPY INTO. Example below%sql CREATE OR REPLACE TABLE copy_into_bronze_te...
- 0 kudos
- 4812 Views
- 1 replies
- 0 kudos
How to download a pytorch model created via notebook and saved in a folder ?
I have created a pytorch model using databricks notebooks and saved it in a folder in workspace. MLFlow is not used.When I try to download the files from the folder it exceeds the download limit. Is there a way to download the model locally into my s...
- 4812 Views
- 1 replies
- 0 kudos
- 3054 Views
- 1 replies
- 0 kudos
Resolved! Query ML Endpoint with R and Curl
I am trying to get a prediction by querying the ML Endpoint on Azure Databricks with R. I'm not sure what is the format of the expected data. Is there any other problem with this code? Thanks!!!
- 3054 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Kaniz, I was able to find the solution. You should post this in the examples when you click "Query Endpoint"You only have code for Browser, Curl, Python, SQL. You should add a tab for RHere is the solution:library(httr)url <- "https://adb-********...
- 0 kudos
- 1985 Views
- 1 replies
- 0 kudos
Security Controls to implement on Machine Learning Persona
Hello,Hope everyone are doing well. You may be aware that we are using Table ACL enabled cluster to ensure the adequate security controls on Databricks. You may be also aware that we can not use Table enabled ACL cluster on Machine Learning Persona. ...
- 1985 Views
- 1 replies
- 0 kudos
