HI all,I am trying to implement MLOps project using https://github.com/databricks/mlops-stacks repo.I have created azure databricks with Premium (+ Role-based access controls) (Click to change) and following bundle creation and deploy using uRL: http...
How are bagging and boosting different when you use them in real machine-learning projects?
The practical differences between bagging and boosting mostly come down to how they build models and how they handle errors:Model Training Approach:Bagging (Bootstrap Aggregating): Builds multiple models in parallel using random subsets of the data. ...
How can I make these people smarter or faster so the final answer is better?
Improving the performance of a Random Forest model on Databricks is usually about data quality, feature engineering, and hyperparameter tuning. Some tips:Feature Engineering:Create meaningful features and remove irrelevant ones.Encode categorical var...
How can I build my own PyTorch machine-learning model and train it faster on Databricks by using multiple machines/GPUs instead of just one?
@Suheb , You may look at the torch distributor. It provides multiple distributed training options, including single-node with multiple-GPU training and multi-node training. Below are the references for you. https://docs.databricks.com/aws/en/machine-...
Hello,I have created a vector search index for a delta table with 1,400 rows. Using this vector index to find matching records on a table with 52M records with the query below ran for 20hrs and failed with: 'HTTP request failed with status: {"error_c...
Hi @RodrigoE ,Your LATERAL subquery calls the Vector Search function once for every row of the 52M-row table, which results in tens of millions of remote calls to the Vector Search endpoint—this is not a nice pattern and will be extremely slow leadin...
When building machine-learning models in Databricks, how should I prepare and transform my data so the model can learn better?
Hi, this is quite a general question, I've put together a list of bullets that will help you in the right direction: Focus on organized storage, flexible transformations, and making features easy to reuse and discover. Use Unity Catalog for govern...
Hello,I am creating a vector search index and selected Compute embeddings for a delta table with 19M records. Delta table has only two columns: ID (selected as index) and Name (selected for embedding). Embedding model is databricks-gte-large-en.Ind...
Your recommendation addressed the issue. Followed the instructions and index initialization took only 8 hours - thank you!
I am new to MLflow and Databricks. How can I begin using MLflow inside Databricks to track and manage my machine learning models?
Hi @Suheb , MLFlow is already pre installed in ML runtime. The question is very vague. You can follow the below documentations to get started with MLFlow on databricks. 1) https://www.databricks.com/product/managed-mlflow2) https://docs.databricks.co...
How do you keep your machine-learning files, notebooks, and code properly organized in Databricks?
Hey @Suheb , I teach a lot of our machine learning training, and over time I’ve talked with many students, customers, and partners about how they approach this. The answers are all over the map, which tells you there’s no single “golden rule” that fi...
Hello!I’m trying to add a tool — Azure Databricks Genie — in Microsoft Copilot Studio for my agent, but I’m running into some difficulties. Is it possible to establish this connection using a Pro cluster, or does it only work with a serverless cluste...
I'm afraid I don't have much further suggestions. I'd suggest you raise a ticket with Microsoft on this.
What should you do when your dataset is uneven—some values appear too many times and others appear very few times—while working in Databricks?
Hi @Suheb ,Refer to really good guide prepared by Databricks team. When you have a skewed dataset the primary things you can do are following:1. Filter skewed values2. Apply Skew hints3. AQE skew optimization4. SaltingMuch detailed description of abo...
There was a model which was migrated from workspace model registry to unity catalog. At the time of initial creation of that model, dependencies to other databricks serving endpoints were configured using "DatabricksServiceEndpoint" config in mlflow....
Workspace model registry worked with workspace-scoped serving endpoints. UC models and UC serving endpoints use metastore-wide semantics and different lookup rules. The saved path inside the model metadata still points to workspace-level endpoints th...
Hello, We have multiple data bricks instances each represents an environment dev,qa,rel,prod etc.. We developed a model in the dev workspace and registered in the UC model registry using mlflow. Now, we are trying to find a best way to deploy this r...
You can use UC's centralized model registry and MLflow’s copy APIs. If all target workspaces attach to the same Unity Catalog metastore, reference and promote models via their 3‑level UC names; use MLflow’s copy_model_version to “copy” the exact arti...
QuestionIt would be great if you could recommend how I go about solving the below problem. I haven't been able to find much help online. A. Background:A1. I have to text manipulation using python (like concatenation , convert to spacy doc , get verbs...
I have to process data for n number of devices which is sending data in every 5 seconds.I have a similar scenario where I have to take last 3 hours of data and process it for all the devices for some key parameters. Now if I am doing it sequentially ...
Hello Team,I hope you are doing well.I am a student currently exploring Databricks and learning how to work with the "ai parse document" function. While experimenting, I encountered a couple of issues related to text extraction from images inside PDF...
Hello @rajcoder! This post appears to duplicate the one you recently posted. A response has already been provided to your recent post. I recommend continuing the discussion in that thread to keep the conversation focused and organised.
