- 560 Views
- 1 replies
- 1 kudos
Resolved! How to install whl from volume for databricks_cluster_policy via terraform.
I would expect resource "databricks_cluster_policy" "cluster_policy" { name = var.policy_name libraries { Volumes { whl = "/Volumes/bronze/config/python.wheel-1.0.3-9-py3-none-any.whl" }}}to work but terraform doesnt recognize "volum...
- 560 Views
- 1 replies
- 1 kudos
- 1 kudos
This workedresource "databricks_cluster_policy" "cluster_policy" { name = var.policy_name libraries { whl = "/Volumes/bronze/config/python.wheel-1.0.3-9-py3-none-any.whl" }}
- 1 kudos
- 1653 Views
- 1 replies
- 0 kudos
Cannot get tracing to work on genai app deployed on databricks
Hi, I have a gradio app that is deployed on databricks. The app is coming from this example provided by databricks. The app works fine, but when I want to add tracing I cannot get it to work. I keep getting the errormlflow.exceptions.MlflowException...
- 1653 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @MisterT , In our docs, it is mentioned we use MLflow 3(major upgrade) with GenAI monitoring enabled. Each agent endpoint is assigned an MLflow experiment, and log agent traces from the endpoint to that experiment in real-time. Internally an MLF...
- 0 kudos
- 1348 Views
- 3 replies
- 1 kudos
Ai Query Prompt Token and Completition token
HiI would like to know how can I get the Completition token and Prompt token quantity when using Ai_Query?Thanks
- 1348 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @Andreyai good day!!For AI_queries, we have documentation from databricks. : https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query I am 100% sure you will get better insights from the documentations. But I have something for...
- 1 kudos
- 486 Views
- 1 replies
- 1 kudos
2025 Data + AI World Tour Atlanta
Attending how to build Intelligent Agents at Databricks Data+AI World Tour 2025#Databricks #Data+AI #DatabricksWorldTour
- 486 Views
- 1 replies
- 1 kudos
- 1 kudos
Great to hear, @agilecoach360! Please share your learnings and experience from the event with the Community, it would be really valuable for everyone. Looking forward to your insights.
- 1 kudos
- 1046 Views
- 2 replies
- 3 kudos
I made an AI assistant for Databricks docs, let me know what you think!
Hello members of the Databricks community!I built this Ask AI chatbot/widget where I gave a custom LLM access to some of Databricks' docs to help answer technical questions for people using Databricks. I tried it on a couple of questions that resembl...
- 1046 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @matte_kapa_bul , how are you doing?First of all, congratulations on the initiative. I’ve tried to do something similar myself, and it’s very useful for locating documentation, but it doesn’t end up being very effective in solving some issues repo...
- 3 kudos
- 1216 Views
- 3 replies
- 0 kudos
Got No such file or directory error while serving the endpoint
Hello Everyone!I'm using Databricks for my MLOps learning, and I'm following the tutorial, and I got an error while serving the endpoint. I need help in this.Problem Overview:I have created a basic LightGBM model and logged it in the Unity Catalog. T...
- 1216 Views
- 3 replies
- 0 kudos
- 0 kudos
To give you a quick recap, I’ve consolidated the code into a single file for clarity. Using the Iris dataset as an example, I first create a basic model with the scikit-learn flavor. Then, I create a PyFunc wrapper from the registered model and, fina...
- 0 kudos
- 2718 Views
- 1 replies
- 1 kudos
Resolved! Addressing Memory Constraints in Scaling XGBoost and LGBM: A Comprehensive Approach for High-Volume
Scaling XGBoost and LightGBM models to handle exceptionally large datasets—those comprising billions to tens of billions of rows—presents a formidable computational challenge, particularly when constrained by the limitations of in-memory processing o...
- 2718 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @fiverrpromotion, As you mention, scaling XGBoost and LightGBM for massive datasets has its challenges, especially when trying to preserve critical training capabilities such as early stopping and handling of sparse features / high-cardinality cat...
- 1 kudos
- 2394 Views
- 1 replies
- 0 kudos
Resolved! Problem with ray train and Databricks Notebook (Strange dbutils error)
Hi everyone,I'm running some code to train a multimodal Hugging Face model with SFTTrainer and TorchTrainer to use all GPU workers. When trying to execute trainer.fit() it gives me a dbutils serialization error,even I am not using dbutils directly in...
- 2394 Views
- 1 replies
- 0 kudos
- 0 kudos
JavierS - The dbutils serialization error occurs in your code because dbutils is only available on the Databricks driver node and cannot be pickled or transferred to Spark or Ray worker nodes. This error can appear even if your code doesn't directly...
- 0 kudos
- 751 Views
- 1 replies
- 2 kudos
Resolved! Access to Databricks partner academy
Hi Team,My company is a Databricks Partner. But I can't get registered for Databricks Partner Academy.I have followed the following steps for Partner Academy RegistrationOpen https://partner-academy.databricks.com/learn in your web browser.Click Logi...
- 751 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @MauGomes ,Don't worry, you already did the best thing you could. Check below thread with exact same issue . The user submitted a ticket and it was resolved by service desk. So, just wait patiently for reply Solved: authorized to access https://pa...
- 2 kudos
- 2896 Views
- 2 replies
- 1 kudos
Resolved! Using merge Schema with spark.read.csv for inconsistent schemas
The Problem:A common data engineering challenge is reading a directory of CSV files where the schemas are inconsistent. For example, some files might have columns in a different order, or be missing certain columns altogether. The standard behavior o...
- 2896 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @JaydeepKhatri here are some helpful points to consider: Is this an officially supported, enhanced feature of the Databricks CSV reader? Based on internal research, this appears to be an undocumented “feature” of Spark running on Databricks. ...
- 1 kudos
- 7561 Views
- 10 replies
- 0 kudos
Who's hiring? Latest Job Postings from the Databricks Community!
More than 10,000 organizations worldwide — including Block, Comcast, Condé Nast, Rivian, Shell and over 60% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to take control of their data and put it to work with AI. Use this thre...
- 7561 Views
- 10 replies
- 0 kudos
- 0 kudos
Job Title - Engineering Data Analyst LeadCompany name- Wartsila Location - Mumbai Required years of experience- 8 years Link to apply - https://careers.wartsila.com/job-invite/46999/Job description;ResponsibilitiesTransform raw data into meaningful i...
- 0 kudos
- 2698 Views
- 6 replies
- 5 kudos
Resolved! Databricks metastore
I have configured a new databricks workspace through my personal azure account. I cannot see the metastore. Since this is my personal account, I am the admin. Can you please guide me on how to enable the metastore. Thank you.
- 2698 Views
- 6 replies
- 5 kudos
- 5 kudos
Hello @anipar Good day!!Can you please confirm again if you can see the metastore here?Log in to your Databricks workspace.Click your username in the top bar and select "Manage Account" to open the account console AZURE (https://accounts.azuredatabri...
- 5 kudos
- 1461 Views
- 1 replies
- 1 kudos
Resolved! Dashboard embed: dashboard id is missing in token claim
We're trying to embed a dashboard following the example at https://learn.microsoft.com/en-us/azure/databricks/dashboards/embedding/external-embed#step-4-use-the-example-app-to-authenticate-and-generate-tokens The embed is displayed but with an error ...
- 1461 Views
- 1 replies
- 1 kudos
- 1 kudos
Nevermind. I accidentally removed the external_viewer_id and external_value parameters. After adding them back it works as expected.
- 1 kudos
- 2034 Views
- 0 replies
- 4 kudos
Understanding Databricks Clean Rooms: Design, Access, and Cost Model
Hi everyone,I wanted to share some insights and clarifications around Databricks Clean Rooms, especially regarding how they are designed and how costs are calculated — something that can be a bit confusing when onboarding collaborators.What Are Clean...
- 2034 Views
- 0 replies
- 4 kudos
- 537 Views
- 1 replies
- 0 kudos
Resolved! Signed up for Customer Academy instead of Partner
Hello. Yesterday, I signed up for an account under my company email as a customer account. However, my intent was to sign up for a partner account and now it won't let me switch back. I sent a ticket to the academy team but have not received a reply ...
- 537 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ericmedina! For issues with Academy access, the correct step is to raise a ticket with the Databricks support team. Since you’ve already submitted one, please allow some time for them to respond.
- 0 kudos
