- 697 Views
- 1 replies
- 1 kudos
2025 Data + AI World Tour Atlanta
Attending how to build Intelligent Agents at Databricks Data+AI World Tour 2025#Databricks #Data+AI #DatabricksWorldTour
- 697 Views
- 1 replies
- 1 kudos
- 1 kudos
Great to hear, @agilecoach360! Please share your learnings and experience from the event with the Community, it would be really valuable for everyone. Looking forward to your insights.
- 1 kudos
- 1583 Views
- 2 replies
- 3 kudos
I made an AI assistant for Databricks docs, let me know what you think!
Hello members of the Databricks community!I built this Ask AI chatbot/widget where I gave a custom LLM access to some of Databricks' docs to help answer technical questions for people using Databricks. I tried it on a couple of questions that resembl...
- 1583 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @matte_kapa_bul , how are you doing?First of all, congratulations on the initiative. I’ve tried to do something similar myself, and it’s very useful for locating documentation, but it doesn’t end up being very effective in solving some issues repo...
- 3 kudos
- 1610 Views
- 3 replies
- 0 kudos
Got No such file or directory error while serving the endpoint
Hello Everyone!I'm using Databricks for my MLOps learning, and I'm following the tutorial, and I got an error while serving the endpoint. I need help in this.Problem Overview:I have created a basic LightGBM model and logged it in the Unity Catalog. T...
- 1610 Views
- 3 replies
- 0 kudos
- 0 kudos
To give you a quick recap, I’ve consolidated the code into a single file for clarity. Using the Iris dataset as an example, I first create a basic model with the scikit-learn flavor. Then, I create a PyFunc wrapper from the registered model and, fina...
- 0 kudos
- 3239 Views
- 1 replies
- 1 kudos
Resolved! Addressing Memory Constraints in Scaling XGBoost and LGBM: A Comprehensive Approach for High-Volume
Scaling XGBoost and LightGBM models to handle exceptionally large datasets—those comprising billions to tens of billions of rows—presents a formidable computational challenge, particularly when constrained by the limitations of in-memory processing o...
- 3239 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @fiverrpromotion, As you mention, scaling XGBoost and LightGBM for massive datasets has its challenges, especially when trying to preserve critical training capabilities such as early stopping and handling of sparse features / high-cardinality cat...
- 1 kudos
- 2847 Views
- 1 replies
- 0 kudos
Resolved! Problem with ray train and Databricks Notebook (Strange dbutils error)
Hi everyone,I'm running some code to train a multimodal Hugging Face model with SFTTrainer and TorchTrainer to use all GPU workers. When trying to execute trainer.fit() it gives me a dbutils serialization error,even I am not using dbutils directly in...
- 2847 Views
- 1 replies
- 0 kudos
- 0 kudos
JavierS - The dbutils serialization error occurs in your code because dbutils is only available on the Databricks driver node and cannot be pickled or transferred to Spark or Ray worker nodes. This error can appear even if your code doesn't directly...
- 0 kudos
- 1270 Views
- 1 replies
- 2 kudos
Resolved! Access to Databricks partner academy
Hi Team,My company is a Databricks Partner. But I can't get registered for Databricks Partner Academy.I have followed the following steps for Partner Academy RegistrationOpen https://partner-academy.databricks.com/learn in your web browser.Click Logi...
- 1270 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @MauGomes ,Don't worry, you already did the best thing you could. Check below thread with exact same issue . The user submitted a ticket and it was resolved by service desk. So, just wait patiently for reply Solved: authorized to access https://pa...
- 2 kudos
- 4818 Views
- 2 replies
- 1 kudos
Resolved! Using merge Schema with spark.read.csv for inconsistent schemas
The Problem:A common data engineering challenge is reading a directory of CSV files where the schemas are inconsistent. For example, some files might have columns in a different order, or be missing certain columns altogether. The standard behavior o...
- 4818 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @JaydeepKhatri here are some helpful points to consider: Is this an officially supported, enhanced feature of the Databricks CSV reader? Based on internal research, this appears to be an undocumented “feature” of Spark running on Databricks. ...
- 1 kudos
- 10767 Views
- 10 replies
- 0 kudos
Who's hiring? Latest Job Postings from the Databricks Community!
More than 10,000 organizations worldwide — including Block, Comcast, Condé Nast, Rivian, Shell and over 60% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to take control of their data and put it to work with AI. Use this thre...
- 10767 Views
- 10 replies
- 0 kudos
- 0 kudos
Job Title - Engineering Data Analyst LeadCompany name- Wartsila Location - Mumbai Required years of experience- 8 years Link to apply - https://careers.wartsila.com/job-invite/46999/Job description;ResponsibilitiesTransform raw data into meaningful i...
- 0 kudos
- 3630 Views
- 6 replies
- 5 kudos
Resolved! Databricks metastore
I have configured a new databricks workspace through my personal azure account. I cannot see the metastore. Since this is my personal account, I am the admin. Can you please guide me on how to enable the metastore. Thank you.
- 3630 Views
- 6 replies
- 5 kudos
- 5 kudos
Hello @anipar Good day!!Can you please confirm again if you can see the metastore here?Log in to your Databricks workspace.Click your username in the top bar and select "Manage Account" to open the account console AZURE (https://accounts.azuredatabri...
- 5 kudos
- 2037 Views
- 1 replies
- 1 kudos
Resolved! Dashboard embed: dashboard id is missing in token claim
We're trying to embed a dashboard following the example at https://learn.microsoft.com/en-us/azure/databricks/dashboards/embedding/external-embed#step-4-use-the-example-app-to-authenticate-and-generate-tokens The embed is displayed but with an error ...
- 2037 Views
- 1 replies
- 1 kudos
- 1 kudos
Nevermind. I accidentally removed the external_viewer_id and external_value parameters. After adding them back it works as expected.
- 1 kudos
- 4764 Views
- 0 replies
- 5 kudos
Understanding Databricks Clean Rooms: Design, Access, and Cost Model
Hi everyone,I wanted to share some insights and clarifications around Databricks Clean Rooms, especially regarding how they are designed and how costs are calculated — something that can be a bit confusing when onboarding collaborators.What Are Clean...
- 4764 Views
- 0 replies
- 5 kudos
- 787 Views
- 1 replies
- 0 kudos
Resolved! Signed up for Customer Academy instead of Partner
Hello. Yesterday, I signed up for an account under my company email as a customer account. However, my intent was to sign up for a partner account and now it won't let me switch back. I sent a ticket to the academy team but have not received a reply ...
- 787 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ericmedina! For issues with Academy access, the correct step is to raise a ticket with the Databricks support team. Since you’ve already submitted one, please allow some time for them to respond.
- 0 kudos
- 5346 Views
- 6 replies
- 1 kudos
Resolved! Spark connect client and server versions should be same for executing UDFs
I am trying to execute pandas UDF in databricks. It gives me the following error on serverless compute,File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b11ff17c-9b25-4ccb-927d-06a7d1ca7221/lib/python3.11/site-packages/pyspark/sql/connect/client/core.p...
- 5346 Views
- 6 replies
- 1 kudos
- 1 kudos
@chinmay0924 - You can change the serverless client image by selecting the environment panel in a Databricks Notebook (on the right pane) or in the "Environment and Libraries" section while configuring a Databricks Job Task. The set of available serv...
- 1 kudos
- 1375 Views
- 1 replies
- 2 kudos
Resolved! How to update comments and constraints on Streaming Tables created by DLT outside the pipeline?
I have a Delta Live Tables (DLT) pipeline that creates a streaming table (e.g., dbrtdflite_catalog.demotest_silver.customer_profile).I want to update table-level and column-level comments as well as constraints on this table.When I try commands like:...
- 1375 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Nexusss7 You can add a comment in the delta live tables, either the MV or the streaming table, in the tag - @Dlt.table() @Dlt.table( comment = "Delta live tables comment" )Here is the syntax for SQL: https://docs.databricks.com/aws/en/d...
- 2 kudos
- 1372 Views
- 1 replies
- 1 kudos
Blackduck scanning on Databricks Workflow
Does anyone know if its compatible scan in blackduck your json based files from Workflows? At least, when its come to the notebook its compatible as blackduck detects python based files but i am wondering why can workflow be scanned as well.
- 1372 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @fjrodriguez , Have you found your answer already? I hope I understand your question correctly. I think Databricks Asset Bundles (or Terraform) can be a useful tool to be added here, if you haven't looked at it already. Yes, you can and should s...
- 1 kudos
