- 1474 Views
- 3 replies
- 0 kudos
Got No such file or directory error while serving the endpoint
Hello Everyone!I'm using Databricks for my MLOps learning, and I'm following the tutorial, and I got an error while serving the endpoint. I need help in this.Problem Overview:I have created a basic LightGBM model and logged it in the Unity Catalog. T...
- 1474 Views
- 3 replies
- 0 kudos
- 0 kudos
To give you a quick recap, I’ve consolidated the code into a single file for clarity. Using the Iris dataset as an example, I first create a basic model with the scikit-learn flavor. Then, I create a PyFunc wrapper from the registered model and, fina...
- 0 kudos
- 3101 Views
- 1 replies
- 1 kudos
Resolved! Addressing Memory Constraints in Scaling XGBoost and LGBM: A Comprehensive Approach for High-Volume
Scaling XGBoost and LightGBM models to handle exceptionally large datasets—those comprising billions to tens of billions of rows—presents a formidable computational challenge, particularly when constrained by the limitations of in-memory processing o...
- 3101 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @fiverrpromotion, As you mention, scaling XGBoost and LightGBM for massive datasets has its challenges, especially when trying to preserve critical training capabilities such as early stopping and handling of sparse features / high-cardinality cat...
- 1 kudos
- 2706 Views
- 1 replies
- 0 kudos
Resolved! Problem with ray train and Databricks Notebook (Strange dbutils error)
Hi everyone,I'm running some code to train a multimodal Hugging Face model with SFTTrainer and TorchTrainer to use all GPU workers. When trying to execute trainer.fit() it gives me a dbutils serialization error,even I am not using dbutils directly in...
- 2706 Views
- 1 replies
- 0 kudos
- 0 kudos
JavierS - The dbutils serialization error occurs in your code because dbutils is only available on the Databricks driver node and cannot be pickled or transferred to Spark or Ray worker nodes. This error can appear even if your code doesn't directly...
- 0 kudos
- 1042 Views
- 1 replies
- 2 kudos
Resolved! Access to Databricks partner academy
Hi Team,My company is a Databricks Partner. But I can't get registered for Databricks Partner Academy.I have followed the following steps for Partner Academy RegistrationOpen https://partner-academy.databricks.com/learn in your web browser.Click Logi...
- 1042 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @MauGomes ,Don't worry, you already did the best thing you could. Check below thread with exact same issue . The user submitted a ticket and it was resolved by service desk. So, just wait patiently for reply Solved: authorized to access https://pa...
- 2 kudos
- 4148 Views
- 2 replies
- 1 kudos
Resolved! Using merge Schema with spark.read.csv for inconsistent schemas
The Problem:A common data engineering challenge is reading a directory of CSV files where the schemas are inconsistent. For example, some files might have columns in a different order, or be missing certain columns altogether. The standard behavior o...
- 4148 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @JaydeepKhatri here are some helpful points to consider: Is this an officially supported, enhanced feature of the Databricks CSV reader? Based on internal research, this appears to be an undocumented “feature” of Spark running on Databricks. ...
- 1 kudos
- 9454 Views
- 10 replies
- 0 kudos
Who's hiring? Latest Job Postings from the Databricks Community!
More than 10,000 organizations worldwide — including Block, Comcast, Condé Nast, Rivian, Shell and over 60% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to take control of their data and put it to work with AI. Use this thre...
- 9454 Views
- 10 replies
- 0 kudos
- 0 kudos
Job Title - Engineering Data Analyst LeadCompany name- Wartsila Location - Mumbai Required years of experience- 8 years Link to apply - https://careers.wartsila.com/job-invite/46999/Job description;ResponsibilitiesTransform raw data into meaningful i...
- 0 kudos
- 3309 Views
- 6 replies
- 5 kudos
Resolved! Databricks metastore
I have configured a new databricks workspace through my personal azure account. I cannot see the metastore. Since this is my personal account, I am the admin. Can you please guide me on how to enable the metastore. Thank you.
- 3309 Views
- 6 replies
- 5 kudos
- 5 kudos
Hello @anipar Good day!!Can you please confirm again if you can see the metastore here?Log in to your Databricks workspace.Click your username in the top bar and select "Manage Account" to open the account console AZURE (https://accounts.azuredatabri...
- 5 kudos
- 1814 Views
- 1 replies
- 1 kudos
Resolved! Dashboard embed: dashboard id is missing in token claim
We're trying to embed a dashboard following the example at https://learn.microsoft.com/en-us/azure/databricks/dashboards/embedding/external-embed#step-4-use-the-example-app-to-authenticate-and-generate-tokens The embed is displayed but with an error ...
- 1814 Views
- 1 replies
- 1 kudos
- 1 kudos
Nevermind. I accidentally removed the external_viewer_id and external_value parameters. After adding them back it works as expected.
- 1 kudos
- 3565 Views
- 0 replies
- 5 kudos
Understanding Databricks Clean Rooms: Design, Access, and Cost Model
Hi everyone,I wanted to share some insights and clarifications around Databricks Clean Rooms, especially regarding how they are designed and how costs are calculated — something that can be a bit confusing when onboarding collaborators.What Are Clean...
- 3565 Views
- 0 replies
- 5 kudos
- 693 Views
- 1 replies
- 0 kudos
Resolved! Signed up for Customer Academy instead of Partner
Hello. Yesterday, I signed up for an account under my company email as a customer account. However, my intent was to sign up for a partner account and now it won't let me switch back. I sent a ticket to the academy team but have not received a reply ...
- 693 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ericmedina! For issues with Academy access, the correct step is to raise a ticket with the Databricks support team. Since you’ve already submitted one, please allow some time for them to respond.
- 0 kudos
- 4968 Views
- 6 replies
- 1 kudos
Resolved! Spark connect client and server versions should be same for executing UDFs
I am trying to execute pandas UDF in databricks. It gives me the following error on serverless compute,File /local_disk0/.ephemeral_nfs/envs/pythonEnv-b11ff17c-9b25-4ccb-927d-06a7d1ca7221/lib/python3.11/site-packages/pyspark/sql/connect/client/core.p...
- 4968 Views
- 6 replies
- 1 kudos
- 1 kudos
@chinmay0924 - You can change the serverless client image by selecting the environment panel in a Databricks Notebook (on the right pane) or in the "Environment and Libraries" section while configuring a Databricks Job Task. The set of available serv...
- 1 kudos
- 1197 Views
- 1 replies
- 2 kudos
Resolved! How to update comments and constraints on Streaming Tables created by DLT outside the pipeline?
I have a Delta Live Tables (DLT) pipeline that creates a streaming table (e.g., dbrtdflite_catalog.demotest_silver.customer_profile).I want to update table-level and column-level comments as well as constraints on this table.When I try commands like:...
- 1197 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Nexusss7 You can add a comment in the delta live tables, either the MV or the streaming table, in the tag - @Dlt.table() @Dlt.table( comment = "Delta live tables comment" )Here is the syntax for SQL: https://docs.databricks.com/aws/en/d...
- 2 kudos
- 1296 Views
- 1 replies
- 1 kudos
Blackduck scanning on Databricks Workflow
Does anyone know if its compatible scan in blackduck your json based files from Workflows? At least, when its come to the notebook its compatible as blackduck detects python based files but i am wondering why can workflow be scanned as well.
- 1296 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @fjrodriguez , Have you found your answer already? I hope I understand your question correctly. I think Databricks Asset Bundles (or Terraform) can be a useful tool to be added here, if you haven't looked at it already. Yes, you can and should s...
- 1 kudos
- 9182 Views
- 5 replies
- 1 kudos
Databricks Job Failure + Service now Integration
Hi Team,Could you please suggest how to raise the service now ticket, in case of Databricks job failure?Regards ,Phanindra
- 9182 Views
- 5 replies
- 1 kudos
- 1 kudos
JSON response from Databricks to ServiceNow, doesn't give enough details. Is there a way we can send customized json with additional details.
- 1 kudos
- 5404 Views
- 8 replies
- 0 kudos
Unable to install the Databricks in AWS
I have followed the instructions given in the following link https://docs.databricks.com/en/admin/workspace/quick-start.html#use-aws-quickstart-to-create-a-databricks-workspace%22There was an error in CFT while deploying the workspace. Please help[ER...
- 5404 Views
- 8 replies
- 0 kudos
- 0 kudos
Hi @rseshadri Please check the following two points:STS Endpoint Activation: Confirm whether the STS endpoint for the us-west-2 region is active. You can verify this from the IAM account settings.Resource Limits: Ensure that the limits for resources ...
- 0 kudos
