cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ramisinghl01
by New Contributor
  • 80 Views
  • 0 replies
  • 0 kudos

PYTEST: Module not found error

Hi,Apologies, as I am trying to use Pytest first time. I know this question has been raised but I went through previous answers but the issue still exists.I am following DAtabricks and other articles using pytest. My structure is simple as -tests--co...

  • 80 Views
  • 0 replies
  • 0 kudos
ismaelhenzel
by Contributor
  • 2782 Views
  • 4 replies
  • 3 kudos

Failure when deploying a custom serving endpoint LLLM

I'm currently experimenting with vector search using Databricks. Everything runs smoothly when I load the model deployed in Unity Catalog into a notebook session and ask questions using Python. However, when I attempt to serve it, I encounter a gener...

ismaelhenzel_0-1715091515103.png
  • 2782 Views
  • 4 replies
  • 3 kudos
Latest Reply
Usmanr000
New Contributor II
  • 3 kudos

"Deploying a custom serving endpoint for LLMs can be challenging, especially when handling model dependencies and scaling issues. Has anyone found a reliable workaround for deployment failures? Also, for those looking for updates on government assist...

  • 3 kudos
3 More Replies
unj1m
by New Contributor III
  • 3389 Views
  • 4 replies
  • 0 kudos

Resolved! What version of Python is used for the 16.1 runtime

I'm trying to create a spark udf for a registered model and getting:Exception: Python versions in the Spark Connect client and server are different. To execute user-defined functions, client and server should have the same minor Python version. Pleas...

  • 3389 Views
  • 4 replies
  • 0 kudos
Latest Reply
AndriusVitkausk
New Contributor III
  • 0 kudos

Does this mean that:1. A new dbx runtime comes out2. Serverless compute automatically switches to the new runtime + new python version3. Any external environments that use serverless ie, local VScode / CICD environments also need to upgrade their pyt...

  • 0 kudos
3 More Replies
nikhil_2212
by New Contributor
  • 178 Views
  • 1 replies
  • 0 kudos

Lakehouse monitoring metrices tables not created automatically.

Hello,I have an external table created in databricks unity catalog workspace and trying to "Create a monitor" for the same from quality tab.While creating the same the dashboard is getting created however the two metrices tables "profile" & "drift" a...

  • 178 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @nikhil_2212! It looks like this post duplicates the one you recently posted. A response has already been provided to the Original post. I recommend continuing the discussion in that thread to keep the conversation focused and organised.

  • 0 kudos
VijayP
by New Contributor
  • 164 Views
  • 1 replies
  • 0 kudos

Stream processing large number of JSON files and handling exception

application writes several JSON  (small)  files and the expected volumes of these files are high ( Estimate: 1 million during the peak season in a hourly window) . As per current design, these files are streamed through Spark Stream and we use autolo...

  • 164 Views
  • 1 replies
  • 0 kudos
Latest Reply
cgrant
Databricks Employee
  • 0 kudos

We have customers that read millions of files per hour+ using Databricks Auto Loader. For high-volume use cases, we recommend enabling file notification mode, which, instead of continuously performing list operations on the filesystem, uses cloud nat...

  • 0 kudos
mrstevegross
by Contributor
  • 560 Views
  • 4 replies
  • 1 kudos

Resolved! How best to measure the time-spent-waiting-for-an-instance?

I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible v...

mrstevegross_0-1741800626468.png mrstevegross_1-1741800790749.png
  • 560 Views
  • 4 replies
  • 1 kudos
Latest Reply
julieAnderson
New Contributor II
  • 1 kudos

 System Logs or Event Timings

  • 1 kudos
3 More Replies
Forssen
by New Contributor II
  • 240 Views
  • 2 replies
  • 1 kudos

Resolved! When is it time to change from ETL in notebooks to whl/py?

Hi!I would like some input/tips from the community regarding when is it time to go from a working solution in notebooks to something more "stable", like whl/py-files?What are the pros/cons with notebooks compared to whl/py?The way i structured things...

  • 240 Views
  • 2 replies
  • 1 kudos
Latest Reply
Isi
Contributor
  • 1 kudos

Hey @Forssen ,My advice:Using .py files and .whl packages is generally more secure and scalable, especially when working in a team. One of the key advantages is that code reviews and version control are much more efficient with .py files, as changes ...

  • 1 kudos
1 More Replies
LasseL
by New Contributor III
  • 221 Views
  • 1 replies
  • 0 kudos

Resolved! Deduplication with rocksdb, should old state files be deleted manually (to manage storage size)?

Hi, I have following streaming setup:I want to remove duplicates in streaming.1) deduplication strategy is defined by two fields: extraction_timestamp and hash (row wise hash)2) watermark strategy: extraction_timestamp with "10 seconds" interval--> R...

  • 221 Views
  • 1 replies
  • 0 kudos
Latest Reply
LasseL
New Contributor III
  • 0 kudos

Found solution. https://kb.databricks.com/streaming/how-to-efficiently-manage-state-store-files-in-apache-spark-streaming-applications <-- these two parameters.

  • 0 kudos
IGRACH
by New Contributor II
  • 409 Views
  • 6 replies
  • 2 kudos

Disable exiting current cell when moving around with keyboard arrows

Is there any way do disable exiting current cell when I move cursor around with arrows. When I press up arrow or down arrow it will exit the current cell and go to another cell. Can that functionally be disabled so when I hold up or down arrow key, c...

  • 409 Views
  • 6 replies
  • 2 kudos
Latest Reply
IGRACH
New Contributor II
  • 2 kudos

Is there any place where I can put this as a request.

  • 2 kudos
5 More Replies
robbdunlap
by New Contributor III
  • 3625 Views
  • 8 replies
  • 22 kudos

Turn Off Auto-reveal of Navigation Sidebar

I work with the navigation sidebar closed and use the stacked hamburgers symbol in the upper left to reveal it when I want. Now, if you mouse over the left edge of the browser window too slowly it will auto-reveal the navigation sidebar. I do not wan...

  • 3625 Views
  • 8 replies
  • 22 kudos
Latest Reply
Advika
Databricks Employee
  • 22 kudos

I've checked with the team, and there's no way to turn this off. However, they are making adjustments to improve the experience, and a fix to refine the sidebar functionality is on the way.

  • 22 kudos
7 More Replies
397973
by New Contributor III
  • 120 Views
  • 1 replies
  • 1 kudos

Is it possible to concatenate two notebooks?

I don't think it's possible but I thought I would check. I need to combine notebooks. While developing I might have code in various notebooks. I read them in with "%run".Then when all looks good I combine many cells into fewer notebooks. Is there any...

  • 120 Views
  • 1 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Hi @397973, Combining multiple notebooks into a single notebook isn't an out-of-the-box feature, but will try to combine %run commands ando output them to see if it works, sort of like: %run "/path/to/notebook1"%run "/path/to/notebook2"

  • 1 kudos
amit_jbs
by New Contributor II
  • 2069 Views
  • 5 replies
  • 2 kudos

In databricks deployment .py files getting converted to notebooks

A critical issue has arisen that is impacting our deployment planning for our client. We have encountered a challenge with our Azure CI/CD pipeline integration, specifically concerning the deployment of Python files (.py). Despite our best efforts, w...

  • 2069 Views
  • 5 replies
  • 2 kudos
Latest Reply
AGivenUser
New Contributor II
  • 2 kudos

Another option is Databricks Asset Bundles.

  • 2 kudos
4 More Replies
ramyav7796
by New Contributor II
  • 186 Views
  • 2 replies
  • 1 kudos

Databricks Lakehouse Monitoring

Hi,I am trying to implement lakehouse monitoring using Inference profile for my inference data that I have, I see that when I create the monitor, two tables get generated profile and drift, I wanted to understand how are these two tables generating a...

  • 186 Views
  • 2 replies
  • 1 kudos
Latest Reply
BigRoux
Databricks Employee
  • 1 kudos

When you create a Databricks Lakehouse Monitoring monitor with an Inference profile, the system automatically generates two metric tables: a profile metrics table and a drift metrics table. Here's how this process works: Background Processing When yo...

  • 1 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Top Kudoed Authors