cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
admo
by • New Contributor III
  • 8605 Views
  • 5 replies
  • 7 kudos

Scaling issue for inference with a spark.mllib model

Hello,I'm writing this because I have tried a lot of different directions to get a simple model inference working with no success.Here is the outline of the job# 1 - Load the base data (~1 billion lines of ~6 columns) interaction = build_initial_df()...

Data Engineering
Reply
  • 8605 Views
  • 5 replies
  • 7 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 7 kudos

It is hard to analyze without Spark UI and more detailed information, but anyway few tips:look for data skews some partitions can be very big some small because of incorrect partitioning. You can use Spark UI to do that but also debug your code a bit...

  • 7 kudos
4 More Replies
User16826992666
by • Valued Contributor
  • 1389 Views
  • 1 replies
  • 0 kudos

Resolved! MLflow Model Serving latency expectations

What kind of latency should I expect when using the built in model serving capability in MLflow. Evaluating whether it would be a good fit for our use case

Data Engineering
Reply
  • 1389 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

What are your throughput requirements in addition to latency. Currently this is in private preview and databricks recommends this only for low throughput and non-critical applications. However, as it move towards GA, this would change. Please get in...

  • 0 kudos
User16789201666
by • Contributor II
  • 1122 Views
  • 0 replies
  • 0 kudos

Hyperopt, how to setup hyper-parameter for categorical vs numerical hyperparameter?

 hp.quniform (“quantized uniform”) or hp.qloguniform to generate integers. hp.choice is the right choice when, for example, choosing among categorical choices (which might in some situations even be integers, but not usually).https://databricks.com/b...

Data Engineering
Reply
  • 1122 Views
  • 0 replies
  • 0 kudos
Anonymous
by • Not applicable
  • 2586 Views
  • 3 replies
  • 1 kudos

What is the difference between mlflow projects and mlflow model?

 They both seem to package it. When should one use one over the other?

Data Engineering
Reply
  • 2586 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

One thing I think it's useful to point out for Databricks users is that you would typically not use MLflow Projects to describe execution of a modeling run. You would just use MLflow directly in Databricks and use Databricks notebooks to manage code ...

  • 1 kudos
2 More Replies
Labels