cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

admo
by New Contributor III
  • 3419 Views
  • 5 replies
  • 7 kudos

Scaling issue for inference with a spark.mllib model

Hello,I'm writing this because I have tried a lot of different directions to get a simple model inference working with no success.Here is the outline of the job# 1 - Load the base data (~1 billion lines of ~6 columns) interaction = build_initial_df()...

  • 3419 Views
  • 5 replies
  • 7 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 7 kudos

It is hard to analyze without Spark UI and more detailed information, but anyway few tips:look for data skews some partitions can be very big some small because of incorrect partitioning. You can use Spark UI to do that but also debug your code a bit...

  • 7 kudos
4 More Replies
User16826992666
by Valued Contributor
  • 987 Views
  • 1 replies
  • 0 kudos

Resolved! MLflow Model Serving latency expectations

What kind of latency should I expect when using the built in model serving capability in MLflow. Evaluating whether it would be a good fit for our use case

  • 987 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

What are your throughput requirements in addition to latency. Currently this is in private preview and databricks recommends this only for low throughput and non-critical applications. However, as it move towards GA, this would change. Please get in...

  • 0 kudos
User16789201666
by Contributor II
  • 807 Views
  • 0 replies
  • 0 kudos

Hyperopt, how to setup hyper-parameter for categorical vs numerical hyperparameter?

 hp.quniform (“quantized uniform”) or hp.qloguniform to generate integers. hp.choice is the right choice when, for example, choosing among categorical choices (which might in some situations even be integers, but not usually).https://databricks.com/b...

  • 807 Views
  • 0 replies
  • 0 kudos
Anonymous
by Not applicable
  • 1879 Views
  • 3 replies
  • 1 kudos

What is the difference between mlflow projects and mlflow model?

 They both seem to package it. When should one use one over the other?

  • 1879 Views
  • 3 replies
  • 1 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

One thing I think it's useful to point out for Databricks users is that you would typically not use MLflow Projects to describe execution of a modeling run. You would just use MLflow directly in Databricks and use Databricks notebooks to manage code ...

  • 1 kudos
2 More Replies
Labels