Databricks Community

admo · 03-17-2022

Hello,I'm writing this because I have tried a lot of different directions to get a simple model inference working with no success.Here is the outline of the job# 1 - Load the base data (~1 billion lines of ~6 columns) interaction = build_initial_df()...

admo · 03-22-2022

The philosophy for the job would be something like this in Scala : feature_dataset.foreachPartition { block => block.grouped(10000).foreach { chunk => run_inference_and_write_to_db(chunk) }Would you know how to do this with pyspark and rdds ?

admo · 03-22-2022

Hello,Thanks for checking, unfortunately no. I believe that my core issue is about not being able to properly set the size of the processed chunks of data given my cluster memory.My job looks like these few map operations : prepare data <200 columns>...

admo · 03-17-2022

Thanks for the help.skew : I did not focus on this point, i'll have a lookspark.sql.shuffle.partitions : already donebigger driver : already donethere is indeed an udf used for the prediction score retrieval, i will have a look tooUpdate : skew : the...

Databricks Community

User Stats

User Activity

Scaling issue for inference with a spark.mllib model

Re: Scaling issue for inference with a spark.mllib model

Re: Scaling issue for inference with a spark.mllib model

Re: Scaling issue for inference with a spark.mllib model