cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Prototype998
by New Contributor III
  • 1409 Views
  • 1 replies
  • 5 kudos

Resolved! Where can we use Broadcast variable?

best situations where we can use broadcast variables ?

  • 1409 Views
  • 1 replies
  • 5 kudos
Latest Reply
Rishabh264
Honored Contributor II
  • 5 kudos

hey @Punit Chauhan​ BV are used in the same way for RDD, DataFrame, and Dataset.When you run a Spark RDD, DataFrame jobs that has the Broadcast variables defined and used, Spark does the following.Spark breaks the job into stages that have distribute...

  • 5 kudos
nthomas
by New Contributor
  • 4838 Views
  • 5 replies
  • 0 kudos

Tips for properly using large broadcast variables?

I'm using a broadcast variable about 100 MB pickled in size, which I'm approximating with: >>> data = list(range(int(10*1e6))) >>> import cPickle as pickle >>> len(pickle.dumps(data)) 98888896Running on a cluster with 3 c3.2xlarge executors, ...

  • 4838 Views
  • 5 replies
  • 0 kudos
Latest Reply
dragoncity
New Contributor II
  • 0 kudos

The Facebook credit can be utilized by the gamers to purchase the pearls. The other route is to finished various sorts of Dragons in the Dragon Book. Dragon City Gems There are various kinds of Dragons, one is amazing, at that point you have the fund...

  • 0 kudos
4 More Replies
Labels