MEMORY_ONLY not working
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-07-2023 10:25 PM
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-07-2023 11:50 PM
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2023 08:35 PM
@Ratna Chaitanya Raju Bandaru
how to convert to rdd before storing. I want to do some transformations
i am attaching the whole code
val input = sc.textFile("dbfs:/FileStore/shared_uploads/Recent9/customerorders.csv")
val mappedInput = input.map(x=>(x.split(",")(0), x.split(",")(2).toFloat))
val totalByCustomer = mappedInput.reduceByKey(_+_)
val premiumCustomers = totalByCustomer.filter(x => x._2 > 5000)
val doubledAmount = premiumCustomers.map(x=>(x._1, x._2*2)).persist(StorageLevel.MEMORY_ONLY)