MEMORY_ONLY not working
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-07-2023 10:25 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-07-2023 11:50 PM
Hi @Gaurav Poojary ,
Can you please try the below as displayed in the image it is working for me without any issues.
Happy Learning!!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2023 08:35 PM
@Ratna Chaitanya Raju Bandaru
how to convert to rdd before storing. I want to do some transformations
i am attaching the whole code
val input = sc.textFile("dbfs:/FileStore/shared_uploads/Recent9/customerorders.csv")
val mappedInput = input.map(x=>(x.split(",")(0), x.split(",")(2).toFloat))
val totalByCustomer = mappedInput.reduceByKey(_+_)
val premiumCustomers = totalByCustomer.filter(x => x._2 > 5000)
val doubledAmount = premiumCustomers.map(x=>(x._1, x._2*2)).persist(StorageLevel.MEMORY_ONLY)

