topic Re: MEMORY_ONLY not working in Data Engineering

MEMORY_ONLY not working

gpzz — Sun, 08 Jan 2023 06:25:09 GMT

val doubledAmount = premiumCustomers.map(x=>(x._1, x._2*2)).persist(StorageLevel.MEMORY_ONLY)

error: not found: value StorageLevel

Chaitanya_Raju — Sun, 08 Jan 2023 07:50:06 GMT

Hi @Gaurav Poojary ,

Can you please try the below as displayed in the image it is working for me without any issues.

Happy Learning!!

gpzz — Fri, 13 Jan 2023 04:35:32 GMT

@Ratna Chaitanya Raju Bandaru

how to convert to rdd before storing. I want to do some transformations

i am attaching the whole code

val input = sc.textFile("dbfs:/FileStore/shared_uploads/Recent9/customerorders.csv")

val mappedInput = input.map(x=>(x.split(",")(0), x.split(",")(2).toFloat))

val totalByCustomer = mappedInput.reduceByKey(_+_)

val premiumCustomers = totalByCustomer.filter(x => x._2 > 5000)

val doubledAmount = premiumCustomers.map(x=>(x._1, x._2*2)).persist(StorageLevel.MEMORY_ONLY)