topic RDD Issue in Unity Catalog in Get Started Discussions

RDD Issue in Unity Catalog

sambgp — Sat, 13 Jul 2024 10:34:10 GMT

################################################ New Code as per UC ################################################

def parse_json(df, *cols, clean=True😞

res = df

for i in cols:

if clean:

res = (

res.withColumn(

F.concat(F.lit('{"data": '), i, F.lit('}'))

)

data_str = res.limit(100).collect()[0].asDict()[i]

schema = spark.sql(f"select schema_of_json('{data_str}')").collect()[0][0]

res = res.withColumn(i, F.from_json(F.col(i), schema))

if clean:

res = res.withColumn(i, F.col(i).data)

return res

################################################ OLD Code as per Non-UC ################################################

# def parse_json(df, *cols, clean=True):

# res = df

# for i in cols:

# if clean:

# res = (

# res.withColumn(

# i,

# F.concat(F.lit('{"data": '), i, F.lit('}'))

# )

# schema = spark.read.json(res.rdd.map(lambda x: x[i])).schema

# res = res.withColumn(i, F.from_json(F.col(i), schema))

# if clean:

# res = res.withColumn(i, F.col(i).data)

# return res

previously we were using rdd.flatmap , but as rdd.flatmap doesnt supports in UC. so we modified the code and now new code we have put limit as 100 to read json data as if i dont put it throws memory error. and if i put limit many elements we get as null which shouldnt happen . its nested JSON. pls help

Re: RDD Issue in Unity Catalog

iakshaykr — Sun, 14 Jul 2024 13:08:43 GMT

@sambgp Chekc this if this can help

https://stackoverflow.com/questions/45873155/out-of-memory-error-when-collecting-data-out-of-spark-cluster ; https://stackoverflow.com/questions/26562033/how-to-set-apache-spark-executor-memory ; https://community.databricks.com/t5/data-engineering/executor-memory-increase-limitation-based-on-node-type/td-p/41077 to increase the memory and maxResultSize for drive and executor .

Re: RDD Issue in Unity Catalog

RishabhTiwari07 — Mon, 22 Jul 2024 16:50:16 GMT

Hi @sambgp ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,

Rishabh