I want read a json from IO variable using PySpark.
My code using pandas:
io = BytesIO()
ftp.retrbinary('RETR '+ file_name, io.write)
io.seek(0)
# With pandas
df = pd.read_json(io)
What I tried using PySpark, but don't work:
io = BytesIO()
ftp.retrbinary('RETR '+ file_name, io.write)
io.seek(0)
df = spark.read\
.format("binaryfile")\ # I already tried with format "json"
.option('inferSchema', 'true')\
.option('header', 'true')\
.json(io)
Obs: Is not possible save in a file and read that after