topic How do i stop pyspark from outputting text in Data Engineering

How do i stop pyspark from outputting text

halox6000 — Tue, 07 May 2024 02:40:59 GMT

I am using a tqdm progress bar to monitor the amount of data records I have collected via API. I am temporarily writing them to a file in the DBFS, then uploading to a Spark DataFrame. Each time I write to a file, I get a message like 'Wrote 8873925 bytes.' How do I stop it from outputting this?

Re: How do i stop pyspark from outputting text

MathieuDB — Thu, 16 Jan 2025 14:31:11 GMT

Hello @halox6000,

You could temporarily redirect console output to a null device for these write operations.

Try this out:

@contextlib.contextmanager def silence_dbutils(): with contextlib.redirect_stdout(io.StringIO()): yield # Usage in your code with silence_dbutils(): dbutils.fs.put("/path/to/your/file.txt", "Your content here", True)