How do i stop pyspark from outputting text

halox6000
New Contributor III

‎05-06-2024 07:40 PM

I am using a tqdm progress bar to monitor the amount of data records I have collected via API. I am temporarily writing them to a file in the DBFS, then uploading to a Spark DataFrame. Each time I write to a file, I get a message like 'Wrote 8873925 bytes.' How do I stop it from outputting this?

0 Kudos

MathieuDB
Databricks Employee
Databricks Employee

‎01-16-2025 06:31 AM

Hello @halox6000,

You could temporarily redirect console output to a null device for these write operations.

Try this out:

@contextlib.contextmanager
def silence_dbutils():
    with contextlib.redirect_stdout(io.StringIO()):
        yield

# Usage in your code
with silence_dbutils():
    dbutils.fs.put("/path/to/your/file.txt", "Your content here", True)
0 Kudos