topic Re: Suppress output in python notebooks in Data Engineering

Suppress output in python notebooks

PriyaV — Wed, 01 Mar 2023 22:27:45 GMT

My dilemma is this - We use PySpark to connect to external data sources via jdbc from within databricks. Every time we issue a spark command, it spits out the connection options including the username, url and password which is not advisable. So, is there a way we can suppress the outputs or maybe even redirect them to some place without printing it on the console?

I did a google search and it said that it is not possible to suppress the output from Python notebooks. But, I want to hear from the experts - is there a way, please?

Re: Suppress output in python notebooks

arun_pamulapati — Fri, 03 Mar 2023 19:49:26 GMT

Databricks recommends using Databricks secrets for sensitive information like usernames and passwords. These get redacted in the prints and logs and are not visible to other users in the workspace in case the notebook is accessible to other team members.

https://docs.databricks.com/security/secrets/index.html

Here is an example of it:

https://docs.databricks.com/security/secrets/example-secret-workflow.html

Re: Suppress output in python notebooks

Anonymous — Fri, 17 Mar 2023 05:05:14 GMT

Hi @Priya Venkatesh

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Suppress output in python notebooks

fried — Fri, 09 Jun 2023 11:47:44 GMT

You can use `%%capture` to suppress output, but I still face the same problem where the spark configuration is not correctly transferred from the setup notebook to the running notebook.

%%capture
%run ../utils/setup.ipynb

Re: Suppress output in python notebooks

Pabeggetur — Sat, 10 Jun 2023 05:46:38 GMT

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic.

youi contact hours

uber eats complaints

Re: Suppress output in python notebooks

Debayan — Thu, 02 Mar 2023 06:17:41 GMT

Hi @Priya Venkatesh ,

From notebook dropdown you can hide the result:

Also, you can try encoding it through base64.

https://docs.python.org/3/library/base64.html

Please let us know if this helps.