cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

dbutils secrets break matplotlib in recent Databricks runtimes

fonini
New Contributor III

Using a Databricks cluster running Databricks Runtime version 15.4 LTS Beta, and with no init-scripts, the following code works fine:

import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.randn(2000, 10))
df.plot(subplots=True, figsize=(10, 20));

However, as soon as I run dbutils.secrets.get('my-scope', 'my-secret-name'), the `df.plot()` call doesn't work anymore (it simply displays nothing at all: no plot, but also no traceback or error message).

We have also observed this behavior in other recent runtimes (specifically 15.3 and 13.3lts), but this problem doesn't happen in our legacy clusters running Databricks Runtime 7.3.

This issue is the only thing keeping us from deleting all of our runtime 7.3 clusters.

4 REPLIES 4

Kaniz_Fatma
Community Manager
Community Manager

Hi @fonini, This issue doesn't occur in older runtimes, such as 7.3, which is why your legacy clusters are unaffected. To work around this, you can try re-importing matplotlib after calling dbutils.secrets.get, or use a different runtime version.

fonini
New Contributor III

Hello @Kaniz_Fatma, thank you for replying.

Do you mean that this is a known problem? Is there anywhere I can track the progress in this issue? We'd prefer using the newest runtime versions we can, in order to benefit from new features, and to avoid having to worry about Databricks deprecating older runtimes.

fonini
New Contributor III

For the record, re-importing matplotlib doesn't help (even after removing 'matplotlib', 'matplotlib.pyplot', and all other matplotlib submodules from sys.modules).

fonini
New Contributor III

OK, I understand the error now:

fonini_0-1723143931099.png

This specific secret is a small alphanumeric string. The probability of it being part of the base64 of a big PNG is quite large.

Is there any workaround for this? I know I can't turn off the redaction of secrets. Can I disable it just for html metadata or anything like that?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group