topic Re: dbutils secrets break matplotlib in recent Databricks runtimes in Administration & Architecture

dbutils secrets break matplotlib in recent Databricks runtimes

fonini — Fri, 02 Aug 2024 21:18:28 GMT

Using a Databricks cluster running Databricks Runtime version 15.4 LTS Beta, and with no init-scripts, the following code works fine:

import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.randn(2000, 10))
df.plot(subplots=True, figsize=(10, 20));

However, as soon as I run dbutils.secrets.get('my-scope', 'my-secret-name'), the `df.plot()` call doesn't work anymore (it simply displays nothing at all: no plot, but also no traceback or error message).

We have also observed this behavior in other recent runtimes (specifically 15.3 and 13.3lts), but this problem doesn't happen in our legacy clusters running Databricks Runtime 7.3.

This issue is the only thing keeping us from deleting all of our runtime 7.3 clusters.

Re: dbutils secrets break matplotlib in recent Databricks runtimes

fonini — Tue, 06 Aug 2024 19:40:29 GMT

Hello @Retired_mod, thank you for replying.

Do you mean that this is a known problem? Is there anywhere I can track the progress in this issue? We'd prefer using the newest runtime versions we can, in order to benefit from new features, and to avoid having to worry about Databricks deprecating older runtimes.

Re: dbutils secrets break matplotlib in recent Databricks runtimes

fonini — Tue, 06 Aug 2024 21:00:09 GMT

For the record, re-importing matplotlib doesn't help (even after removing 'matplotlib', 'matplotlib.pyplot', and all other matplotlib submodules from sys.modules).

Re: dbutils secrets break matplotlib in recent Databricks runtimes

fonini — Thu, 08 Aug 2024 19:09:22 GMT

OK, I understand the error now:

This specific secret is a small alphanumeric string. The probability of it being part of the base64 of a big PNG is quite large.

Is there any workaround for this? I know I can't turn off the redaction of secrets. Can I disable it just for html metadata or anything like that?