Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
We're using the following method (generated by using dbx) to access dbutils, e.g. to retrieve parameters from secret scopes: @staticmethod
def _get_dbutils(spark: SparkSession) -> "dbutils":
try:
from pyspark.dbutils import...
I have like 40 mounts and some of them were done based on the scopes which now I want to remove. How can I list the mounts for specyfic scope or even better, get the list of all the mounts alongside the scopes?
Hey there @Bartosz Wachocki Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fro...
I have created a key in Azure Key Vault to store my secrets in it. In order to use it securely in Azure DataBricks, have created the secret scope and configured the Azure Key Vault properties. Out of curiosity, just wanted to check whether my key is ...
@Kaniz Fatma is any fix coming soon for this? this is a big security loophole The docs say that "To ensure proper control of secrets you should use Workspace object access control (limiting permission to run commands) " --- if i prevent access to ru...
The below code executes a 'get' api method to retrieve objects from s3 and write to the data lake.The problem arises when I use dbutils.secrets.get to get the keys required to establish the connection to s3my_dataframe.rdd.foreachPartition(partition ...
Howdy @Sandesh Puligundla - Thank you for your question. Thank you for your patience. I'd like to give this a bit longer to see how the community responds. Hang tight!
