Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I need to remove an older admin that previously set up the Databricks Account. However, I get an error (even through I am also an Account Admin).How do I replace a prior account admin? Or at least remove their admin status and/or disable the accoun...
I escalated via my Databricks rep yesterday and got an answer that seemed to be along the lines that "something is wrong here". He is going to try to find out internally and possibly work with the product development folks to come up with a solution...
My dilemma is this - We use PySpark to connect to external data sources via jdbc from within databricks. Every time we issue a spark command, it spits out the connection options including the username, url and password which is not advisable. So, is ...
You can't create a Personal Access Token on account level to use REST APIs. If you want to use SCIM on account level, on the account console settings you'll find the user provisioning tab. In there you can generate de SCIM token. If you want to acces...
We made another major release for Security Analysis Tool (SAT) with Unity Catalog and Delta sharing checks, Terraform deployments, and faster analysis if you have many workspaces. If you are on Azure Databricks there are new step-by-step video-based ...
a bit old, but I just faced the same issue, specifying a custom EncryptionMaterialsProvider (as described in the previous post) did the trick for me but I did had to also specify my kms endpoint, just because my region:"fs.s3.cse.kms.endpoint" -> "km...
While trying to setup a Power BI connection to the Azure Delta Lake we ran into several issues around Service Principals. 1) The API listed on the learn.microsoft site (link 1 below) indicates that there is an API you can use to create SP tokens. Wh...
Hello, good morning everyone, I have a problem, I have IP access List activated in my Databricks workspace and I have connections to powerBI and Azure DevOps, in Power BI I have already added all the public IP ranges that Azure gives me, which are ab...
Hi @David Cantos Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
We a facing a situation and I would like to understand from the Databricks side what is the best practice regarding that. Question: Is it possible to have a cluster with a fixed Global IP on Databricks?DetailsWe have a vendor that has a SQL Server da...
@Arnold Souza If you file a support to Azure support they can help customize the Vnet by unlocking it as the Azure Databricks resources are deployed in a managed resource group. Your plan B also should be the way to go if option 1 does not work as e...
Currently when you add new users to the Databricks workspace they get added to a "Users" group that has full access to the workspace. There should be a way to use group security to explicitly deny access to those same settings. This setting should ov...
@dean james I am not sure about your case why you want to deny access to the group once you create it. Anyhow, we can use deacticate/activate an user using "2.0/preview/scim/v2/Users/{id}" rest API endpoint. We can also deactivate users that have no...
Documentation Update January 13 - 19Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, ...
Conducting a security review or vendor assessment of Databricks and looking to learn more about our security features, compliance information, and privacy policies?You can find the latest on Databricks security features, architecture, compliance and ...
I would like to know how we can integrate Databricks with External KMS providers, like currently it is doing with AWS KMS and Azure Key Valut?Can we import keys from any other KMS?
@Anubhav Gupta Databricks is hosted on the cloud provider which means that all resources used by databricks in the backend are in the cloud. For instance, if you create a cluster, the VMs are launched in AWS as EC2 instances. So the integration of K...
I'm looking for some information regarding log4j vulnerability - if any databricks runtime should be changed manually by the client or when specific update will be applied.I know I can go through the docs by myself, finding out which log4j library is...
Thanks for sharing such important facts. I got the best security service provider website on the internet. They are excellence in the field of security.
In Databricks 10.1 it is possible to define in the "Spark Config" of the cluster something like:spark.fernet {{secrets/myscope/encryption-key}} . In my case my scopes are tied to Azure Key Vault.With that I can make a query as follows:%sql
SELECT d...
This solution exposes the entire secret if I use commands like belowsql("""explain select upper("${spark.fernet.email}") as data """).display()Please dont use this