cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
jlb0001
by • New Contributor III
  • 2694 Views
  • 3 replies
  • 1 kudos

[AWS] How do you replace the Account Admin?

I need to remove an older admin that previously set up the Databricks Account. However, I get an error (even through I am also an Account Admin).How do I replace a prior account admin? Or at least remove their admin status and/or disable the accoun...

Databricks Permssion Error - Cannot Disable Original Account
Data Engineering
Reply
  • 2694 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
jlb0001
New Contributor III
  • 1 kudos

I escalated via my Databricks rep yesterday and got an answer that seemed to be along the lines that "something is wrong here". He is going to try to find out internally and possibly work with the product development folks to come up with a solution...

  • 1 kudos
2 More Replies
PriyaV
by • New Contributor II
  • 13014 Views
  • 5 replies
  • 10 kudos

Suppress output in python notebooks

My dilemma is this - We use PySpark to connect to external data sources via jdbc from within databricks. Every time we issue a spark command, it spits out the connection options including the username, url and password which is not advisable. So, is ...

Data Engineering
Reply
  • 13014 Views
  • 5 replies
  • 10 kudos
View Replies
Latest Reply
Pabeggetur
New Contributor II
  • 10 kudos

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic.youi contact hoursuber eats complaints

  • 10 kudos
4 More Replies
Ismail1
by • New Contributor III
  • 2747 Views
  • 3 replies
  • 3 kudos

Resolved! Generating an Account console PAT token

I can't seem to find any documentation on generating an account console PAT token, Can anyone link me to it or guide me?

Data Engineering
Reply
  • 2747 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
fkseki
New Contributor III
  • 3 kudos

You can't create a Personal Access Token on account level to use REST APIs. If you want to use SCIM on account level, on the account console settings you'll find the user provisioning tab. In there you can generate de SCIM token. If you want to acces...

  • 3 kudos
2 More Replies
arun_pamulapati
by • Databricks Employee
  • 1152 Views
  • 1 replies
  • 1 kudos

www.youtube.com

We made another major release for Security Analysis Tool (SAT) with Unity Catalog and Delta sharing checks, Terraform deployments, and faster analysis if you have many workspaces. If you are on Azure Databricks there are new step-by-step video-based ...

Data Engineering
Reply
  • 1152 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Thank you for sharing @Arun Pamulapati​!!!

  • 1 kudos
sajith_appukutt
by • Honored Contributor II
  • 2285 Views
  • 2 replies
  • 1 kudos

Resolved! How can I configure S3 Client-Side Encryption (CSE-KMS ) for my data pipeline

Data Engineering
Reply
  • 2285 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
AdrianRojas
New Contributor II
  • 1 kudos

a bit old, but I just faced the same issue, specifying a custom EncryptionMaterialsProvider (as described in the previous post) did the trick for me but I did had to also specify my kms endpoint, just because my region:"fs.s3.cse.kms.endpoint" -> "km...

  • 1 kudos
1 More Replies
Chris_Shehu
by • Valued Contributor III
  • 4311 Views
  • 7 replies
  • 4 kudos

On-Behalf of tokens disabled for Azure Environments?

While trying to setup a Power BI connection to the Azure Delta Lake we ran into several issues around Service Principals. ​1) The API listed on the learn.microsoft site (link 1 below) indicates that there is an API you can use to create SP tokens. Wh...

Data Engineering
Reply
  • 4311 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
meetskorun
New Contributor II
  • 4 kudos

hello,i am new here from india, here to share some thoughts with you all

  • 4 kudos
6 More Replies
dcantos
by • New Contributor II
  • 5725 Views
  • 5 replies
  • 0 kudos

IP Access list Databricks

Hello, good morning everyone, I have a problem, I have IP access List activated in my Databricks workspace and I have connections to powerBI and Azure DevOps, in Power BI I have already added all the public IP ranges that Azure gives me, which are ab...

Data Engineering
Reply
  • 5725 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @David Cantos​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
4 More Replies
Arnold_Souza
by • New Contributor III
  • 7167 Views
  • 4 replies
  • 1 kudos

Connect Databricks to a database protected by a firewall

We a facing a situation and I would like to understand from the Databricks side what is the best practice regarding that. Question: Is it possible to have a cluster with a fixed Global IP on Databricks?DetailsWe have a vendor that has a SQL Server da...

Diagram
Data Engineering
Reply
  • 7167 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Arnold Souza​ If you file a support to Azure support they can help customize the Vnet by unlocking it as the Azure Databricks resources are deployed in a managed resource group. Your plan B also should be the way to go if option 1 does not work as e...

  • 1 kudos
3 More Replies
Chris_Shehu
by • Valued Contributor III
  • 3364 Views
  • 4 replies
  • 2 kudos

Resolved! No Explicit Deny for User security configurations at the group level?

Currently when you add new users to the Databricks workspace they get added to a "Users" group that has full access to the workspace. There should be a way to use group security to explicitly deny access to those same settings. This setting should ov...

image image image
Data Engineering
Reply
  • 3364 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@dean james​ I am not sure about your case why you want to deny access to the group once you create it. Anyhow, we can use deacticate/activate an user using "2.0/preview/scim/v2/Users/{id}" rest API endpoint. We can also deactivate users that have no...

  • 2 kudos
3 More Replies
Sujitha
by • Databricks Employee
  • 845 Views
  • 1 replies
  • 2 kudos

Documentation Update January 13 - 19 Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, a...

Documentation Update January 13 - 19Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, ...

Data Engineering
Reply
  • 845 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

thanks for the details

  • 2 kudos
isaac_gritz
by • Databricks Employee
  • 1455 Views
  • 1 replies
  • 6 kudos

Databricks Security Review

Conducting a security review or vendor assessment of Databricks and looking to learn more about our security features, compliance information, and privacy policies?You can find the latest on Databricks security features, architecture, compliance and ...

Data Engineering
Reply
  • 1455 Views
  • 1 replies
  • 6 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 6 kudos

thanks man

  • 6 kudos
AnubhavG
by • Contributor
  • 3734 Views
  • 5 replies
  • 18 kudos

Resolved! External KMS integration with Databricks like AWS KMS, Azure Key Vault.

I would like to know how we can integrate Databricks with External KMS providers, like currently it is doing with AWS KMS and Azure Key Valut?Can we import keys from any other KMS?

Data Engineering
Reply
  • 3734 Views
  • 5 replies
  • 18 kudos
View Replies
Latest Reply
Vivian_Wilfred
Databricks Employee
  • 18 kudos

@Anubhav Gupta​ Databricks is hosted on the cloud provider which means that all resources used by databricks in the backend are in the cloud. For instance, if you create a cluster, the VMs are launched in AWS as EC2 instances. So the integration of K...

  • 18 kudos
4 More Replies
pawelmitrus
by • Contributor
  • 1862 Views
  • 3 replies
  • 3 kudos

log4j vulnerability - action plan for clients

I'm looking for some information regarding log4j vulnerability - if any databricks runtime should be changed manually by the client or when specific update will be applied.I know I can go through the docs by myself, finding out which log4j library is...

Data Engineering
Reply
  • 1862 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Mr_Srinivasa
New Contributor II
  • 3 kudos

Thanks for sharing such important facts. I got the best security service provider website on the internet. They are excellence in the field of security. 

  • 3 kudos
2 More Replies
Cassio
by • New Contributor II
  • 3436 Views
  • 4 replies
  • 3 kudos

Resolved! "SparkSecurityException: Cannot read sensitive key" error when reading key from Spark config

In Databricks 10.1 it is possible to define in the "Spark Config" of the cluster something like:spark.fernet {{secrets/myscope/encryption-key}} . In my case my scopes are tied to Azure Key Vault.With that I can make a query as follows:%sql   SELECT d...

Data Engineering
Reply
  • 3436 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Soma
Valued Contributor
  • 3 kudos

This solution exposes the entire secret if I use commands like belowsql("""explain select upper("${spark.fernet.email}") as data """).display()Please dont use this

  • 3 kudos
3 More Replies
Labels