cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
JH
by • New Contributor II
  • 3414 Views
  • 2 replies
  • 0 kudos

Resolved! Does CVE-2020-13949 (vulnerability) affect the data plane and its security patch is included in Databricks 10.4 LTS?

Here is the previous discussion. https://community.databricks.com/s/question/0D58Y0000ACcIv2SQF/does-thrift-only-exist-in-databrick-control-plane   I have the following questions. Does CVE-2020-13949 affect the data plane or not?Do you know from whic...

Administration & Architecture
Reply
  • 3414 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jimin Hsieh​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...

  • 0 kudos
1 More Replies
JH
by • New Contributor II
  • 8815 Views
  • 0 replies
  • 0 kudos

Is CVE-2023-22946 managed by databricks?

Hi Databricks,   We want to understand a vulnerability issue - CVE-2023-22946.   does this CVE affect Azure databricks users? Does it affect data plane?Is it managed by databricks?Should end users do anything to mitigate it if this issue is a shared ...

Administration & Architecture
Reply
  • 8815 Views
  • 0 replies
  • 0 kudos
Teja07
by • New Contributor II
  • 2825 Views
  • 1 replies
  • 0 kudos

Number of DBU's utilized in Azure

We would like to know the number of DBU's utilized cluster level/workspace level which was hosted in Azure. We were able to see the amount spend but not able to see the DBU's. If we get any idea of no.of DBU's it would help us to think of reserved DB...

Administration & Architecture
Reply
  • 2825 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@Mani Teja G​ we don't have screen like aws for azure, but in azure you can monitor everything in cost management --> cost analysis and select tags as Databricks

  • 0 kudos
Kit
by • New Contributor III
  • 1878 Views
  • 1 replies
  • 0 kudos

How can I set up a DBU usage alert

Hi,I would like to set up a DBU usage alert, such that I will get an email notification when the DBU usage of last X hour exceeds the budget.How can I do that?Thanks,Kit

Administration & Architecture
Reply
  • 1878 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16752242622
Valued Contributor
  • 0 kudos

Hello @Kit Yam Tse​ You can download and analyse usage logs to monitor the DBUs.Check this out: https://docs.databricks.com/administration-guide/account-settings-e2/usage.htmlwe also have usage analysis dashboard notebook:https://docs.databricks.com/...

  • 0 kudos
DataDNA
by • New Contributor III
  • 11389 Views
  • 9 replies
  • 0 kudos

Resolved! AWS Databricks Pay-as-you-Go

Hi Team,   We are currently using AWS cloud service for our AI/ML Project. We wanted to use Databricks based GPU service for model training purposes. We do not find a Pay-as-you-go option in marketplace for this purposes. Can someone help me on this ...

Administration & Architecture
Reply
  • 11389 Views
  • 9 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

https://databricks.com/product/aws-pricingI guess you try to roll out something else?

  • 0 kudos
8 More Replies
Dunken
by • New Contributor III
  • 4549 Views
  • 4 replies
  • 4 kudos

Resolved! Databricks and DDD

Our architecture is according to Domain Driven Design. The data is therefore distributed among different domains.   We would like to run workloads on top of our data but we would like to avoid to have a dedicated (duplicated) data lake just for Datab...

Administration & Architecture
Reply
  • 4549 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

So basically you do not want to persist data outside of your source systems.I think the so called 'Kappa architecture' could be a fit, where everything is treated like a stream.Hubert already mentioned Kafka, which is an excellent source to build thi...

  • 4 kudos
3 More Replies
William_Scardua
by • Valued Contributor
  • 3877 Views
  • 4 replies
  • 5 kudos

Resolved! GDPR/LGPD Compliance

How do you work to compliance the GDPR/LGPD, in special to people data ? have any suggestion ?

Administration & Architecture
Reply
  • 3877 Views
  • 4 replies
  • 5 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 5 kudos

there is identifying PII data and handling/storing PII data.Identifying can be done with Purview, Macie, other tools. Those are not free ofc, so if your env is pretty big it can be interesting. Otherwise, you could also do manual checks.For storin...

  • 5 kudos
3 More Replies
baatchus
by • New Contributor III
  • 7026 Views
  • 4 replies
  • 7 kudos

Resolved! Architecture choice, streaming data

I have sensor data coming into Azure Event Hub and need some help in deciding how to best ingest it into the Data Lake and Delta Lake:   Option 1: azure event hub > databricks structured streaming > delta lake (bronze)   Option 2: azure event hub > e...

Administration & Architecture
Reply
  • 7026 Views
  • 4 replies
  • 7 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 7 kudos

If batch job is possible and you need to process data I would use probably:azure event hub from (events after previous job run) > databricks job process as dataframe > save df to delta lakeno streaming or capturing needed in that case

  • 7 kudos
3 More Replies
User16752244127
by • Contributor
  • 2541 Views
  • 2 replies
  • 1 kudos

How is Databricks integrated with AWS?

Administration & Architecture
Reply
  • 2541 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Brian_Dirking
New Contributor II
  • 1 kudos

A great place to learn more about Databricks integrations with AWS services is https://www.databricks.com/aws There is information on this page regarding integrations with Glue, SageMaker, Redshift and others. Many of these pages also point to our bl...

  • 1 kudos
1 More Replies
Anonymous
by • Not applicable
  • 11512 Views
  • 3 replies
  • 0 kudos

Resolved! What happens when multiple people write to the same Delta table?

Administration & Architecture
Reply
  • 11512 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
BigRoux
Databricks Employee
  • 0 kudos

Multiple writers (inserts/appends) present no problems with Delta. You can have two users appending data to a Delta table at the same time without issue. Updates, deletes, merges, and compaction can run into issues when multiple user are trying to d...

  • 0 kudos
2 More Replies
User16826987838
by • Contributor
  • 1516 Views
  • 1 replies
  • 0 kudos

Is GP3 EBS type supported in Databricks. If so, how can I enable that in the databricks console?

Administration & Architecture
Reply
  • 1516 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Taha
Databricks Employee
  • 0 kudos

Not as of this comment, but our product team is aware of this request and are working on getting this implemented soon!Keep in mind that for feature requests and direct updates/communication from our product team, you can post here (or just vote up o...

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
li.common.scroll-to.top