cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Avvar2022
by Contributor
  • 1875 Views
  • 1 replies
  • 1 kudos

Limited number of workspaces vs workspace for department or line of business ?

We are just getting started with databricks currently we have 1 workspace for each environment (DEV, QA and PRD). we have started with 1 workspace but there is already getting flooded with new workspace requests?   is there any check list/criteria fo...

  • 1875 Views
  • 1 replies
  • 1 kudos
Latest Reply
Mounika_Tarigop
Databricks Employee
  • 1 kudos

I believe DEV, QA and PRD are the right segregate . We may need this because this will be easy to categorize the production and QA workload - based upon the amount of data cluster processes (Meaning more DBU) which we can restrict by the company work...

  • 1 kudos
KKo
by Contributor III
  • 1935 Views
  • 1 replies
  • 0 kudos

Delta Live Table tables in Data Tab

If I use this code (CREATE STREAMING LIVE TABLE Employee) in dlt pipeline, where does the Employee table gets created by default, if no storage location is specified? How can I create this table in Data tab within a database (a.k.a: Schema) lets say ...

  • 1935 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Kris Koirala​ :When you create a streaming live table in Databricks Delta Lake using the code CREATE STREAMING LIVE TABLE Employee, the table is created in the default database called default. If no storage location is specified, the table is stored...

  • 0 kudos
AK031
by New Contributor II
  • 2971 Views
  • 3 replies
  • 0 kudos

If I come via Databricks Partner connect and subscribe a partner product then how is the billing done and what api is used for publishing usage information to databricks?

If I come via Databricks Partner connect and subscribe a partner product then how is the billing done and what api is used for publishing usage information to databricks?

  • 2971 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Atul Karn​ We haven't heard from you since the last response from @Kaniz Fatma​ â€‹, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others....

  • 0 kudos
2 More Replies
mriccardi
by New Contributor II
  • 10040 Views
  • 1 replies
  • 0 kudos

Spark Streaming: Checkpoint corrupted

Hi Everyone!   Today 4 streaming jobs started to fail out of nowhere due to: StreamingQueryException: [STREAM_FAILED] Query [id = ####, runId = ####] terminated with exception: dbfs:/mnt/path/my_table/sources/0/0 doesn't exist (latestId: 8, compactIn...

  • 10040 Views
  • 1 replies
  • 0 kudos
Latest Reply
Vartika
Databricks Employee
  • 0 kudos

Hi @Martin Riccardi​,We haven't heard from you since the last response from @Kaniz Fatma​ , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ot...

  • 0 kudos
niklas
by Contributor
  • 5343 Views
  • 2 replies
  • 3 kudos

Resolved! How can I set the data access for each SQL warehouse individually?

As far as the official documentation from Microsoft says and I understood it correctly, each SQL warehouse uses the same service principal for data access. But I want to control the access to the data (Azure Data Lake Storage) for each SQL warehouse ...

  • 5343 Views
  • 2 replies
  • 3 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 3 kudos

It can be done but it's very tricky. You should create a service principal to whom you will grant permission over some tables then you can generate a token using the SP to access only this data.

  • 3 kudos
1 More Replies
Mett
by New Contributor II
  • 16717 Views
  • 2 replies
  • 0 kudos

How to move from an embedded spark inside a spring boot microservices to azure databricks?

Hi,   My question will be more about the architecture solutions and potential implementation of these solutions.   The project :   Multiple microservices each one deployed on a Tomcat and each Tomcat is running in one of the tree VM existing in Azure...

infra execJob target architecture
  • 16717 Views
  • 2 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

My main question is why you use spark in the first place, but that is another discussion.If you want to use databricks, you will have to launch a spark job in one way or another on the databricks cluster with spark-submit or by using the jobs api.How...

  • 0 kudos
1 More Replies
db_db
by New Contributor
  • 9087 Views
  • 2 replies
  • 2 kudos
  • 9087 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @janaki jagadeesh​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...

  • 2 kudos
1 More Replies
JH
by New Contributor II
  • 3348 Views
  • 2 replies
  • 0 kudos

Resolved! Does CVE-2020-13949 (vulnerability) affect the data plane and its security patch is included in Databricks 10.4 LTS?

Here is the previous discussion. https://community.databricks.com/s/question/0D58Y0000ACcIv2SQF/does-thrift-only-exist-in-databrick-control-plane   I have the following questions. Does CVE-2020-13949 affect the data plane or not?Do you know from whic...

  • 3348 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jimin Hsieh​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...

  • 0 kudos
1 More Replies
JH
by New Contributor II
  • 8778 Views
  • 0 replies
  • 0 kudos

Is CVE-2023-22946 managed by databricks?

Hi Databricks,   We want to understand a vulnerability issue - CVE-2023-22946.   does this CVE affect Azure databricks users? Does it affect data plane?Is it managed by databricks?Should end users do anything to mitigate it if this issue is a shared ...

  • 8778 Views
  • 0 replies
  • 0 kudos
Teja07
by New Contributor II
  • 2775 Views
  • 1 replies
  • 0 kudos

Number of DBU's utilized in Azure

We would like to know the number of DBU's utilized cluster level/workspace level which was hosted in Azure. We were able to see the amount spend but not able to see the DBU's. If we get any idea of no.of DBU's it would help us to think of reserved DB...

  • 2775 Views
  • 1 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@Mani Teja G​ we don't have screen like aws for azure, but in azure you can monitor everything in cost management --> cost analysis and select tags as Databricks

  • 0 kudos
Kit
by New Contributor III
  • 1838 Views
  • 1 replies
  • 0 kudos

How can I set up a DBU usage alert

Hi,I would like to set up a DBU usage alert, such that I will get an email notification when the DBU usage of last X hour exceeds the budget.How can I do that?Thanks,Kit

  • 1838 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16752242622
Valued Contributor
  • 0 kudos

Hello @Kit Yam Tse​ You can download and analyse usage logs to monitor the DBUs.Check this out: https://docs.databricks.com/administration-guide/account-settings-e2/usage.htmlwe also have usage analysis dashboard notebook:https://docs.databricks.com/...

  • 0 kudos