cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

hari
by Contributor
  • 18607 Views
  • 9 replies
  • 2 kudos

Resolved! Max job concurrency per workspace

Per the documentation, a workspace is limited to 1k concurrent job runs.Can somebody clarify how the concurrency limit is set i.e:Is it 1k concurrent runs across all jobs in the workspaceIs it 1k concurrent runs for a single jobAlso, is there any way...

  • 18607 Views
  • 9 replies
  • 2 kudos
Latest Reply
hari
Contributor
  • 2 kudos

Hi @Kaniz Fatma​ Weners gave some great suggestions on the issue we are dealing with. But some confirmation on the question from data bricks side would be much appreciated

  • 2 kudos
8 More Replies
Bittu6084
by New Contributor II
  • 12246 Views
  • 3 replies
  • 5 kudos

Resolved! How can we alter table with auto increment column for a delta table

How can we alter table with auto increment column for a delta tableI have tried this but not working:ALTER TABLE dbgtpTest.student ADD COLUMN Student_Id identity(100,1)any Suggestions will be helpful

  • 12246 Views
  • 3 replies
  • 5 kudos
Latest Reply
Hubert-Dudek
Databricks MVP
  • 5 kudos

@Rakesh Reddy Badam​ , For ALTER ... ADD COLUMN in doc is only SYNC IDENTITY.If you want to add an identity column to the existing table just create a new table with an identity column and then copy the data.

  • 5 kudos
2 More Replies
Hemant
by Valued Contributor II
  • 3005 Views
  • 1 replies
  • 1 kudos

Is there any way to use key vault on offline environment azure databricks?

We are working for client in offline environment(VM is private) azure databricks, we store some credentials in azure key vault, since azure databricks is run on offline environment, i am unable to create secret scopes on azure databricks, it always t...

  • 3005 Views
  • 1 replies
  • 1 kudos
Latest Reply
Hemant
Valued Contributor II
  • 1 kudos

Hi @Kaniz Fatma​ apologies for the delayed response, I earlier mentioned that the data bricks run in a private environment which means no connectivity to InterNet, I already followed the link you have shared but didn't get the result. Either I have t...

  • 1 kudos
AmanSehgal
by Honored Contributor III
  • 4269 Views
  • 2 replies
  • 12 kudos

Resolved! Error on using Databricks Account API - invalid endpoint

I'm trying to configure audit logs for databricks and as a part of it I need to configure S3 storage using endpoint storage-configurations.But for some reason I'm getting invalid endpoint. I'm using following URL with basicAuth.https://accounts.cloud...

  • 4269 Views
  • 2 replies
  • 12 kudos
Latest Reply
Anonymous
Not applicable
  • 12 kudos

Hey there @Aman Sehgal​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...

  • 12 kudos
1 More Replies
MMMM
by New Contributor III
  • 14073 Views
  • 8 replies
  • 2 kudos

cluster issue

I signed up with databricks free trial but I am not be able to create new cluster , could you help me in this regard ?When i create cluster nothing happens and it terminates automatically , I'm on aws

  • 14073 Views
  • 8 replies
  • 2 kudos
Latest Reply
MMMM
New Contributor III
  • 2 kudos

Could you help in answer all my questions 1 by one again

  • 2 kudos
7 More Replies
Dicer
by Valued Contributor
  • 4668 Views
  • 5 replies
  • 5 kudos

Resolved! Azure Databricks: AnalysisException: Database 'bf' not found

I wanted to save my delta tables in my Databricks database. When I saveAsTable, there is an error message Azure Databricks: AnalysisException: Database 'bf' not found​Ye, There is no database named "bf" in my database.Here is my full code:import os i...

  • 4668 Views
  • 5 replies
  • 5 kudos
Latest Reply
Dicer
Valued Contributor
  • 5 kudos

Some data can be saved as delta tables while some cannot.

  • 5 kudos
4 More Replies
mani238
by New Contributor III
  • 16174 Views
  • 2 replies
  • 2 kudos

Resolved! Run result unavailable: job failed with error message Library installation failed for library due to user error for jar: "dbfs:/my-jar.jar"

Run result unavailable: job failed with error message Library installation failed for library due to user error for jar: "dbfs:/my-jar.jar" . Error messages: Library installation attempted on the driver node of cluster 0510-013936-3cc6d9kw and failed...

  • 16174 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey there @manivannan p​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 2 kudos
1 More Replies
Dataminion
by New Contributor
  • 4306 Views
  • 1 replies
  • 0 kudos

No module named dlt

Cluster version 10.5 on AWS.Trying to import dlt gave error above.​Am I missing something?​

  • 4306 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hey there @Cathy Low​ Does @Kaniz Fatma​ 's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help. We'd love to hear from y...

  • 0 kudos
SailajaB
by Valued Contributor III
  • 5378 Views
  • 2 replies
  • 6 kudos

Resolved! How to pass a variable which holds a value to child notebook using run command

Hello,We have 3 notebooks as below. And trying to send a variable where we pass the value to it via job to child notebook with %run commandNotebooks:notebook_par,notebook_child1 and notebook_child2.In parent note book we are calling notebook_child1 u...

  • 5378 Views
  • 2 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hey there @Sailaja B​ How's it going?Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Chee...

  • 6 kudos
1 More Replies
prasadvaze
by Valued Contributor II
  • 47600 Views
  • 10 replies
  • 8 kudos

When to use delta lake versus relational database as a source for BI reporting?

Assume all of your data exists in delta tables and also in SQL server so you have a choice to report from either. Can someone share thoughts on "In what scenario you would not want report created from delta table and instead use the traditional rel...

  • 47600 Views
  • 10 replies
  • 8 kudos
Latest Reply
PCJ
New Contributor II
  • 8 kudos

Hi @Kaniz Fatma​  - I would like a follow-up on @prasad vaze​ question regarding unsupported referential integrity. How does one work around that, using best practices as Databricks sees it?

  • 8 kudos
9 More Replies
Steveo
by New Contributor II
  • 1573 Views
  • 1 replies
  • 0 kudos

In Databricks, when saving a query, the changes do no persist. Any advice?

When returning to the query, the changes have reverted.Is this because the query is open by another user?How can this be resolved? #databricks

  • 1573 Views
  • 1 replies
  • 0 kudos
Latest Reply
Steveo
New Contributor II
  • 0 kudos

We ARE using the save feature. But we are still loosing changes. Is this because we have multiple Databricks tabs open in separate Chrome browsers? Across 2 users.

  • 0 kudos
gazzyjuruj
by Contributor II
  • 25708 Views
  • 9 replies
  • 8 kudos

Resolved! Failed to start cluster

Hi, I ran the cluster more than 5-6 times with it failing to start since this past morning (about 11-12 hours now) since i'm facing this problem.Attaching screenshot below and also typing in case someone comes from the web to this thread in future.Pr...

IMG_2152 IMG_2151
  • 25708 Views
  • 9 replies
  • 8 kudos
Latest Reply
worthpicker
New Contributor II
  • 8 kudos

The cluster will be able to start and the nodes will automatically obtain the updated cluster configuration data.

  • 8 kudos
8 More Replies
labromb
by Contributor
  • 2928 Views
  • 0 replies
  • 0 kudos

Converting a text widget to a list

Hi, Been working on some parallel notebook code, which I have ported to python from the example on the DB website and added some exception handling and that works fine. What I would like to do is paramterise the input but am not succeeding as the fun...

  • 2928 Views
  • 0 replies
  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels