cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

yzhang
by New Contributor III
  • 2963 Views
  • 5 replies
  • 0 kudos

Cannot find such info if Databricks supports nested jobs or tasks. For example, I have a 'job_a', which contains list of tasks, and another &#...

Cannot find such info if Databricks supports nested jobs or tasks. For example, I have a 'job_a', which contains list of tasks, and another 'job_b', also contains a list of tasks. Now I'd like to have a 'job_all' that will run both 'job_a' and 'job_b...

  • 2963 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Yanan Zhang​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the response and select the one that best answers yo...

  • 0 kudos
4 More Replies
Chris_Shehu
by Valued Contributor III
  • 3515 Views
  • 4 replies
  • 2 kudos

Resolved! No Explicit Deny for User security configurations at the group level?

Currently when you add new users to the Databricks workspace they get added to a "Users" group that has full access to the workspace. There should be a way to use group security to explicitly deny access to those same settings. This setting should ov...

image image image
  • 3515 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@dean james​ I am not sure about your case why you want to deny access to the group once you create it. Anyhow, we can use deacticate/activate an user using "2.0/preview/scim/v2/Users/{id}" rest API endpoint. We can also deactivate users that have no...

  • 2 kudos
3 More Replies
andrew0117
by Contributor
  • 4518 Views
  • 4 replies
  • 0 kudos

Resolved! Can merge() function be applied to dataframe?

if I have two dataframes df_target and df_source, can I do df_target.as("t).merge(df_source.as("s"), "s.id=t.id").whenMatched().updateAll().whenNotMatched.insertAll.execute(). when I tried the code above, I got the error "merge is not a member of the...

  • 4518 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @andrew li​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
3 More Replies
JJL
by New Contributor II
  • 15150 Views
  • 3 replies
  • 3 kudos

Resolved! Does Spark SQL can perform UPDATE with INNER JOIN and LIKE with '%' + [column] + '%' ?

Hi All,I came from MS SQL and just started to learning more about Spark SQLHere is one part that I'm trying to perform. In MS SQL, it can be easily done, but it seems like it doesn't in SparkSo, I want to make a simple update to the record, if the co...

  • 15150 Views
  • 3 replies
  • 3 kudos
Latest Reply
oleole
Contributor
  • 3 kudos

@Hubert Dudek​ Hello, I'm having the same issue with using UPDATE in spark sql and came across your answer. When you say "replace source_table_reference with view" in MERGE, do you mean to replace "P" with "VIEW" that looks something as below:%sql ME...

  • 3 kudos
2 More Replies
Anonymous
by Not applicable
  • 5721 Views
  • 1 replies
  • 1 kudos

Databricks-connect configured with service principal token but unable to retrieve information to local machine

installed databricks-connect and configured with service principal token, able to start cluster when I use command spark=SparkSession\.builder\.getOrCreate() But when trying to retrieve s3 bucket data to local machine or even i run test command ex...

  • 5721 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @divya08Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
Gaurav_Raj
by New Contributor III
  • 2732 Views
  • 3 replies
  • 3 kudos

Resolved! Lakehouse Fundamentals Accreditation Badge not received after the course completion

I completed the Databricks Lakehouse Fundamentals Accreditation course today, but I didn't receive my badge yet.I even checked in: https://credentials.databricks.com/ but shows no record/ credentials. see the screenshot below. Please help me out with...

image
  • 2732 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Gaurav Raj​ Thank you for posting your question in our community! We are happy to assist you.Every best answer marked contributes to the growth and success of our community.Regards

  • 3 kudos
2 More Replies
RengarLee
by Contributor
  • 8605 Views
  • 10 replies
  • 3 kudos

Resolved! Databricks write to Azure Data Explorer writes suddenly become slower

Now, I write to Azure Data explorer using Spark streaming. one day, writes suddenly become slower. restart is no effect.I have a questions about Spark Streaming to Azure Data explorer.Q1: What should I do to get performance to reply?Figure 1 shows th...

  • 8605 Views
  • 10 replies
  • 3 kudos
Latest Reply
RengarLee
Contributor
  • 3 kudos

I'm so sorry, I just thought the issue wasn't resolvedSolutionSet maxFilesPerTrigger and maxBytesPerTrigger Enable autpoptimizeReason for the first day, it processes larger files and then eventually process smaller files。Detailed reason B...

  • 3 kudos
9 More Replies
MetaRossiVinli
by Contributor
  • 4679 Views
  • 1 replies
  • 1 kudos

Resolved! Find root path to Repo for .py file import

I want to import a Python function stored in the following file path:`<repo>/lib/lib_helpers.py`I want to import the function from any file in my repo. For instance from these:`<repo>/notebooks/etl/bronze/dlt_bronze_elt``<repo>/workers/job_worker`It ...

  • 4679 Views
  • 1 replies
  • 1 kudos
Latest Reply
MetaRossiVinli
Contributor
  • 1 kudos

Ok, I figured it out. If you just make it a Python module by adding an empty `__init__.py`, Databricks will load it on start. Then, you can just import it.

  • 1 kudos
Ancil
by Contributor II
  • 9389 Views
  • 7 replies
  • 4 kudos

Resolved! Cannot create table The associated location ('dbfs:/mnt/[REDACTED]/folder/table_location') is not empty but it's not a Delta table

Hi Team, I have created new databricks workspace for using private network. I want to read/write/update delta table created in old databricks and stored in ADLS as delta.Any one please help me on this?.....I tried to create delta table in same loca...

  • 9389 Views
  • 7 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Ancil P A​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 4 kudos
6 More Replies
faycal_merouane
by New Contributor III
  • 4201 Views
  • 5 replies
  • 6 kudos

Resolved! Databricks Community Edition Not able to Login Account

hello, I'm not able to log in to my Databricks community account, I'm getting the following error when trying to log in:You entered an invalid email or password, or your workspace access is locked because of unusual activity. For help, see Community ...

  • 4201 Views
  • 5 replies
  • 6 kudos
Latest Reply
faycal_merouane
New Contributor III
  • 6 kudos

hi @Vidula Khanna​, I appreciate your help, I will try to contact them using help@databricks.com.thank you for your assistance.best regards.

  • 6 kudos
4 More Replies
m2chrisp
by New Contributor II
  • 7178 Views
  • 3 replies
  • 0 kudos

Resolved! Deploying Databricks via ARM, then configuring via databricks-cli: creating a repo results in INVALID_STATE error.

Hi,I'm using databricks-cli to configure a newly-deployed Azure Databricks instance. The ARM deployment works fine, and then the databricks cli commands to create a secrets scope and add users also works just fine.Then I add a GitCredential to Datab...

  • 7178 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Chris Platts​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Thanks and Regards

  • 0 kudos
2 More Replies
KJ_Saravana
by New Contributor III
  • 7464 Views
  • 6 replies
  • 9 kudos

Resolved! Databricks cluster Init scripts on ABFSS location

HI,I have an init script which works on DBFS location during the cluster start up, but when the same shell script file is placed on ABFSS location (ADLS Gen 2 storage) I get the following init script failure error and the cluster is unable to start.E...

  • 7464 Views
  • 6 replies
  • 9 kudos
Latest Reply
Anonymous
Not applicable
  • 9 kudos

Hi @Saravana KJ​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest pr...

  • 9 kudos
5 More Replies
KVNARK
by Honored Contributor II
  • 2733 Views
  • 1 replies
  • 5 kudos

Resolved! sudden decrease in rewards point in Rewards portal

@Vidula Khanna​ May I know why the reward points decreased in Databricks community rewards portal. Could anyone help on this. I haven't ordered for anything. But today when I checked there was sudden decrease in points and the no of points came from ...

  • 2733 Views
  • 1 replies
  • 5 kudos
jogender
by New Contributor
  • 2113 Views
  • 1 replies
  • 0 kudos

The hire react js developer from the best react js development company. We provide the most experienced hire reactjs developer as per the requirements...

The hire react js developer from the best react js development company. We provide the most experienced hire reactjs developer as per the requirements cost.

  • 2113 Views
  • 1 replies
  • 0 kudos
Latest Reply
Sovchenko
New Contributor II
  • 0 kudos

Thanks for sharing! By the way, if you are interested in the costs of hire react js developers, I advise you to read this: https://www.cleveroad.com/blog/how-much-does-it-cost-to-hire-a-react-developer/

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels