cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MonuDatabricks
by New Contributor II
  • 609 Views
  • 1 replies
  • 0 kudos

Resolved! Using Autoloader with merge

Hi Everyone, I have been trying to use autoloader with foreach so that I could able to use merge into in databricks, but while using I have been getting below error.error-Found error inside foreachBatch Python processMy code-from delta.tables import ...

  • 609 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

It seems the columns of your join condition are not found.  Are they in the dataframes/table?Also try to put the whole join condition in a single string:"s.JeHeaderId = t.JeHeaderId and s.JeLineId = t.JeLineId"

  • 0 kudos
DBricksNewbie
by New Contributor III
  • 1146 Views
  • 2 replies
  • 0 kudos

Resolved! Can I give different git branches in the same repo for different tasks in a data bricks workflow

I have 2 tasks (T1 &T2) that run in branch B1 of Repo1.I have created a new task (depends on T2 ) which points to a different branch B2  of same Repo1.Is it possible to run them in the same workflow pipeline? When I tried to set this up, databricks c...

  • 1146 Views
  • 2 replies
  • 0 kudos
Latest Reply
DBricksNewbie
New Contributor III
  • 0 kudos

I was to able to find a workaround. Created separate jobs for those that need to be in different branch (testing tasks) and then ran all of them from a new job.

  • 0 kudos
1 More Replies
MKTexas13
by New Contributor II
  • 1113 Views
  • 1 replies
  • 0 kudos

Resolved! Setting a preset list of values in a task parameter in databricks job

I want to be able to have a user select from a preset list of values for a task parameter when they kick off a job with the "Run now with different parameters" option. In a notebook I am able to use dbutils.widgets.dropdown() to set the list of value...

  • 1113 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Unfortunately providing a job params dropdown list is not currently available, you can alway do a Run with different params, but the user will have to change them manually and not with a predefined list.

  • 0 kudos
colinhoad
by New Contributor
  • 356 Views
  • 1 replies
  • 0 kudos

New icon for SQL Editor looks like a broken image

Hey - I may be showing my age here, but I felt compelled to point out that at a glance, the new icon for a SQL Editor tab in the Databricks UI looks an awful lot like a broken image link icon, from the days of Internet Explorer. This, subconsciously,...

87338258.png colinhoad_1-1728893941310.png
  • 356 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Is this still showing broking image? Is this only happening in Explorer, if you try Chrome for example does it work?Can you share an screenshot of your workspace to better understand how it shows?  

  • 0 kudos
Rafael-Sousa
by Contributor II
  • 708 Views
  • 2 replies
  • 1 kudos

Resolved! Internal Error with MERGE Command in Spark SQL

I'm trying to perform a MERGE between two tables (customers and customers_update) using Spark SQL, but I’m encountering an internal error during the planning phase. The error message suggests it might be a bug in Spark or one of the plugins in use.He...

  • 708 Views
  • 2 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

The issue you encountered with the MERGE statement in Spark SQL, which was resolved by specifying the database and metastore, is likely related to how Spark handles table references during the planning phase. The internal error you faced suggests a b...

  • 1 kudos
1 More Replies
Fkebbati
by New Contributor II
  • 447 Views
  • 1 replies
  • 0 kudos

Ingress/Egress private endpoint

Hello , We have configured our Databricks environment with private endpoint connections injected into our VNET, which includes two subnets (public and private). We have disabled public IPs and are using Network Security Groups (NSGs) on the subnet, a...

  • 447 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @Fkebbati ,There always be some costs related to data transfer between those account. Let's have a look at private link pricing page. So it's expected, but MS likes to hide this kind of information  

  • 0 kudos
ladyleet
by New Contributor
  • 367 Views
  • 1 replies
  • 1 kudos

Best way to find Databricks Certified Professionals?

Hi all! We have a few Databricks certified folks on our team, but we are looking for more! What is the best way to find a list of or know who is certified? We are looking for North America / Europe / LATAM / South America. I am part of the Microsoft ...

  • 367 Views
  • 1 replies
  • 1 kudos
Latest Reply
Rafael-Sousa
Contributor II
  • 1 kudos

Hey @ladyleet ,There is the community of Databricks certified: Databricks Certified Credential Holder Directory.

  • 1 kudos
Amrit23
by New Contributor
  • 497 Views
  • 1 replies
  • 0 kudos

Amazon MSK integration with Databricks

Hello Everyone,I am a beginner in the world of Data bricks. I am trying to achieve a use case which involves : consuming messages from Amazon MSK and creating a Delta table in Data bricks.I need to get some insights on what all accesses are supposed ...

  • 497 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hey @Amrit23 , Databricks Assistant can help you write the code. You need to use spark.readStream() to access the stream.  https://docs.databricks.com/en/connect/streaming/kafka.html And for the Access Permissions: Amazon MSK: Ensure you have the nec...

  • 0 kudos
SachinJanani
by New Contributor II
  • 615 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks SQL as alternative to Spark thrift server

We are currently using Spark as our SQL engine with Thrift Server but are evaluating Databricks Serverless SQL as a potential alternative. We have a few specific questions:Does Databricks Serverless SQL support custom Spark extensions?Can we configur...

Community Platform Discussions
Databricks SQL
databrickssql
  • 615 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi @SachinJanani , 1 - Databricks Serverless SQL does not support custom Spark extensions (Advanced Spark Configs/Libraries etc). This is because the serverless environment is designed to be highly optimized and managed by Databricks, which limits th...

  • 0 kudos
bvraravind
by New Contributor II
  • 813 Views
  • 1 replies
  • 0 kudos

Resolved! Unable to access Azure blob storage with SAS token

I am following Microsoft documentation to connect from Databricks workspace to Azure blob storage. but it is not working. Any help is greatly appreciated. Below is the codespark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows....

  • 813 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @bvraravind, The error you are encountering is due to an incorrect configuration setting in your code. The error message indicates that the configuration fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net is not recognized Verify th...

  • 0 kudos
Tejsharma
by New Contributor
  • 1408 Views
  • 1 replies
  • 0 kudos

Troubleshooting the Error "Credential was not sent or was of an unsupported type for this API"

I previously worked on Databricks Asset Bundle (DAB) using a Service Principal token, and it was successful. However, when I attempted it again now, I encountered an error.Error: failed to compute file content for {{.project_name}}/databricks.yml.tmp...

  • 1408 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Which type of token are you currently using, is it an Oauth token or Obo token? Have you generated a new token for testing?

  • 0 kudos
MC97
by New Contributor
  • 500 Views
  • 1 replies
  • 0 kudos

Update on CTE

So I am reflecting a business logic from on prem to azure databricks . what on prem did is created the table and after that updated . I have to construct that as a single query . Example Create or replace table table1with CTE 1 as () ,CTE 2 as (selec...

  • 500 Views
  • 1 replies
  • 0 kudos
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

An actual "Update", it may not be possible, but have you consider and will something like this work for you? This is simulating updates within the query without actual UPDATE statements: CREATE OR REPLACE TABLE table1 AS WITH CTE1 AS ( -- Your in...

  • 0 kudos
dbx_687_3__1b3Q
by New Contributor III
  • 2408 Views
  • 5 replies
  • 4 kudos

Impersonating a user

How do I impersonate a user? I can't find any documentation that explains how to do this or even hint that it's possible.Use case: I perform administrative tasks like assign grants and roles to catalogs, schemas, and tables for the benefit of busines...

  • 2408 Views
  • 5 replies
  • 4 kudos
Latest Reply
NandiniN
Databricks Employee
  • 4 kudos

DB-I-8117 this one is mentioned to be considered for future so adding votes for sure will help.

  • 4 kudos
4 More Replies
Prasad_Koneru
by New Contributor III
  • 1327 Views
  • 2 replies
  • 0 kudos

Databricks grant update calatog catlog_name --json @privileges.json not updating privileges

Hi Team, I am trying to update the catalog permission privileges using databricks cli command Grant by appending json file but which is not updating the prIviliges, please help on grant update command usage.Command using :  databricks grants update c...

  • 1327 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ravivarma
Databricks Employee
  • 0 kudos

Hello @Prasad_Koneru   If the command is not updating the privileges as expected, there could be a few reasons for this.  Firstly, ensure that the JSON file is correctly formatted and contains the correct privilege assignments. The privileges.json fi...

  • 0 kudos
1 More Replies
abueno
by New Contributor III
  • 4036 Views
  • 3 replies
  • 8 kudos

Resolved! Find and replace

Hi,Is there a "Find and replace" option to edit SQL code?  I am not referring to the "replace" function but something similar to Control  + shift + F in Snowflake or Control + F in MS Excel.

  • 4036 Views
  • 3 replies
  • 8 kudos
Latest Reply
DBKENGR
New Contributor III
  • 8 kudos

is there an option to find-replace just within a cell instead of entire notebook?

  • 8 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors