cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MonuDatabricks
by New Contributor II
  • 2254 Views
  • 1 replies
  • 0 kudos

Resolved! Using Autoloader with merge

Hi Everyone, I have been trying to use autoloader with foreach so that I could able to use merge into in databricks, but while using I have been getting below error.error-Found error inside foreachBatch Python processMy code-from delta.tables import ...

  • 2254 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

It seems the columns of your join condition are not found.  Are they in the dataframes/table?Also try to put the whole join condition in a single string:"s.JeHeaderId = t.JeHeaderId and s.JeLineId = t.JeLineId"

  • 0 kudos
NagarajuT
by New Contributor
  • 2116 Views
  • 1 replies
  • 0 kudos

Connect to SQL Developer using Custom JDBC

Hello,I'm trying to connect databricks SQL to the SQL Developer using custom JDBC.I'm getting errorjdbc:databricks:<server>:443;HttpPath=<HttpPath>;UID=token;PWD=<password> RegardsNaga

  • 2116 Views
  • 1 replies
  • 0 kudos
Latest Reply
ShaliniC
New Contributor II
  • 0 kudos

Hi,We are trying to test if we can connect sql developer to databricks. did it work for you?Regards, shalini

  • 0 kudos
tyorisoo
by New Contributor III
  • 1846 Views
  • 6 replies
  • 0 kudos

Unity Catalog About Metastore

Registered on 2024/10 from AWS marketplace.We have created a customer management VPC and manually created the workspace.No specific metastore settings were made when the workspace was created.In the catalog screen of the account console,unity catalog...

  • 1846 Views
  • 6 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @tyorisoo,I hope you are doing well!Metastore manages metadata, not catalog information, schema information, table information, function information, access control information, etc. In the current state, the metastore configuration is not done...

  • 0 kudos
5 More Replies
Surajv
by New Contributor III
  • 2472 Views
  • 2 replies
  • 1 kudos

What is the quota limit for using create user token api?

Hi Community, I was going through this doc: https://docs.databricks.com/api/workspace/tokens/create to and got to know, that there is a quota limit to how many token one can generate using the api: POST /api/2.0/token/create, having breached the thre...

  • 2472 Views
  • 2 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Hello@Surajv, Q1: What is and how to find out the quota limit? The quota limit for creating user tokens via the API (POST /api/2.0/token/create) is essential to manage token usage. Each user can have multiple personal access tokens in a Databricks wo...

  • 1 kudos
1 More Replies
DBricksNewbie
by New Contributor III
  • 1985 Views
  • 2 replies
  • 0 kudos

Resolved! Can I give different git branches in the same repo for different tasks in a data bricks workflow

I have 2 tasks (T1 &T2) that run in branch B1 of Repo1.I have created a new task (depends on T2 ) which points to a different branch B2  of same Repo1.Is it possible to run them in the same workflow pipeline? When I tried to set this up, databricks c...

  • 1985 Views
  • 2 replies
  • 0 kudos
Latest Reply
DBricksNewbie
New Contributor III
  • 0 kudos

I was to able to find a workaround. Created separate jobs for those that need to be in different branch (testing tasks) and then ran all of them from a new job.

  • 0 kudos
1 More Replies
MKTexas13
by New Contributor III
  • 1777 Views
  • 1 replies
  • 0 kudos

Resolved! Setting a preset list of values in a task parameter in databricks job

I want to be able to have a user select from a preset list of values for a task parameter when they kick off a job with the "Run now with different parameters" option. In a notebook I am able to use dbutils.widgets.dropdown() to set the list of value...

  • 1777 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Unfortunately providing a job params dropdown list is not currently available, you can alway do a Run with different params, but the user will have to change them manually and not with a predefined list.

  • 0 kudos
colinhoad
by New Contributor
  • 747 Views
  • 1 replies
  • 0 kudos

New icon for SQL Editor looks like a broken image

Hey - I may be showing my age here, but I felt compelled to point out that at a glance, the new icon for a SQL Editor tab in the Databricks UI looks an awful lot like a broken image link icon, from the days of Internet Explorer. This, subconsciously,...

87338258.png colinhoad_1-1728893941310.png
  • 747 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Is this still showing broking image? Is this only happening in Explorer, if you try Chrome for example does it work?Can you share an screenshot of your workspace to better understand how it shows?  

  • 0 kudos
benito
by New Contributor
  • 863 Views
  • 1 replies
  • 0 kudos

Databricks Initial Costs AWS

I have a new premium account. I set up a cost dashboard (see attached) after I create a new workspace using AWS Quickstart, where I see some costs. Why do I have this If I am not using Databricks at all? How can I save the costs?

databricks.png
  • 863 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Are you seeing this data from the Usage tab in the Account console? Does it allow you to filter it by SKU?

  • 0 kudos
Rafael-Sousa
by Contributor II
  • 1621 Views
  • 2 replies
  • 1 kudos

Resolved! Internal Error with MERGE Command in Spark SQL

I'm trying to perform a MERGE between two tables (customers and customers_update) using Spark SQL, but I’m encountering an internal error during the planning phase. The error message suggests it might be a bug in Spark or one of the plugins in use.He...

  • 1621 Views
  • 2 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

The issue you encountered with the MERGE statement in Spark SQL, which was resolved by specifying the database and metastore, is likely related to how Spark handles table references during the planning phase. The internal error you faced suggests a b...

  • 1 kudos
1 More Replies
Fkebbati
by New Contributor II
  • 935 Views
  • 1 replies
  • 0 kudos

Ingress/Egress private endpoint

Hello , We have configured our Databricks environment with private endpoint connections injected into our VNET, which includes two subnets (public and private). We have disabled public IPs and are using Network Security Groups (NSGs) on the subnet, a...

  • 935 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @Fkebbati ,There always be some costs related to data transfer between those account. Let's have a look at private link pricing page. So it's expected, but MS likes to hide this kind of information  

  • 0 kudos
Phani1
by Valued Contributor II
  • 13467 Views
  • 4 replies
  • 2 kudos

Masking techniques for more PII columns

Hi Databricks Team,We would appreciate it if you could inform us about the situations when Column-Masking, Row-Level Filtering, and Attributed-Based Masking should be utilized, as well as the recommended technique for handling large data volumes cont...

  • 13467 Views
  • 4 replies
  • 2 kudos
Latest Reply
NandiniN
Databricks Employee
  • 2 kudos

Agree with @Meghla-C , https://databricks.aha.io/ideas/ideas/DB-I-7941 this was the feature request and if you see the status, it is in preview.

  • 2 kudos
3 More Replies
ladyleet
by New Contributor
  • 727 Views
  • 1 replies
  • 1 kudos

Best way to find Databricks Certified Professionals?

Hi all! We have a few Databricks certified folks on our team, but we are looking for more! What is the best way to find a list of or know who is certified? We are looking for North America / Europe / LATAM / South America. I am part of the Microsoft ...

  • 727 Views
  • 1 replies
  • 1 kudos
Latest Reply
Rafael-Sousa
Contributor II
  • 1 kudos

Hey @ladyleet ,There is the community of Databricks certified: Databricks Certified Credential Holder Directory.

  • 1 kudos
Amrit23
by New Contributor
  • 1207 Views
  • 1 replies
  • 0 kudos

Amazon MSK integration with Databricks

Hello Everyone,I am a beginner in the world of Data bricks. I am trying to achieve a use case which involves : consuming messages from Amazon MSK and creating a Delta table in Data bricks.I need to get some insights on what all accesses are supposed ...

  • 1207 Views
  • 1 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hey @Amrit23 , Databricks Assistant can help you write the code. You need to use spark.readStream() to access the stream.  https://docs.databricks.com/en/connect/streaming/kafka.html And for the Access Permissions: Amazon MSK: Ensure you have the nec...

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels