cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Anubhav2603
by New Contributor
  • 623 Views
  • 2 replies
  • 0 kudos

DLT Pipeline Design

I am new to DLT and trying to understand the process. My bronze table will receive incremental data from SAP in real time. In my bronze table, we are not maintaining any history and any data older than 2 weeks will be deleted. This data from bronze w...

  • 623 Views
  • 2 replies
  • 0 kudos
Latest Reply
alexajames
New Contributor III
  • 0 kudos

The scenario you mentioned can be efficiently managed using a Delta Live Table (DLT) combined with a CDC (Change Data Capture) flow. From what I understand, the bronze table is always in append mode. Check out the link below for more details. Hope th...

  • 0 kudos
1 More Replies
MR_DHC
by New Contributor II
  • 514 Views
  • 6 replies
  • 2 kudos

Execute Immediate not working to fetch table name based on year

 I am trying to pass the year as argument so it can be used in the table name. Ex: there are tables like Claims_total_2021 , Claims_total_2022 and so on till 2025. Now I want to pass the year in parameter , say 2024 and it must fetch the table Claims...

  • 514 Views
  • 6 replies
  • 2 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 2 kudos

Hi, I believe the solution shared by Martison would fix this issue. In Databricks SQL, when using EXECUTE IMMEDIATE, the SQL string must be a single variable or single string literal, not an inline expression using string concatenation ('...' || clai...

  • 2 kudos
5 More Replies
BS_THE_ANALYST
by Esteemed Contributor II
  • 605 Views
  • 6 replies
  • 20 kudos

Resolved! Databricks Community Innovators - Program

Hi, I'd like to know what's happened with the Databricks Community innovators program? https://community.databricks.com/t5/databricks-community-innovators/bg-p/databricks-community-news-members Is this still alive? I've applied and emailed: databrick...

  • 605 Views
  • 6 replies
  • 20 kudos
Latest Reply
BS_THE_ANALYST
Esteemed Contributor II
  • 20 kudos

Hi Mandy! Thanks for the introduction and update. I'm really looking forward to being a part of everything moving forward. Just echoing what @TheOC mentioned above, I'm happy to help/assist in whatever way I can.Absolutely buzzing based off the info ...

  • 20 kudos
5 More Replies
BS_THE_ANALYST
by Esteemed Contributor II
  • 1104 Views
  • 11 replies
  • 43 kudos

Databricks Monthly Spotlight - Discontinued?

Hi, I'm curious what's happened with the Databricks monthly spotlight? https://community.databricks.com/t5/databricks-community-innovators/bg-p/databricks-community-news-members I can there hasn't been anyone in the spotlight since April 2025. Has th...

  • 1104 Views
  • 11 replies
  • 43 kudos
Latest Reply
TheOC
Contributor III
  • 43 kudos

Hi all,I just wanted to jump in this thread to say how motivating this kind of conversation is. It's great that we've got passion from users and Databricks employees alike to ensure integrity in the Community - it's really reassuring as someone start...

  • 43 kudos
10 More Replies
SinchBhat
by New Contributor
  • 307 Views
  • 3 replies
  • 3 kudos

Python

Guys am not able to access pyspark in free edition somebody help me.

  • 307 Views
  • 3 replies
  • 3 kudos
Latest Reply
BS_THE_ANALYST
Esteemed Contributor II
  • 3 kudos

@szymon_dybczak it may be that @SinchBhat has the notebook set to SQL as the default language ?Could also, by mistake, be using the SQL editor.@SinchBhat could you provide some screenshots of what your UI looks like please. It'll help to resolve the ...

  • 3 kudos
2 More Replies
mexcram
by New Contributor II
  • 421 Views
  • 1 replies
  • 0 kudos

Workflow entry point not working on runtime 16.4-LTS

Hello all,I'm developing a python code that it is packaged as a wheel and installed inside a docker image. However, since this program requires numpy >= 2.0 I'm forced to use the runtime 16.4-LTS.When I try to run it as a workflow on databricks I'm e...

Screenshot 2025-06-18 at 07.53.09.png Screenshot 2025-06-18 at 07.54.24.png
  • 421 Views
  • 1 replies
  • 0 kudos
Latest Reply
Nishant1994
New Contributor II
  • 0 kudos

Facing same issue . 

  • 0 kudos
joypillai
by New Contributor II
  • 244 Views
  • 3 replies
  • 2 kudos

Resolved! Not able to login Databricks

I am trying to login in databricks but when I enter the OTP it says something went wrong

  • 244 Views
  • 3 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 2 kudos

HI @joypillai ,You want to login to databricks free edition? Could you provide screenshot?

  • 2 kudos
2 More Replies
Sifflet
by New Contributor II
  • 274 Views
  • 1 replies
  • 1 kudos

Best practices for reducing noise in data quality monitoring?

Hi all,We’ve been improving our data quality monitoring for several pipelines, but we keep running into the same problem — too many alerts, most of which aren’t actionable. Over time, it becomes harder to trust them.Right now, we’re doing:Freshness c...

  • 274 Views
  • 1 replies
  • 1 kudos
Latest Reply
WiliamRosa
Contributor
  • 1 kudos

Hi @Sifflet, This is genuinely complex—and while you mentioned alerting and monitoring, in my experience the biggest lever to reduce noise is to treat problems at the source (i.e., in the transformation layer). Make the transformations enforce the co...

  • 1 kudos
Anubhav2603
by New Contributor
  • 612 Views
  • 1 replies
  • 0 kudos

DLT Pipeline

I am working on DLT pipeline and have one question. As explained on this page (https://docs.databricks.com/aws/en/dlt/tutorial-pipelines?language=Python), we will end up creating 3 tables i.e. customer_cdc_bronze, customer_cdc_clean and customer. Thi...

  • 612 Views
  • 1 replies
  • 0 kudos
Latest Reply
nayan_wylde
Honored Contributor III
  • 0 kudos

With this optimized approach, I would suggest creating view for Clean table:1. Bronze Table: Raw CDC data (full storage)2. Clean View: No physical storage - computed on-demand3. Silver Table: Final processed data with SCD2 historyResult: ~67% storage...

  • 0 kudos
Mihai_Cog
by Contributor
  • 10302 Views
  • 7 replies
  • 0 kudos

Issue with Visual Studio Code Databricks extension

Hello,I successfully installed the extension and connected it to my databricks account. But when I try to select the repo (which already exists under repos in my databricks repo account) for syncing , I don't see it.I use Azure Devops (Git repo) as s...

  • 10302 Views
  • 7 replies
  • 0 kudos
Latest Reply
LearningRate
New Contributor II
  • 0 kudos

I installed VS code 2 (strange name and version from MS) then i see all the option .... thanks just if someone also facing this issue

  • 0 kudos
6 More Replies
Judith
by New Contributor III
  • 5092 Views
  • 3 replies
  • 3 kudos

Connect to Onelake using Service Principal, Unity Catalog and Databricks Access Connector

We are trying to connect Databricks to OneLake, to read data from a Fabric workspace into Databricks, using a notebook. We also use Unity Catalog. We are able to read data from the workspace with a Service Principal like this:from pyspark.sql.types i...

Judith_0-1739892045239.png Judith_1-1739891020619.png
  • 5092 Views
  • 3 replies
  • 3 kudos
Latest Reply
nayan_wylde
Honored Contributor III
  • 3 kudos

One thing you can check if the Databricks Access connector have storage blob contributor access on the Datalake

  • 3 kudos
2 More Replies
prakashhinduja2
by New Contributor
  • 241 Views
  • 1 replies
  • 0 kudos

Prakash Hinduja (Geneva) How fix SQL errors like INVALID_IDENTIFIER when running workflows?

I am Prakash Hinduja, a global financial strategist with deep roots in India and a current base in Geneva Switzerland (Swiss).I have been running into SQL errors like INVALID_IDENTIFIER when executing workflows in Databricks. I’ve checked for typos a...

  • 241 Views
  • 1 replies
  • 0 kudos
Latest Reply
Khaja_Zaffer
Contributor III
  • 0 kudos

Hello @prakashhinduja2 Are you getting error with this code? SQLSTATE: 42602Does your code has any Unquoted identifiers?

  • 0 kudos
smanblicks
by New Contributor III
  • 677 Views
  • 3 replies
  • 1 kudos

Resolved! Not able to create Databriks Compute in Central US

I Created my databricks account in Central US and I am not able to create Compute . So  I need help to create Compute.

  • 677 Views
  • 3 replies
  • 1 kudos
Latest Reply
brycejune
New Contributor III
  • 1 kudos

If everything looks fine, opening a support ticket with Databricks or your cloud provider (Azure/AWS) would be the quickest way to resolve this and get your compute set up.

  • 1 kudos
2 More Replies
momo0101
by New Contributor II
  • 886 Views
  • 2 replies
  • 2 kudos

Resolved! Databricks Claude Access Error - Permission Denied

I'm using databricks-claude-sonnet-3.7 through Azure Databricks, and it was working until yesterday, but when I accessed it now, I got this error: Error: 403 {"error_code":"PERMISSION_DENIED","message":"PERMISSION_DENIED: Endpoint databricks-claude-s...

  • 886 Views
  • 2 replies
  • 2 kudos
Latest Reply
momo0101
New Contributor II
  • 2 kudos

@szymon_dybczak Thank you! I'll wait a couple days and try again. Much appreciated!

  • 2 kudos
1 More Replies
varshanagarajan
by New Contributor
  • 1555 Views
  • 2 replies
  • 1 kudos

Pandas API on Spark creates huge query plans

Hello,I have a piece of code written in Pyspark and Pandas API on Spark. On comparing the query plans, I see Pandas API on Spark creates huge query plans whereas Pyspark plan is a tiny one. Furthermore, with Pandas API on spark, we see a lot of incon...

  • 1555 Views
  • 2 replies
  • 1 kudos
Latest Reply
BS_THE_ANALYST
Esteemed Contributor II
  • 1 kudos

@FRB1984 could you provide some examples? I'm curious. My first thoughts would be around the shuffling. Check this out: https://spark.apache.org/docs/3.5.4/api/python/user_guide/pandas_on_spark/best_practices.html . There's an argument to be made abo...

  • 1 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels