cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

mexcram
by New Contributor II
  • 305 Views
  • 1 replies
  • 0 kudos

Workflow entry point not working on runtime 16.4-LTS

Hello all,I'm developing a python code that it is packaged as a wheel and installed inside a docker image. However, since this program requires numpy >= 2.0 I'm forced to use the runtime 16.4-LTS.When I try to run it as a workflow on databricks I'm e...

Screenshot 2025-06-18 at 07.53.09.png Screenshot 2025-06-18 at 07.54.24.png
  • 305 Views
  • 1 replies
  • 0 kudos
Latest Reply
Nishant1994
New Contributor II
  • 0 kudos

Facing same issue . 

  • 0 kudos
shrutigupta12
by New Contributor II
  • 4666 Views
  • 10 replies
  • 2 kudos

Resolved! DataBricks Certification Exam Got Suspended. Require Immediate support

Hello @Cert-Team  @Certificate Team,Request Id# 00432042I encountered a pathetic experience while attempting my Databricks Certified Data Engineer Professional certification exam. This is a completely unethical process to harass the examinee and lose...

  • 4666 Views
  • 10 replies
  • 2 kudos
Latest Reply
whocomplains
New Contributor II
  • 2 kudos

I've been asked to constantly show my room, multiple times without any reason, which is a violation of my privacy. Who knows what/who is watching your room on the other side, and besides, there is no reason given, even though I asked multiple times f...

  • 2 kudos
9 More Replies
Sifflet
by New Contributor II
  • 161 Views
  • 1 replies
  • 1 kudos

Best practices for reducing noise in data quality monitoring?

Hi all,We’ve been improving our data quality monitoring for several pipelines, but we keep running into the same problem — too many alerts, most of which aren’t actionable. Over time, it becomes harder to trust them.Right now, we’re doing:Freshness c...

  • 161 Views
  • 1 replies
  • 1 kudos
Latest Reply
WiliamRosa
New Contributor II
  • 1 kudos

Hi @Sifflet, This is genuinely complex—and while you mentioned alerting and monitoring, in my experience the biggest lever to reduce noise is to treat problems at the source (i.e., in the transformation layer). Make the transformations enforce the co...

  • 1 kudos
Anubhav2603
by New Contributor
  • 253 Views
  • 1 replies
  • 0 kudos

DLT Pipeline

I am working on DLT pipeline and have one question. As explained on this page (https://docs.databricks.com/aws/en/dlt/tutorial-pipelines?language=Python), we will end up creating 3 tables i.e. customer_cdc_bronze, customer_cdc_clean and customer. Thi...

  • 253 Views
  • 1 replies
  • 0 kudos
Latest Reply
nayan_wylde
Honored Contributor
  • 0 kudos

With this optimized approach, I would suggest creating view for Clean table:1. Bronze Table: Raw CDC data (full storage)2. Clean View: No physical storage - computed on-demand3. Silver Table: Final processed data with SCD2 historyResult: ~67% storage...

  • 0 kudos
Mihai_Cog
by Contributor
  • 9550 Views
  • 7 replies
  • 0 kudos

Issue with Visual Studio Code Databricks extension

Hello,I successfully installed the extension and connected it to my databricks account. But when I try to select the repo (which already exists under repos in my databricks repo account) for syncing , I don't see it.I use Azure Devops (Git repo) as s...

  • 9550 Views
  • 7 replies
  • 0 kudos
Latest Reply
LearningRate
New Contributor II
  • 0 kudos

I installed VS code 2 (strange name and version from MS) then i see all the option .... thanks just if someone also facing this issue

  • 0 kudos
6 More Replies
Judith
by New Contributor III
  • 4050 Views
  • 3 replies
  • 3 kudos

Connect to Onelake using Service Principal, Unity Catalog and Databricks Access Connector

We are trying to connect Databricks to OneLake, to read data from a Fabric workspace into Databricks, using a notebook. We also use Unity Catalog. We are able to read data from the workspace with a Service Principal like this:from pyspark.sql.types i...

Judith_0-1739892045239.png Judith_1-1739891020619.png
  • 4050 Views
  • 3 replies
  • 3 kudos
Latest Reply
nayan_wylde
Honored Contributor
  • 3 kudos

One thing you can check if the Databricks Access connector have storage blob contributor access on the Datalake

  • 3 kudos
2 More Replies
prakashhinduja2
by New Contributor
  • 145 Views
  • 1 replies
  • 0 kudos

Prakash Hinduja (Geneva) How fix SQL errors like INVALID_IDENTIFIER when running workflows?

I am Prakash Hinduja, a global financial strategist with deep roots in India and a current base in Geneva Switzerland (Swiss).I have been running into SQL errors like INVALID_IDENTIFIER when executing workflows in Databricks. I’ve checked for typos a...

  • 145 Views
  • 1 replies
  • 0 kudos
Latest Reply
Khaja_Zaffer
Contributor
  • 0 kudos

Hello @prakashhinduja2 Are you getting error with this code? SQLSTATE: 42602Does your code has any Unquoted identifiers?

  • 0 kudos
smanblicks
by New Contributor II
  • 214 Views
  • 3 replies
  • 1 kudos

Resolved! Not able to create Databriks Compute in Central US

I Created my databricks account in Central US and I am not able to create Compute . So  I need help to create Compute.

  • 214 Views
  • 3 replies
  • 1 kudos
Latest Reply
brycejune
New Contributor III
  • 1 kudos

If everything looks fine, opening a support ticket with Databricks or your cloud provider (Azure/AWS) would be the quickest way to resolve this and get your compute set up.

  • 1 kudos
2 More Replies
giuseppe_esq
by New Contributor III
  • 371 Views
  • 5 replies
  • 1 kudos

Resolved! Error when trying to Create a STREAMING TABLE using Databricks SQL

Hi all,I am currently using the Databricks Free Edition, and trying to create a streaming table, using the following SQL script:%sqlCREATE OR REFRESH STREAMING TABLE sql_csv_autoloaderSCHEDULE EVERY 1 WEEKASSELECT *FROM STREAM read_files(  '/Volumes/...

  • 371 Views
  • 5 replies
  • 1 kudos
Latest Reply
WayneRevenite
New Contributor III
  • 1 kudos

Yes - I was able to do it no problem on Free, with the exact steps above, using the default supplied compute.

  • 1 kudos
4 More Replies
momo0101
by New Contributor II
  • 368 Views
  • 2 replies
  • 2 kudos

Resolved! Databricks Claude Access Error - Permission Denied

I'm using databricks-claude-sonnet-3.7 through Azure Databricks, and it was working until yesterday, but when I accessed it now, I got this error: Error: 403 {"error_code":"PERMISSION_DENIED","message":"PERMISSION_DENIED: Endpoint databricks-claude-s...

  • 368 Views
  • 2 replies
  • 2 kudos
Latest Reply
momo0101
New Contributor II
  • 2 kudos

@szymon_dybczak Thank you! I'll wait a couple days and try again. Much appreciated!

  • 2 kudos
1 More Replies
varshanagarajan
by New Contributor
  • 1379 Views
  • 2 replies
  • 1 kudos

Pandas API on Spark creates huge query plans

Hello,I have a piece of code written in Pyspark and Pandas API on Spark. On comparing the query plans, I see Pandas API on Spark creates huge query plans whereas Pyspark plan is a tiny one. Furthermore, with Pandas API on spark, we see a lot of incon...

  • 1379 Views
  • 2 replies
  • 1 kudos
Latest Reply
BS_THE_ANALYST
Honored Contributor III
  • 1 kudos

@FRB1984 could you provide some examples? I'm curious. My first thoughts would be around the shuffling. Check this out: https://spark.apache.org/docs/3.5.4/api/python/user_guide/pandas_on_spark/best_practices.html . There's an argument to be made abo...

  • 1 kudos
1 More Replies
sumner-williams
by New Contributor II
  • 189 Views
  • 1 replies
  • 1 kudos

Resolved! Table Counts

Hello,My company loads a lot of tables into a databricks schema. I would like to build a dashboard on what has been loaded, but SQL commands like select * from information_schema do not work. Instead we have SHOW TABLES {FROM} LIKE {}; And that fails...

  • 189 Views
  • 1 replies
  • 1 kudos
Latest Reply
BS_THE_ANALYST
Honored Contributor III
  • 1 kudos

Just trying to rule out some of the lower-hanging stuff. When you run your SQL statements i.e.  select * from information_schemaAre you using the correct namespace syntax i.e. {catalog_here}.information_schemaAre you using Unity Catalog?Example of th...

  • 1 kudos
DataDp
by New Contributor
  • 215 Views
  • 3 replies
  • 0 kudos

What permissions are needed to fix [INSUFFICIENT_PERMISSIONS] User does not have permission toSELECT

 Hi,I am getting the following error in Databricks when running a SELECT query:  [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501Context:Environment: Unity Catalog enabledI am tryin...

  • 215 Views
  • 3 replies
  • 0 kudos
Latest Reply
WiliamRosa
New Contributor II
  • 0 kudos

If you’re getting an “Insufficient Permissions” error in Databricks, it usually means your user is missing one or more privileges required for the action you’re trying to perform. In Unity Catalog, for example, querying a view in dedicated compute mo...

  • 0 kudos
2 More Replies
leticialima__
by New Contributor III
  • 553 Views
  • 4 replies
  • 2 kudos

Resolved! add new column to a table and failing the previous jobs

 Hello community! I’m new to Databricks and currently working on a project structured in Bronze / Silver / Gold layers using Delta Lake and Change Data Feed.I recently added 3 new columns to a table and initially applied these changes via PySpark SQ...

  • 553 Views
  • 4 replies
  • 2 kudos
Latest Reply
Khaja_Zaffer
Contributor
  • 2 kudos

Hello @leticialima__ Good dayCan you please share the error observed on the driver log. is it : [Errno 13] Permission denied or No such file or directory? Please let me know the error on the driver log.  THank you. 

  • 2 kudos
3 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels