- 344 Views
- 3 replies
- 3 kudos
Python
Guys am not able to access pyspark in free edition somebody help me.
- 344 Views
- 3 replies
- 3 kudos
- 3 kudos
@szymon_dybczak it may be that @SinchBhat has the notebook set to SQL as the default language ?Could also, by mistake, be using the SQL editor.@SinchBhat could you provide some screenshots of what your UI looks like please. It'll help to resolve the ...
- 3 kudos
- 488 Views
- 1 replies
- 0 kudos
Workflow entry point not working on runtime 16.4-LTS
Hello all,I'm developing a python code that it is packaged as a wheel and installed inside a docker image. However, since this program requires numpy >= 2.0 I'm forced to use the runtime 16.4-LTS.When I try to run it as a workflow on databricks I'm e...
- 488 Views
- 1 replies
- 0 kudos
- 0 kudos
Facing same issue .
- 0 kudos
- 280 Views
- 3 replies
- 2 kudos
Resolved! Not able to login Databricks
I am trying to login in databricks but when I enter the OTP it says something went wrong
- 280 Views
- 3 replies
- 2 kudos
- 2 kudos
HI @joypillai ,You want to login to databricks free edition? Could you provide screenshot?
- 2 kudos
- 295 Views
- 1 replies
- 1 kudos
Best practices for reducing noise in data quality monitoring?
Hi all,We’ve been improving our data quality monitoring for several pipelines, but we keep running into the same problem — too many alerts, most of which aren’t actionable. Over time, it becomes harder to trust them.Right now, we’re doing:Freshness c...
- 295 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Sifflet, This is genuinely complex—and while you mentioned alerting and monitoring, in my experience the biggest lever to reduce noise is to treat problems at the source (i.e., in the transformation layer). Make the transformations enforce the co...
- 1 kudos
- 717 Views
- 1 replies
- 0 kudos
DLT Pipeline
I am working on DLT pipeline and have one question. As explained on this page (https://docs.databricks.com/aws/en/dlt/tutorial-pipelines?language=Python), we will end up creating 3 tables i.e. customer_cdc_bronze, customer_cdc_clean and customer. Thi...
- 717 Views
- 1 replies
- 0 kudos
- 0 kudos
With this optimized approach, I would suggest creating view for Clean table:1. Bronze Table: Raw CDC data (full storage)2. Clean View: No physical storage - computed on-demand3. Silver Table: Final processed data with SCD2 historyResult: ~67% storage...
- 0 kudos
- 10604 Views
- 7 replies
- 0 kudos
Issue with Visual Studio Code Databricks extension
Hello,I successfully installed the extension and connected it to my databricks account. But when I try to select the repo (which already exists under repos in my databricks repo account) for syncing , I don't see it.I use Azure Devops (Git repo) as s...
- 10604 Views
- 7 replies
- 0 kudos
- 0 kudos
I installed VS code 2 (strange name and version from MS) then i see all the option .... thanks just if someone also facing this issue
- 0 kudos
- 5389 Views
- 3 replies
- 3 kudos
Connect to Onelake using Service Principal, Unity Catalog and Databricks Access Connector
We are trying to connect Databricks to OneLake, to read data from a Fabric workspace into Databricks, using a notebook. We also use Unity Catalog. We are able to read data from the workspace with a Service Principal like this:from pyspark.sql.types i...
- 5389 Views
- 3 replies
- 3 kudos
- 3 kudos
One thing you can check if the Databricks Access connector have storage blob contributor access on the Datalake
- 3 kudos
- 258 Views
- 1 replies
- 0 kudos
Prakash Hinduja (Geneva) How fix SQL errors like INVALID_IDENTIFIER when running workflows?
I am Prakash Hinduja, a global financial strategist with deep roots in India and a current base in Geneva Switzerland (Swiss).I have been running into SQL errors like INVALID_IDENTIFIER when executing workflows in Databricks. I’ve checked for typos a...
- 258 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @prakashhinduja2 Are you getting error with this code? SQLSTATE: 42602Does your code has any Unquoted identifiers?
- 0 kudos
- 742 Views
- 3 replies
- 1 kudos
Resolved! Not able to create Databriks Compute in Central US
I Created my databricks account in Central US and I am not able to create Compute . So I need help to create Compute.
- 742 Views
- 3 replies
- 1 kudos
- 1 kudos
If everything looks fine, opening a support ticket with Databricks or your cloud provider (Azure/AWS) would be the quickest way to resolve this and get your compute set up.
- 1 kudos
- 1057 Views
- 2 replies
- 2 kudos
Resolved! Databricks Claude Access Error - Permission Denied
I'm using databricks-claude-sonnet-3.7 through Azure Databricks, and it was working until yesterday, but when I accessed it now, I got this error: Error: 403 {"error_code":"PERMISSION_DENIED","message":"PERMISSION_DENIED: Endpoint databricks-claude-s...
- 1057 Views
- 2 replies
- 2 kudos
- 2 kudos
@szymon_dybczak Thank you! I'll wait a couple days and try again. Much appreciated!
- 2 kudos
- 1613 Views
- 2 replies
- 1 kudos
Pandas API on Spark creates huge query plans
Hello,I have a piece of code written in Pyspark and Pandas API on Spark. On comparing the query plans, I see Pandas API on Spark creates huge query plans whereas Pyspark plan is a tiny one. Furthermore, with Pandas API on spark, we see a lot of incon...
- 1613 Views
- 2 replies
- 1 kudos
- 1 kudos
@FRB1984 could you provide some examples? I'm curious. My first thoughts would be around the shuffling. Check this out: https://spark.apache.org/docs/3.5.4/api/python/user_guide/pandas_on_spark/best_practices.html . There's an argument to be made abo...
- 1 kudos
- 506 Views
- 1 replies
- 1 kudos
Resolved! Table Counts
Hello,My company loads a lot of tables into a databricks schema. I would like to build a dashboard on what has been loaded, but SQL commands like select * from information_schema do not work. Instead we have SHOW TABLES {FROM} LIKE {}; And that fails...
- 506 Views
- 1 replies
- 1 kudos
- 1 kudos
Just trying to rule out some of the lower-hanging stuff. When you run your SQL statements i.e. select * from information_schemaAre you using the correct namespace syntax i.e. {catalog_here}.information_schemaAre you using Unity Catalog?Example of th...
- 1 kudos
- 517 Views
- 3 replies
- 0 kudos
What permissions are needed to fix [INSUFFICIENT_PERMISSIONS] User does not have permission toSELECT
Hi,I am getting the following error in Databricks when running a SELECT query: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501Context:Environment: Unity Catalog enabledI am tryin...
- 517 Views
- 3 replies
- 0 kudos
- 0 kudos
If you’re getting an “Insufficient Permissions” error in Databricks, it usually means your user is missing one or more privileges required for the action you’re trying to perform. In Unity Catalog, for example, querying a view in dedicated compute mo...
- 0 kudos
- 833 Views
- 4 replies
- 2 kudos
Resolved! add new column to a table and failing the previous jobs
Hello community! I’m new to Databricks and currently working on a project structured in Bronze / Silver / Gold layers using Delta Lake and Change Data Feed.I recently added 3 new columns to a table and initially applied these changes via PySpark SQ...
- 833 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @leticialima__ Good dayCan you please share the error observed on the driver log. is it : [Errno 13] Permission denied or No such file or directory? Please let me know the error on the driver log. THank you.
- 2 kudos
- 255 Views
- 1 replies
- 0 kudos
Steamlit in Databricks
HiI have developed a streamlit app locally on my desktop using dummy data, and now I want to be able to use actual data stored in azure blog storage. I have tried to run the same code within a notebook, but keep on getting dependency errors. Is there...
- 255 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @aw1! What exact dependency errors or permission failures are you getting? Can you please share the error message?
- 0 kudos
