- 2136 Views
- 2 replies
- 0 kudos
Resolved! Reduce Query Latency
While preparing for Databricks Engineer Associate exam, came across a below question.Question: he data analysis team has reported slow Databricks SQL query performance when using their always-on SQL endpoint, especially during periods of high concurr...
- 2136 Views
- 2 replies
- 0 kudos
- 1704 Views
- 3 replies
- 1 kudos
Resolved! incorrect commit timestamp after deep clone.
i have deep cloned a table, then did update but the update time stamp is less than deep clone timestamp version 0.look like there is an issue in the deep clone. here is the output, _commit_timestamp order is not in sync with _commit_versiontimecard_...
- 1704 Views
- 3 replies
- 1 kudos
- 1 kudos
do we need to do time synchronization in serverless. as we are planning to move from job cluster to serverless and using commit_timestamp for CDF it can cause issues. thankssugun
- 1 kudos
- 2001 Views
- 1 replies
- 1 kudos
Not able to call external API when using Databricks free edition
Hello guys , I'm new to Databricks and trying to create ETL pipelines for my personal projects . I'm using the Databricks free Edition but whenever I try to call an API to an external source to extract Data I always get this error .I have attached th...
- 2001 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @AKB_10 This is a common issue with Databricks Community Edition (free tier). Here's what's happening and how to fix it:Root CauseDatabricks Community Edition has network restrictions that block outbound API calls to external services like Spotify...
- 1 kudos
- 7449 Views
- 11 replies
- 1 kudos
SET configuration in SQL DLT pipeline does not work
Hi,I'm trying to set a dynamic value to use in a DLT query, and the code from the example documentation does not work.SET startDate='2020-01-01'; CREATE OR REFRESH LIVE TABLE filtered AS SELECT * FROM my_table WHERE created_at > ${startDate};It is g...
- 7449 Views
- 11 replies
- 1 kudos
- 1 kudos
@smit_tw Have you tried setting it on the "Advanced" tab as my previous suggests?
- 1 kudos
- 960 Views
- 3 replies
- 0 kudos
Compute and SQL Warehouse Permission button has disappeared
Hi,I can no longer see the permission button for all my all-purpose and SQL WH compute options in any of my workspaces. I've tried everything I can think off, but no joy. I'm an account admin.Has anyone else seen this or experiencing the same probl...
- 960 Views
- 3 replies
- 0 kudos
- 0 kudos
Check to see if you are in the Admins group for each workspace you see this issue.
- 0 kudos
- 1205 Views
- 3 replies
- 0 kudos
Error Creating the Workspace
Hi Team, Good Day!Getting the below error message when creating the Databricks Workspace, Moreover databricks is inside the VPC with private subnets.We already have a separate workspace which is using the same private subnet. Now when i try to create...
- 1205 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @gowthamr-TU ,I don't have much experiance with AWS, but the error message is quite straightforward. Check your inbound rules of the Network ACL (acl-0e9b635f05cdd3f9a) associated with the subnets. It looks like this ACL is blocking inbound traffi...
- 0 kudos
- 2096 Views
- 2 replies
- 4 kudos
error: [RETRIES_EXCEEDED] The maximum number of retries has been exceeded.
Whenever trying to run any sql or python command showing this error, I am running on server less compute. I am new to Databricks and starting to practice using free edition, I have tried restoring the cluster it is not working
- 2096 Views
- 2 replies
- 4 kudos
- 4 kudos
Hello @Bedd__! Can you please try to detach and then reattach the compute? Also, try switching the environment version and applying the changes to see if that helps resolve the issue.
- 4 kudos
- 1216 Views
- 2 replies
- 0 kudos
Import excel spreadsheet to a database table
How to Import excel spreadsheet to a database table.
- 1216 Views
- 2 replies
- 0 kudos
- 0 kudos
An alternative is also to store the flat file into a Volume in Unity Catalog. You can then access it in a similar way: import pandas as pd volume_path= "/Volumes/<catalog>/<schema>/<volume_folder>/your_file.xlsx" pd.read_excel(volume_path, sheet_nam...
- 0 kudos
- 12888 Views
- 7 replies
- 1 kudos
[RETRIES_EXCEEDED] Error When Displaying DataFrame in Databricks Using Serverless Compute
Hi Databricks Community,I am encountering an issue when trying to display a DataFrame in a Python notebook using serverless compute. The operation seems to fail after several retries, and I get the following error message:[RETRIES_EXCEEDED] The maxim...
- 12888 Views
- 7 replies
- 1 kudos
- 1 kudos
I'm also getting the same error. I'm trying to create a CATALOG. %sqlCREATE CATALOG IF NOT EXISTS `catalog_sql_databricks`;USE CATALOG `catalog_sql_databricks`;[RETRIES_EXCEEDED] The maximum number of retries has been exceeded.
- 1 kudos
- 1261 Views
- 1 replies
- 2 kudos
Resolved! Prakash Hinduja Geneva (Swiss) Can I use tools like Great Expectations with Databricks?
Hi everyone,I am Prakash Hinduja from Geneva, Switzerland (Swiss) currently exploring ways to improve data quality checks in my Databricks pipelines and came across Great Expectations. I’d love to know if anyone here has experience using it with Data...
- 1261 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi Prakash,Yes, Great Expectations integrates well with Databricks and is commonly used to enforce data quality checks in pipelines. For example, validating schema, nulls, ranges, or business rules. You can use it in a few ways: Directly in Python n...
- 2 kudos
- 6882 Views
- 5 replies
- 1 kudos
Put file into volume within Databricks
Hi! From a Databricks job, i want to copy a workspace file into volume. how can i do that?I tried`dbutils.fs.cp("/Workspace/path/to/the/file", "/Volumes/path/to/destination")`but got Public DBFS root is disabled. Access is denied on path: /Workspac...
- 6882 Views
- 5 replies
- 1 kudos
- 1 kudos
I do have one question, i think this post is the best suitable. I do want to override a wheel files into a Volume i do have already created in my CICD process. I do have something like this: - ${{if parameters.filesPackages}}: - $...
- 1 kudos
- 798 Views
- 1 replies
- 0 kudos
[INTERNAL_ERROR] Cannot refresh quality dashboard
Hi all,I'm encountering an INTERNAL_ERROR issue when refreshing a Databricks Lakehouse Monitoring job. Here's the full error message:`ProfilingError: INTERNAL_ERROR. Please contact the Databricks team for further assistance and include the refresh id...
- 798 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you! I'll modify my query based on your explanation. Currently, I'm manually parsing the custom metrics output data types, which works but isn't ideal. I'll implement proper data type formatting through asset bundles to ensure the UI receives c...
- 0 kudos
- 974 Views
- 2 replies
- 0 kudos
Enabled IP access list for azure databricks workspace but it is not working
Hi,We enabled IP access list for azure databricks workspace using REST API and we are able to see the IPs in allow and block list but it is not working and we are able to login to Web UI from any IP address and run the queries. Does this approach not...
- 974 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @san11 what is the error you are getting? I mean on any error on web uican you also share the screenshot of the IPs which you allowed on the azure portal?
- 0 kudos
- 631 Views
- 2 replies
- 1 kudos
Resolved! which option is better
Can anyone please help me which option is better and why ?Thanks a lot
- 631 Views
- 2 replies
- 1 kudos
- 1 kudos
+1 @radothede @Pratikmsbsvm How are you transforming from bronze to silver? DLT?
- 1 kudos
- 39834 Views
- 10 replies
- 4 kudos
Resolved! To get Number of rows inserted after performing an Insert operation into a table
Consider we have two tables A & B.qry = """INSERT INTO Table ASelect * from Table B where Id is null """spark.sql(qry)I need to get the number of records inserted after running this in databricks.
- 39834 Views
- 10 replies
- 4 kudos
- 4 kudos
in case someone is looking for purely SQL based solution: (add LIMIT 1 to the query if you are looking for last op only) select t.timestamp, t.operation, t.operationMetrics.numOutputRows as numOutputRows from ( DESCRIBE HISTORY <catalog>.<schema>....
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
