- 248 Views
- 1 replies
- 0 kudos
Resolved! Databricks Free Edition | RETRIES_EXCEEDED issue
Hi Team,I am not able to fire below commands i am getting error %sqlCREATE SCHEMA IF NOT EXISTS workspace.gold; The maximum number of retries has been exceeded.Tried all the available option but not working ThanksNikhil Samaiya
- 248 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @samaiyanik! Could you please try the suggestions shared in the post below and let us know if that helps resolve the issue?Similar Post: error: [RETRIES_EXCEEDED] The maximum number of retries has been exceeded
- 0 kudos
- 679 Views
- 3 replies
- 1 kudos
Databricks recommended Approach to load data vault 2.0
Hi,Please share the recommended approach to load Data Vault 2.0 .Overview1. Current Landscape - Lakehouse (Bronze/Silver/Gold)2. Data Vault 2.0 to be created in Silver layer.3. Bronze data will be made available in delta table using ETL Questions1. ...
- 679 Views
- 3 replies
- 1 kudos
- 1 kudos
Kindly provide your valuable input and suggestion for the above questions
- 1 kudos
- 3344 Views
- 5 replies
- 0 kudos
Spark SQL vs serverless SQL
Are there any benchmarks showing performance and cost differences between running SQL workloads on Spark SQL vs Databricks SQL (specially serverless SQL)?Our customer is hesitant about getting locked into Databricks SQL as opposed to being able to ru...
- 3344 Views
- 5 replies
- 0 kudos
- 0 kudos
Spark SQL serves as the SQL interface for Spark applications, whereas Databricks SQL is a more advanced, warehouse-optimized product built around SQL Warehouses, which utilize multiple Spark clusters. This architectural difference can lead to noticea...
- 0 kudos
- 387 Views
- 3 replies
- 4 kudos
DLT table deletion
If we delete the DLT pipeline, the tables would get deleted.But in a DLT pipeline which creates 5 tables , if I comment out logic of 1 table, that table is not deleted from the catalog, even though full refresh of the pipeline is done.Does anyone kno...
- 387 Views
- 3 replies
- 4 kudos
- 4 kudos
Don't confuse DLT and LDP (Lakeflow Declarative Pipelines) as though behind the scenes they work very similarly, the UI and the developer experience has changed immensely and very important new features have been added. I used DLT extensively and in ...
- 4 kudos
- 346 Views
- 1 replies
- 0 kudos
Troubleshooting AutoLoader
Hi there, I am running into a bit of an issue displaying some AutoLoader readStream data. Can I get some assistance to understand how to properly troubleshoot this? I've looked at logs before, but frankly it's not clear where to look exactly:First, "...
- 346 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm also working with AutoLoader these days to create an ingestion pattern and troubleshooting it can be tricky.I wonder if you could pick a single file (whose full path / location / uri you exactly know) and read it without autoloader. Just with spa...
- 0 kudos
- 291 Views
- 1 replies
- 2 kudos
Resolved! Sample Data Reflecting but Uploaded File reflecting
Step1: I uploaded CSV file manually in data bricks Step 2: Connector created and active between Salesforce and DatabricksStep 3: Creating Data Streams in Salesforce Data CloudSample Topics are reflecting , matching between what i see in data bricks ...
- 291 Views
- 1 replies
- 2 kudos
- 2 kudos
I resolved it myselfStep1: workspace --> manage permissions step 2: chose all permissionsstep 3: went to raw uploaded file and share via delta sharingStep4: in salesforce data stream i got the raw file
- 2 kudos
- 664 Views
- 2 replies
- 1 kudos
Databricks support updating multiple target rows with single matching source row in merge query?
Hi,I am getting this error in merge statement. DeltaUnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same target row in the Delta table in possibly conflicting ways.Does Databricks suppor...
- 664 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @szymon_dybczak ,Thanks for your reply. The above code is working fine which means multiple updates can be done from a single source target. So, it may be when there are complex matching conditions/values, merge query gives error.I cannot send you...
- 1 kudos
- 6669 Views
- 6 replies
- 6 kudos
Usage of Azure DevOps System.AccessToken as PAT in Databricks
Hi there! I'm trying to use Azure DevOps Pipeline to automate Azure Databricks Repos API. Im using the following workflow:Get an Access Token for a Databricks Service Principal using a Certificate (which works great)Usage REST Api to generate Git Cre...
- 6669 Views
- 6 replies
- 6 kudos
- 6 kudos
@Adrian Ehrsam​ The PAT limit has been increased to 2048 now. Please check.
- 6 kudos
- 313 Views
- 1 replies
- 0 kudos
Request to create a cluster failed with an exception: RESOURCE_EXHAUSTED
My compute is erroring out with errorClusters are failing to launch. Cluster launch will be retried.Request to create a cluster failed with an exception: RESOURCE_EXHAUSTED: Cannot create the resource, please try again later.Any suggestions ?
- 313 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ManojkMohan! This error may occur if your account has reached the serverless compute quota in the region. This quota limits the number of serverless compute resources you can use concurrently. You can find more details here: Quotas for Serverl...
- 0 kudos
- 914 Views
- 1 replies
- 2 kudos
Merge slows down when the table grows with liquid clustering enabled.
Hi Everyone, I have a source table and target table and MERGE statement that is inserting/updating records every couple of minutes. The clustering keys are set up to match the 2 merge join columns.I noticed that with time the processing time increase...
- 914 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @filipniziol ,I dealt with a large table of about a TB in size with liquid clustering enabled. Even with Liquid Clustering, selects and joins on the clustered columns took longer as the table grew. So I don't think it performs as fast as the table...
- 2 kudos
- 768 Views
- 5 replies
- 0 kudos
Databricks System Table system.billing.usage Not Capturing Job Data in Real-Time
We’ve observed that the system.billing.usage table in Databricks is not capturing job usage data in real-time. There appears to be a noticeable delay between when jobs are executed and when their corresponding usage records appear in the system table...
- 768 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @szymon_dybczak ,Is there any alternative approach to find the DBU usage of current running jobs ?
- 0 kudos
- 447 Views
- 2 replies
- 1 kudos
Resolved! Not able to find lab for Data Engineering Learning Path
I am not able to find the data engineering learning path , i did open partner databricks academy lab which redirected to uplimit where i also enrolled myself to instructor led course but not able to see any labs.
- 447 Views
- 2 replies
- 1 kudos
- 1 kudos
You are right the self paced e-learning courses does not include any labs. However, they are available on instructor led courses available on Uplimit. I recently enrolled for one and here is how it worked for me:1. On Uplimit portal enroll for an upc...
- 1 kudos
- 863 Views
- 3 replies
- 0 kudos
Resolved! Authentication failure Lakeflow SQL Server Ingestion
Hi all I am trying to create a Lakeflow Ingestion Pipeline for SQL Server, but I am running into the following authentication error when using my Databricks Database User for the connection:Gateway is stopping. Authentication failure while obtaining ...
- 863 Views
- 3 replies
- 0 kudos
- 297 Views
- 1 replies
- 0 kudos
Programmatically set minimum workers for a job cluster based on file size?
I’m running an ingestion pipeline with a Databricks job:A file lands in S3A Lambda is triggeredThe Lambda runs a Databricks jobThe incoming files vary a lot in size, which makes processing times vary as well. My job cluster has autoscaling enabled, b...
- 297 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Alena, Jobs API has update functionality to be able to do that: https://docs.databricks.com/api/workspace/jobs_21/updateIf for some reason you can’t update your pipeline before you trigger it you can also consider creating a new job with desired c...
- 0 kudos
- 673 Views
- 2 replies
- 0 kudos
Question on best method to deliver Azure SQL Server data into Databricks Bronze and Silver.
Hi,We have a Azure SQL Server (replicating from an On Prem SQL Server) that is required to be in Databricks bronze and beyond.This database has 100s of tables that are all required. Size of tables will vary from very small up to the biggest tables 1...
- 673 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Nick,Have you tried the SQL Server connector with Lakeflow Connect? This should provide native connection to your SQL server, potentially allowing for incremental updates and CDC setup. https://learn.microsoft.com/en-us/azure/databricks/ingestion...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 484 | |
| 349 | |
| 287 |
