- 446 Views
- 1 replies
- 2 kudos
Resolved! Sample Data Reflecting but Uploaded File reflecting
Step1: I uploaded CSV file manually in data bricks Step 2: Connector created and active between Salesforce and DatabricksStep 3: Creating Data Streams in Salesforce Data CloudSample Topics are reflecting , matching between what i see in data bricks ...
- 446 Views
- 1 replies
- 2 kudos
- 2 kudos
I resolved it myselfStep1: workspace --> manage permissions step 2: chose all permissionsstep 3: went to raw uploaded file and share via delta sharingStep4: in salesforce data stream i got the raw file
- 2 kudos
- 2616 Views
- 2 replies
- 1 kudos
Databricks support updating multiple target rows with single matching source row in merge query?
Hi,I am getting this error in merge statement. DeltaUnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same target row in the Delta table in possibly conflicting ways.Does Databricks suppor...
- 2616 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @szymon_dybczak ,Thanks for your reply. The above code is working fine which means multiple updates can be done from a single source target. So, it may be when there are complex matching conditions/values, merge query gives error.I cannot send you...
- 1 kudos
- 8081 Views
- 6 replies
- 6 kudos
Usage of Azure DevOps System.AccessToken as PAT in Databricks
Hi there! I'm trying to use Azure DevOps Pipeline to automate Azure Databricks Repos API. Im using the following workflow:Get an Access Token for a Databricks Service Principal using a Certificate (which works great)Usage REST Api to generate Git Cre...
- 8081 Views
- 6 replies
- 6 kudos
- 6 kudos
@Adrian Ehrsam​ The PAT limit has been increased to 2048 now. Please check.
- 6 kudos
- 1382 Views
- 1 replies
- 2 kudos
Merge slows down when the table grows with liquid clustering enabled.
Hi Everyone, I have a source table and target table and MERGE statement that is inserting/updating records every couple of minutes. The clustering keys are set up to match the 2 merge join columns.I noticed that with time the processing time increase...
- 1382 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @filipniziol ,I dealt with a large table of about a TB in size with liquid clustering enabled. Even with Liquid Clustering, selects and joins on the clustered columns took longer as the table grew. So I don't think it performs as fast as the table...
- 2 kudos
- 1692 Views
- 5 replies
- 0 kudos
Databricks System Table system.billing.usage Not Capturing Job Data in Real-Time
We’ve observed that the system.billing.usage table in Databricks is not capturing job usage data in real-time. There appears to be a noticeable delay between when jobs are executed and when their corresponding usage records appear in the system table...
- 1692 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @szymon_dybczak ,Is there any alternative approach to find the DBU usage of current running jobs ?
- 0 kudos
- 809 Views
- 2 replies
- 1 kudos
Resolved! Not able to find lab for Data Engineering Learning Path
I am not able to find the data engineering learning path , i did open partner databricks academy lab which redirected to uplimit where i also enrolled myself to instructor led course but not able to see any labs.
- 809 Views
- 2 replies
- 1 kudos
- 1 kudos
You are right the self paced e-learning courses does not include any labs. However, they are available on instructor led courses available on Uplimit. I recently enrolled for one and here is how it worked for me:1. On Uplimit portal enroll for an upc...
- 1 kudos
- 1632 Views
- 3 replies
- 0 kudos
Resolved! Authentication failure Lakeflow SQL Server Ingestion
Hi all I am trying to create a Lakeflow Ingestion Pipeline for SQL Server, but I am running into the following authentication error when using my Databricks Database User for the connection:Gateway is stopping. Authentication failure while obtaining ...
- 1632 Views
- 3 replies
- 0 kudos
- 700 Views
- 1 replies
- 0 kudos
Programmatically set minimum workers for a job cluster based on file size?
I’m running an ingestion pipeline with a Databricks job:A file lands in S3A Lambda is triggeredThe Lambda runs a Databricks jobThe incoming files vary a lot in size, which makes processing times vary as well. My job cluster has autoscaling enabled, b...
- 700 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Alena, Jobs API has update functionality to be able to do that: https://docs.databricks.com/api/workspace/jobs_21/updateIf for some reason you can’t update your pipeline before you trigger it you can also consider creating a new job with desired c...
- 0 kudos
- 906 Views
- 2 replies
- 0 kudos
Question on best method to deliver Azure SQL Server data into Databricks Bronze and Silver.
Hi,We have a Azure SQL Server (replicating from an On Prem SQL Server) that is required to be in Databricks bronze and beyond.This database has 100s of tables that are all required. Size of tables will vary from very small up to the biggest tables 1...
- 906 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Nick,Have you tried the SQL Server connector with Lakeflow Connect? This should provide native connection to your SQL server, potentially allowing for incremental updates and CDC setup. https://learn.microsoft.com/en-us/azure/databricks/ingestion...
- 0 kudos
- 558 Views
- 1 replies
- 0 kudos
Unable to Upcast DECIMAL Field in Autoloader
I’m using Autoloader to read Parquet files and write them to a Delta table. I want to enforce a schema in which Column1 is defined as DECIMAL(10,2). However, in the Parquet files being ingested, Column1 is defined as DECIMAL(8,2).When Autoloader read...
- 558 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Yit,To potentially simplify your issue, why not read this column as String in your stream and then cast it to DECIMAL(10, 2) afterwards? That should eliminate the rescue behaviour. Kerem Durak
- 0 kudos
- 629 Views
- 2 replies
- 0 kudos
Resolved! Compute kind SERVERLESS_REPL_VM is not allowed to use cluster scoped libraries.
i have a s3 uri 's3://salesforcedatabricksorders/orders_data.xlsx' , i have created a connector between data bricks and salesfoce, i am first gettig the orders_data.xlsx to databricks layer perform basic transformation on it and then send it to sales...
- 629 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello,I’ve come across the same issue reading an Excel file into a PySpark dataframe via Serverless compute. As the error states with Serverless, you cannot install a cluster scoped library so you have to use notebook scoped libraries (%pip install…)...
- 0 kudos
- 1296 Views
- 1 replies
- 1 kudos
Resolved! How to Create Metadata driven Data Pipeline in Databricks
I am creating a Data Pipeline as shown below.1. Files from multiple input source is coming to respective folder in bronze layer.2. Using Databricks to perform Transformation and load transformed data to Azure SQL. also to ADLS Gen2 Silver (not shown ...
- 1296 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Pratikmsbsvm ,It's totally realistic requirement. In fact you can find many articles that suggests some approaches how to design such control table. Take for example following article: https://medium.com/dbsql-sme-engineering/a-primer-for-metadat...
- 1 kudos
- 1941 Views
- 2 replies
- 0 kudos
how to restrict creation serving endpoints in databricks to a user
Is it possible somehow to restrict creation of the serving endpoints to specific users? I want to grant Workspace access under the Entitlements of the specific group, but I want not to allow users of this group create serving endpoints.The only way I...
- 1941 Views
- 2 replies
- 0 kudos
- 954 Views
- 1 replies
- 1 kudos
Resolved! How to Retrieve the spark.statistics.createdAt When Statistics Were Last Updated in Databricks?
Hi everyone,I regularly (once a week) run the analyze table compute statistics command on all my tables in Databricks to keep statistics up to date for query optimization.In the Spark table UI catalog, I can see some statistics metadata like spark.st...
- 954 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Sainath368! sql.statistics.createdAt reflects the epoch time when statistics were created. Unfortunately, there's no direct command available to check when the statistics were last updated. As a workaround, you can manually set the current tim...
- 1 kudos
- 1313 Views
- 3 replies
- 1 kudos
dbutils.notebook.run() getting general error instead specific
Hi, In a python file using dbutils.notebook.run() I'm running specific notebook.The notebook is failling but i'm getting a general error log instead the real specific log.When I'm running the notebook directly - I'm getting the specific error log.gen...
- 1313 Views
- 3 replies
- 1 kudos
- 1 kudos
@Vinay_M_RBTW, when trying to run a job using Databricks API, I encounter the same issue (general "FAILED: Workload failed"):from databricks.sdk import WorkspaceClient client = WorkspaceClient() run = client.jobs.run_now(job_id) error message:state_...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
