- 3170 Views
- 1 replies
- 0 kudos
What is the equivalent of "if exists()" in databricks sql?
What is the equivalent of the below sql server syntax in databricks sql? there are cases where i need to execute a block of sql code on certain conditions. I know this can be achieved with spark.sql, but the problem with spark.sql() is it does not p...
- 3170 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @reachrishav, In Databricks SQL, you can replicate SQL Server's conditional logic using `CASE` statements and `MERGE` operations. Since Databricks SQL doesn't support `IF EXISTS` directly, you can create a temporary view to check your condition an...
- 0 kudos
- 6225 Views
- 1 replies
- 1 kudos
List and iterate over files in Databricks workspace
Hi DE Community,I need to be able to list/iterate over a set of files in a specific directory within the Databricks workspace. For example:"/Workspace/SharedFiles/path/to/file_1"..."/Workspace/SharedFiles/path/to/file_n"Thanks for your direction and ...
- 6225 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @semsim ,You can use File system utility (dbutils.fs)Databricks Utilities (dbutils) reference | Databricks on AWSWork with files on Databricks | Databricks on AWSdbutils.fs.ls("file:/Workspace/Users/<user-folder>/")
- 1 kudos
- 3254 Views
- 1 replies
- 0 kudos
DLT - Get pipeline_id and update_id
I need to insert pipeline_id and update_id in my Delta Live Table (DLT), the point being to know which pipeline created which row. How can I obtain this information?I know you can get job_id and run_id from widgets but I don't know if these are the s...
- 3254 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Zeruno ,Those values are rather static. Maybe you can design process that as a first step will extract information from List Pipepline API and save them in delta table.List pipelines | Pipelines API | REST API reference | Databricks on AWSThan in...
- 0 kudos
- 1179 Views
- 2 replies
- 0 kudos
csv file processing
whats best possible solution to process csv file in databricks.Please consider scalability,optimization, qa give m best solution...
- 1179 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @vadi, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will help...
- 0 kudos
- 3121 Views
- 1 replies
- 0 kudos
Setup dbt-core with Azure Entra ID
Hey team, I'm trying to standardize the development environment setup in our team. I've written up a shell script that I want our devs to run in WSL2 after setup. The shell script does the following:1. setup Azure CLI - install and authenticate2. Ins...
- 3121 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @Retired_mod thanks for the response. I persisted a little more with the logs and the issue appears to be related to WSL2 not having a backend credential manager to handle management of tokens supplied by the OAuth process. To be honest, this is ...
- 0 kudos
- 1071 Views
- 0 replies
- 0 kudos
Azure Databricks Data Load
Hi All,I have 10 tables present on On-prem MS SQL DB and want to load 10 table data incrementally into Bronze delta table as append only. From Bronze to Silver , using merge query I want to load latest record into Silver delta table . Whatever latest...
- 1071 Views
- 0 replies
- 0 kudos
- 8250 Views
- 3 replies
- 2 kudos
Resolved! Configure multiple source paths for auto loader
I am currently using two streams to monitor data in two different containers on an Azure storage account. Is there any way to configure an autoloader to read from two different locations? The schemas of the files are identical.
- 8250 Views
- 3 replies
- 2 kudos
- 2 kudos
@Morten Stakkeland​ :Yes, it's possible to configure an autoloader to read from multiple locations.You can define multiple CloudFiles sources for the autoloader, each pointing to a different container in the same storage account. In your case, since ...
- 2 kudos
- 31667 Views
- 7 replies
- 4 kudos
Resolved! use job parameters in scripts
Hi CommunityI made some research, but I wasn't lucky, and I'm a bit surprised I can't find anything about it.So, I would simply access the job parameters when using python scripts (not notebooks).My flow doesn't use notebooks, but I still need to dri...
- 31667 Views
- 7 replies
- 4 kudos
- 4 kudos
The only working workaround I found has been provided in another threadRe: Retrieve job-level parameters in Python - Databricks Community - 44720I will repost it here (thanks @julio_resende )You need to push down your parameters to a task level. Eg:C...
- 4 kudos
- 2226 Views
- 2 replies
- 0 kudos
How to know actual size of delta and non-delta tables also the no of files actually exists on S3.
I have set of delta and non-delta tables, their data is on AWS s3, I want to know the total size of my delta and non-delta table in actual excluding files belongs to operations DELETE, VACCUM etc. , also I need to know how much files each delta versi...
- 2226 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Shiva3, To manage the size of Delta and non-Delta tables on AWS S3, excluding irrelevant files, start by using `DESCRIBE HISTORY` to monitor Delta table metrics and `VACUUM` to clean up old files, setting a retention period as needed. For non-Del...
- 0 kudos
- 3411 Views
- 1 replies
- 0 kudos
Databricks workflow creation using databricks sdk programming
I am trying to create Databricks workflow using sdk programming. I am successful in this but struck at how to use libraries whl files in the task from yaml file means which sdk package or code to be used to associate library whl in the notebook/pytho...
- 3411 Views
- 1 replies
- 0 kudos
- 3413 Views
- 1 replies
- 1 kudos
Databricks job stalls without error, unable to pin-point error, all compute metrics seem ok
I have a job that gets stuck on "Determining DBIO File fragment" and I have not been able to figure out why this job keeps getting stuck. I monitor the job cluster metrics through out the job and it doesnt seem like its hitting any bottlenecks with m...
- 3413 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @a-sky, This message indicates that Databricks is figuring out which file fragments are cached, which can be slow, especially with frequent cluster scaling. To address this, you can try disabling delta caching with `spark.conf.set("spark.databrick...
- 1 kudos
- 4110 Views
- 1 replies
- 0 kudos
Issue with round off value while loading to delta table
I have a float dataype column in delta table and data to be loaded should be rounded off to 2 decimal places. I'm casting the column to DECIMAL(18,10) type and then using round function from pyspark.sql.function for rounding off values to 2 decimal p...
- 4110 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @DMehmuda, The issue arises because floating-point numbers in Delta tables can retain more decimal places than expected. To ensure values are stored with the correct precision, explicitly cast the column to `DECIMAL(18,2)` before writing to the De...
- 0 kudos
- 2423 Views
- 1 replies
- 0 kudos
Handling Concurrent Writes to a Delta Table by delta-rs and Databricks Spark Job
Hi @dennyglee, @Retired_mod.If I am writing data into a Delta table using delta-rs and a Databricks job, but I lose some transactions, how can I handle this?Given that Databricks runs a commit service and delta-rs uses DynamoDB for transaction logs, ...
- 2423 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @prem14f, To manage lost transactions, implement retry logic with automatic retries and ensure idempotent writes to avoid duplication. For concurrent writers, use optimistic concurrency control, which allows for conflict detection and resolution d...
- 0 kudos
- 5565 Views
- 1 replies
- 1 kudos
Resolved! Connection error when accessing dbutils secrets
We have daily running pipelines that need to access dbutils secrets for API keys. However, the dbutils.secrets.get function within our python code we get the following error:org.apache.http.conn.HttpHostConnectException: Connect to us-central1.gcp.da...
- 5565 Views
- 1 replies
- 1 kudos
- 3123 Views
- 3 replies
- 5 kudos
DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account
Hi there!I believe I might have identified a bug with DBR 15.4 LTS Beta. The basic task of saving data to a delta table, as well as an even more basic operation of saving a file to cloud storage, is failing on 15.4, but working perfectly fine on 15.3...
- 3123 Views
- 3 replies
- 5 kudos
- 5 kudos
We have the same issue since yesterday (6/8/2024), running on DBR 15.3 or 15.4 LTS Beta. It seems to have something to do with large table's indeed. Tried with multiple .partition sizes.
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
