- 695 Views
- 1 replies
- 0 kudos
How to use serverless clusters in DAB deployments with Unity Catalog in private network?
Hi everyone,I’m deploying Jobs and Pipelines using Databricks Asset Bundles (DAB) in an Azure Databricks workspace configured with private networking. I’m trying to use serverless compute for some workloads, but I’m running into issues when Unity Cat...
- 695 Views
- 1 replies
- 0 kudos
- 0 kudos
A lot of questions Concerning usage of serverless clusters in databricks.yml and assuming you're using those clusters in jobs, you must define them in the job definition. Take a look here: https://github.com/databricks/bundle-examples/tree/main/know...
- 0 kudos
- 408 Views
- 2 replies
- 0 kudos
Resolved! Loading CSV from private S3 bucket
Trying to load a csv file from a private S3 bucketplease clarify requirements to do this- Can I do it in community edition (if yes then how)? How to do it in premium version?I have IAM role and I also access key & secret
- 408 Views
- 2 replies
- 0 kudos
- 0 kudos
Assuming you have these pre-requisites: A private S3 bucket (e.g., s3://my-private-bucket/data/file.csv)An IAM user or role with access (list/get) to that bucketThe AWS Access Key ID and Secret Access Key (client and secret)The most straightforward w...
- 0 kudos
- 28237 Views
- 14 replies
- 12 kudos
Resolved! dbutils or other magic way to get notebook name or cell title inside notebook cell
Not sure it exists but maybe there is some trick to get directly from python code:NotebookNameCellTitlejust working on some logger script shared between notebooks and it could make my life a bit easier
- 28237 Views
- 14 replies
- 12 kudos
- 12 kudos
I got the solution to work in terms of printing the notebook that I was running; however, what if you have notebook A that calls a function that prints the notebook name, and you run notebook B that %runs notebook A? I get the notebook B's name when...
- 12 kudos
- 913 Views
- 3 replies
- 4 kudos
Resolved! DATA_SOURCE_NOT_FOUND Error with MongoDB (Suggestions in other similar posts have not worked)
I am trying to load data from MongoDB into Spark. I am using the Community/Free version of DataBricks so my Jupiter Notebook is in a Chrome browser.Here is my code:from pyspark.sql import SparkSession spark = SparkSession.builder \ .config("spar...
- 913 Views
- 3 replies
- 4 kudos
- 4 kudos
Hey @kahrees , Good Day! I tested this internally, and I was able to reproduce the issue. Screenshot below: You’re getting [DATA_SOURCE_NOT_FOUND] ... mongodb because the MongoDB Spark connector jar isn’t actually on your cluster’s classpath. On D...
- 4 kudos
- 852 Views
- 3 replies
- 2 kudos
Resolved! Does VACUUM on Delta Lake also clean Iceberg metadata when using Iceberg Uniform feature?
I'm working with Delta tables using the Iceberg Uniform feature to enable Iceberg-compatible reads. I’m trying to understand how metadata cleanup works in this setup.Specifically, does the VACUUM operation—which removes old Delta Lake metadata based ...
- 852 Views
- 3 replies
- 2 kudos
- 2 kudos
Here’s how to approach cleaning and maintaining Apache Iceberg metadata on Databricks, and how it differs from Delta workflows. First, know your table type For Unity Catalog–managed Iceberg tables, Databricks runs table maintenance for you (predicti...
- 2 kudos
- 862 Views
- 1 replies
- 1 kudos
Resolved! Should I enable Liquid Clustering based on table size distribution?
Hi everyone,I’m evaluating whether Liquid Clustering would be beneficial for the tables based on the sizes. Below is the size distribution of tables in my environment:Size Bucket Table Count Large (> 1 TB)3Medium (10 GB – 1 TB)284Small (< 10 GB)17,26...
- 862 Views
- 1 replies
- 1 kudos
- 1 kudos
Greetings @pooja_bhumandla Based on your size distribution, enabling Liquid Clustering can provide meaningful gains—but you’ll get the highest ROI by prioritizing your medium and large tables first and selectively applying it to small tables where q...
- 1 kudos
- 859 Views
- 5 replies
- 1 kudos
Resolved! Location of spark.scheduler.allocation.file
In DBR 164.LTS, I am trying to add the following Spark config: spark.scheduler.allocation.file: file:/Workspace/init/fairscheduler.xmlBut the all purpose cluster is throwing this error Spark error: Driver down cause: com.databricks.backend.daemon.dri...
- 859 Views
- 5 replies
- 1 kudos
- 1 kudos
Here's some solutions without using DBFS.. Yes, there are solutions for using the Spark scheduler allocation file on Databricks without DBFS, but options are limited and depend on your environment and access controls. Alternatives to DBFS for Schedu...
- 1 kudos
- 769 Views
- 4 replies
- 1 kudos
Resolved! Is there any way to run jobs from github actions and catch the results?
Hi all,Is there any way to run jobs from github actions and catch the results?Of course, I can do this if I use the API or CLI.But I found the actions for notebook: https://github.com/marketplace/actions/run-databricks-notebook Compared to this, wri...
- 769 Views
- 4 replies
- 1 kudos
- 1 kudos
OK, thank you for your advices, I will consider to use asset bundles for this.
- 1 kudos
- 500 Views
- 2 replies
- 0 kudos
What is the Best Practice of Maintaining the Delta table loaded in Streaming?
Hi Team,We have our Bronze(append) Silver(append) and Gold(merge) Tables loaded using spark streaming continuously with trigger as processing time(3 secs).We Also Run our Maintenance Job on the Table like OPTIMIZE,VACCUM and we perform DELETE for som...
- 500 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Mark,But the real problem is our streaming job runs 365 days 24 *7 and we cant afford any further latency to our data flowing to gold layer. We don't have any window to pause or slower our streaming and we continuously get the data feed actually s...
- 0 kudos
- 255 Views
- 1 replies
- 0 kudos
DLT PARAMETERIZATION FROM JOBS PARAMETERS
I have created a dlt pipeline notebook which creates tables based on a config file that has the configuration of the tables that need to be created . now what i want is i want to run my pipeline every 30 min for 4 tables from config and every 3 hours...
- 255 Views
- 1 replies
- 0 kudos
- 0 kudos
Define "parameters" in job as usual and then, try to capture them in DLT by using similar code to this one:dlt.conf.get("PARAMETER_NAME", "PARAMETER_DEFAULT_VALUE")It should get parameter values from job if value exists, otherwise it'll set the defau...
- 0 kudos
- 305 Views
- 2 replies
- 1 kudos
Issue with Unity Catlog on Azure
when I create Databricks workspace on Azure and tries to login on https://accounts.azuredatabricks.net/ it redirects to my workspace. Where as on Azure subscription I am the owner, I created this azure subscription and Databricks workspace is also cr...
- 305 Views
- 2 replies
- 1 kudos
- 1 kudos
Clearly, you don't have "admin account" permissions. Try to click in the workspace drop-down and then, check if you can see and click in "Manage Account" to confirm BUT it will be very likely you are not allowed to access.You must be Azure Global Adm...
- 1 kudos
- 2457 Views
- 2 replies
- 0 kudos
Resolved! Optimizing Azure Functions for Performance and Cost with Variable Workloads
Hey, everyone!!I use Azure Functions in a project where the workloads change a lot. Sometimes it's quiet, and other times we get a lot of traffic.Azure Functions is very scalable, but I've had some trouble with cold starts and keeping costs down.I'm ...
- 2457 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey!!!Cold starts on Azure Functions Premium can still bite if your instances go idle long enough — even with pre-warmed instances.What usually helps is bumping the `preWarmedInstanceCount` to at least 1 per plan (so there’s always a warm worker), an...
- 0 kudos
- 741 Views
- 2 replies
- 2 kudos
Resolved! DQX usage outside Databricks
Hello, When evaluating data quality frameworks for PySpark pipelines, I came across DQX. I noticed it's available on PyPI (databricks-labs-dqx) and GitHub, which is great for accessibility.However, I'm trying to understand the licensing requirements....
- 741 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks a lot for the quick response, @ManojkMohan! This was very helpful.I'll keep this in mind.
- 2 kudos
- 658 Views
- 3 replies
- 2 kudos
Resolved! Moving tables between pipelines in production
We are testing an ingestion from kafka to databricks using a streaming table. The streaming table was created by a DAB deployed to "production" which runs as a service principal. This means the service principal is the "owner" of the table.We now wan...
- 658 Views
- 3 replies
- 2 kudos
- 2 kudos
You’ve hit two limitations:Streaming tables don’t allow SET OWNER – ownership cannot be changed.Lakeflow pipeline ID changes require pipeline-level permissions – if you’re not the pipeline owner, you can’t run ALTER STREAMING TABLE ... SET PIPELINE_I...
- 2 kudos
- 1806 Views
- 4 replies
- 3 kudos
When working with large data sets in Databricks, what are best practices to avoid memory out-of-memo
How can I optimize Databricks to handle large datasets without running into memory or performance problems?
- 1806 Views
- 4 replies
- 3 kudos
- 3 kudos
Hey! Great question — I’ve run into this issue quite a few times while working with large datasets in Databricks, and out-of-memory errors can be a real headache. One of the biggest things that helps is making sure your cluster configuration matches ...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
