- 49 Views
- 2 replies
- 0 kudos
Databricks Apps - Deployment process - Need help !!
Hi all,Do we have any guidelines or a CI/CD approach for deploying a Databricks app from one environment (e.g., DEV) to PROD?We are currently facing the following challenge:We created a Databricks app in the DEV environment through the UI. At runtime...
- 49 Views
- 2 replies
- 0 kudos
- 0 kudos
@Sanjeeb2024 Definitely DABs are good approach here. One thing I would add here is to keep the deployment identity and App run time Identity separate.For DEV -> PROD, the recommended pattern is to deploy the app with DAB from CI/CD, ideally using a d...
- 0 kudos
- 16 Views
- 0 replies
- 0 kudos
Databricks vs Snowflake Pyspark Performance
Hi experts, now the competition between cloud providers are fierce and brutal , I come across this post which compares Pyspark performance on databricks and Snowflakehttps://l1nk.dev/1y33h0z although the metics tested are not rigorous and exhaustive...
- 16 Views
- 0 replies
- 0 kudos
- 304 Views
- 1 replies
- 2 kudos
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.Earlier, I used to think streaming meant processing every record the ...
- 304 Views
- 1 replies
- 2 kudos
- 2 kudos
Absolutely agree. The key takeaway for me is that streaming doesn't automatically mean ultra-low latency. Choosing between processingTime and AvailableNow, and lower-latency approaches, should really come down to the business SLA, data arrival patter...
- 2 kudos
- 38 Views
- 1 replies
- 0 kudos
What is a Data Skipping in Delta Lake?
Hi everyone,I’m learning about Delta Lake performance and came across data skipping.I understand that it can help Databricks avoid reading unnecessary data when running queries, but I’d like to understand its purpose more clearly.For example, if an o...
- 38 Views
- 1 replies
- 0 kudos
- 0 kudos
@gowri_databrick Data skipping is a built in optimization that uses file-level statistics (minimum values, maximum values and null counts) to automatically skip reading data files that don't contain relevant data for your query. When you write data t...
- 0 kudos
- 93 Views
- 2 replies
- 1 kudos
Clarification on Automating Serverless Compute Permissions via SDK/API
Hi Team,I'm trying to automate granting Can Use permissions on Default Interactive Compute for workspace groups across multiple Databricks workspaces. While reviewing the Serverless Compute access control documentation and the Access Control Rule Set...
- 93 Views
- 2 replies
- 1 kudos
- 1 kudos
@saicharan1 you found a gap in the Azure documentationDatabricks creates Default Interactive Compute and Default Automated Compute objects in each workspace. Workspace admins manage their permissions, so you must grant CAN_USE in each workspace, even...
- 1 kudos
- 47 Views
- 0 replies
- 0 kudos
Who is The Best Cold Email Service Provider in 2026?
I have been working in the SMMA field for more than 5 years. So I have very strong experience about that.I tried many cold email providers; I use more than 10k emails from every provider, but everyone has a common problem. They will now give you a lo...
- 47 Views
- 0 replies
- 0 kudos
- 195 Views
- 5 replies
- 3 kudos
Resolved! Error While Running Workloads on Databricks Free Edition – Serverless Compute
Hi Team,I am using the Databricks Free Edition and I am facing an issue while running workloads using Serverless Compute.When I try to run the workload, I receive the following error:“An error occurred while trying to provision serverless compute. Pl...
- 195 Views
- 5 replies
- 3 kudos
- 61 Views
- 0 replies
- 0 kudos
Lakeflow connect Ingestion pipeline notification for gateway pipeline
Hi Guys,As you guys know that when we are building lake flow connect ingestion pipeline in UI. The pipeline consist both gateway pipeline and ingestion pipeline together. We have notification for the ingestion pipeline but not for gateway pipeline. I...
- 61 Views
- 0 replies
- 0 kudos
- 173 Views
- 7 replies
- 1 kudos
Lakaflow connect Data Ingestion gatway pipeline fail often and restarts automatically
Hello Team,I have designed Lakeflow connect data ingestion pipeline using managed connector SQL Server. The source is SQL Server created the connection and Change Tracking enabled at the source level. Also, I have give required permission and everyth...
- 173 Views
- 7 replies
- 1 kudos
- 1 kudos
@srikanthp24 Looking at the details, it does look like minimal volume.I could check on the chunking setting but is not available at the pipeline level currently. It make sense to raise this to Databricks as Support ticket for triaging further by look...
- 1 kudos
- 85 Views
- 3 replies
- 0 kudos
VS Code and Connect - Manual Environment Setup
The current version of the extension forces usage of certain python packaging / project tools. I used to be able to use the extension with my project which uses pixi, by setting up the dependencies myself in pyproject.toml / pixi.toml.The current ver...
- 85 Views
- 3 replies
- 0 kudos
- 0 kudos
I’m running into the same kind of issue with projects that already have their own environment setup. Being able to manually select the Python environment/kernel without the extension forcing uv changes would make Connect much more flexible.
- 0 kudos
- 146 Views
- 2 replies
- 3 kudos
Resolved! Issue with auto complete in the datarbicks notebook
I'm using sql to run a lot of queries in the notebook. Since this week it started to pop up with the Suggested code every time I made an adjustment to the code and my cursor keeps changing between the green and red highlighted lines (previously it on...
- 146 Views
- 2 replies
- 3 kudos
- 3 kudos
@leahli I was facing similar issue before with red/green diff appearing frequently. These are related to Genie Code Quick fix, not the normal grey autocomplete suggestions.With Settings → Developer → Genie Code Quick Fix = On, a failed SQL statement ...
- 3 kudos
- 154 Views
- 5 replies
- 3 kudos
What is a Warehouse in Databricks SQL?
Hi everyone,I’m learning Databricks SQL and came across SQL warehouses.I understand that a SQL warehouse provides compute resources for running SQL queries, but I’m not clear about how it is different from other compute options in Databricks.When sho...
- 154 Views
- 5 replies
- 3 kudos
- 3 kudos
Hello, @gowri_databrick Here is a quick example that makes it super clear.Try creating a new notebook and changing the attached compute resource to Serverless SQL Warehouse.You'll immediately get an error saying: "SQL Warehouse only support SQL and M...
- 3 kudos
- 2657 Views
- 3 replies
- 2 kudos
Resolved! Migrating Talend ETL Jobs to Databricks – Best Practices & Challenges
Hi All,I’m currently working on a Proof of Concept (POC) to migrate existing Talend ETL jobs to Databricks. The goal is to leverage Databricks for data processing and orchestration while moving away from Talend.I’d appreciate insights on the followin...
- 2657 Views
- 3 replies
- 2 kudos
- 2 kudos
this page is no longer available https://blog.aidetic.in/replaced-multiple-etl-tools-like-talend-with-databricks-5841e87bdaeaDo you have any other resources
- 2 kudos
- 91 Views
- 1 replies
- 0 kudos
Improvement in DAB for alert_task
Hello,I'm facing with a difference between DAB and the UI concerning alert_task on job ressource.I have an alert with a SQL query using some parameters, installed by DAB. When I want to use it un job ressource in DAB we can't use "parameters" in aler...
- 91 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, This is a known limitation today. The official documentation for the SQL alert task confirms it under the Limitations section: "SQL alert tasks do not support parameters. If you need to use parameterised queries, consider using a SQL task inste...
- 0 kudos
- 174 Views
- 4 replies
- 3 kudos
Partitioning vs Liquid Clustering (per-table):
Can PARTITION BY and CLUSTER BY (Liquid Clustering) be used simultaneously on the same table? If we use only PARTITION BY, is there a negative performance impact on materialized-view refreshes in Silver/Gold? Since materialized views read only increm...
- 174 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi, as others have said they can't coexist. The docs are clear: "Clustering is not compatible with partitioning or ZORDER." You pick one or the other. If you have existing partitioned tables, you can convert them (DBR 18.1+): ALTER TABLE my_table REP...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
