- 3009 Views
- 3 replies
- 0 kudos
How to Connect Databricks with Web and Mobile Apps
Hi everyone,I’m exploring ways to leverage Databricks for building data-driven web and mobile applications and wanted to get some insights from this community. Databricks is great for processing large datasets, running analytics, and building machine...
- 3009 Views
- 3 replies
- 0 kudos
- 0 kudos
Check Databricks Apps - you pass databricks resources and then use databricks-sdk to interact with them.
- 0 kudos
- 1465 Views
- 1 replies
- 3 kudos
Data Engineering Lessons
Getting into the data space can feel overwhelming, with so many tools, terms, and technologies. But after years inExpect failure. Design for it.Jobs will fail. The data will be late. Build systems that can recover gracefully, and continually monitor ...
- 1465 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @boitumelodikoko A few more principles I always share with people entering the data space:Observability is non-negotiable.If you can’t see what your pipelines are doing, you can’t fix what breaks.Good logging, metrics, and alerts save countless ho...
- 3 kudos
- 2241 Views
- 2 replies
- 0 kudos
On Demand Pool Configuration & Policy definition
I'm using Job cluster and created compute policies for library management and now I'm trying to use pools in databricks. I'm getting error like this : Cluster validation error: Validation failed for azure_attributes.spot_bid_max_price from pool, the ...
- 2241 Views
- 2 replies
- 0 kudos
- 0 kudos
This error occurs because instance pools require a concrete spot bid max price value, even if the cluster policy marks it as unlimited. Set an explicit value (e.g., 100) directly in the instance pool configuration, or switch the pool to on-demand nod...
- 0 kudos
- 8517 Views
- 2 replies
- 2 kudos
How to resolve "cannot import name 'Iterable' from 'collections'" error?
I'm running a DBR/Spark job using a container. I've set docker_image.url to `docker.io/databricksruntime/standard:13.3-LTS`, as well as the Spark env var `DATABRICKS_RUNTIME_VERSION=13.3`. At runtime, however, I'm encountering this error: ImportError...
- 8517 Views
- 2 replies
- 2 kudos
- 2 kudos
Go to Compute → Your Cluster / Job ComputeChange Databricks Runtime to:Databricks Runtime 13.3 LTSRe-run your job with the same container.
- 2 kudos
- 2395 Views
- 6 replies
- 4 kudos
Resolved! Extract all users from Databricks Groups
Hey everyone,we are trying to get an overview of all users that we have in our databricks groups. We have tried to do so with the REST API as well as the SQL-queries (with normal developer accounts as well as workspace administrator accounts). The pr...
- 2395 Views
- 6 replies
- 4 kudos
- 4 kudos
Use the Databricks SQL system users tableSELECT * FROM system.usersOnly shows fully provisioned users Users pending invitation may not appear.
- 4 kudos
- 1036 Views
- 1 replies
- 1 kudos
Help me with the databricks streamlit application related doubt
Hi Databricks community ,Hi I have a doubt regarding databricks streamlit application . I have a databricks streamlit application that takes input values from the user through streamlit UI. Now I want to store these input values in a delta table in U...
- 1036 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @gokkul ,Your app service principal needs to have a proper permission to write to UC table. You also need to use python databricks sdk to interact with UC object (i.e read/save a table).You can get some inspiration from following databricks cookbo...
- 1 kudos
- 6103 Views
- 7 replies
- 4 kudos
Resolved! How to Optimize Data Pipeline Development on Databricks for Large-Scale Workloads?
Hi everyone,I’m working on building and optimizing data pipelines in Databricks, especially for large-scale workloads, and I want to learn from others who have hands-on experience with performance tuning, architecture decisions, and best practices.I’...
- 6103 Views
- 7 replies
- 4 kudos
- 4 kudos
Optimizing Databricks pipelines for large-scale workloads mostly comes down to smart architecture + efficient Spark practices.Key tips from real-world users:Use Delta Lake – for ACID transactions, incremental updates, and schema enforcement.Partition...
- 4 kudos
- 5095 Views
- 5 replies
- 4 kudos
Find value in any column in a table
Hi,I'm not sure if this is a possible scenario, but is there, by any chance a way to query all the columns of a table for searching a value? Explanation: I want to search for a specific value in all the columns of a databricks table. I don't know whi...
- 5095 Views
- 5 replies
- 4 kudos
- 653 Views
- 0 replies
- 1 kudos
Databricks Advent Calendar Edition
Santa is coming to Lakehouse Town! From now until Christmas, I’ll be your guide on a Databricks journey – sharing one powerful Databricks feature every single day. https://www.linkedin.com/posts/bianca-stratulat_databricks-lakehouse-dataengineering-a...
- 653 Views
- 0 replies
- 1 kudos
- 689 Views
- 1 replies
- 1 kudos
How to build a chatbot in Databricks for ad‑hoc analytics questions?
Hi everyone,I’m exploring the idea of creating a chatbot within Databricks that can handle ad‑hoc business analytics queries. For example, I’d like users to be able to ask questions such as:“How many sales did we have in 2025?”“Which products had the...
- 689 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @laeforceable , if you have data in databricks platform then you can use Genie Room Capability to ask questions related to data..
- 1 kudos
- 2177 Views
- 3 replies
- 2 kudos
Resolved! Azure databrics Learning tutorials ADB+SQL,ADB+PYSPARK,ADB+PYTHON
Suggest me the best learning tutorials of Azure databricks with the combinations of pyspark,python,sql is the any learning web based tutorials from Databricks suggest me the best one from scratch to advanced
- 2177 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @vijaypodili ,I can recommend Data Engineering Learning Path on Databricks Academy:https://customer-academy.databricks.com/On Udemy, there's an excellent course that covers all the important aspects of working with Databricks on a daily basis:Data...
- 2 kudos
- 6399 Views
- 9 replies
- 0 kudos
Resolved! Programatic selection of serverless compute for notebooks environment version
Hello,I have a case where I am executing notebooks from an external system using databricks api /api/2.2/jobs/runs/submit. This has always been non problematic with the job compute, but due to the quite recent serverless for notebooks support being i...
- 6399 Views
- 9 replies
- 0 kudos
- 0 kudos
Not so sure about the general but in eu-west-3, we could specify the serverless environment version using DAB using the `environments` block and `spec` params:resources: jobs: pipeline: name: "[${bundle.target}]pipeline" webhook_not...
- 0 kudos
- 2563 Views
- 2 replies
- 0 kudos
informatica jobs from data bricks
Hi TeamHow can we call informatica jobs from data bricks? could you please suggest on this.Regards,Phanindra
- 2563 Views
- 2 replies
- 0 kudos
- 0 kudos
unsure how above answer helps here @Phani1 - The only way I could think of is - call Informatica jobs (specifically Informatica Cloud Data Integration (CDI) mappings or tasks) from Databricks by leveraging REST APIs.Direct API - Trigger an Informatic...
- 0 kudos
- 1835 Views
- 5 replies
- 6 kudos
Resolved! Synchronising metadata (e.g., tags) across schemas under Unity Catalog (Azure)
Hello all,I hope you are doing great!I want to synchronise metadata (e.g., description, comments, tags) across schemas under the Unity Catalog (e.g., test.dev, test.uat). For example, under the schema test.dev, there is a sales table with multiple co...
- 1835 Views
- 5 replies
- 6 kudos
- 6 kudos
It's completely fine, and I do understand. Thank you for your time and effort here!
- 6 kudos
- 1444 Views
- 3 replies
- 1 kudos
Resolved! Databricks Scenarios
I’m a data engineer with some experience in Databricks. I’m looking for real-life scenarios that are commonly encountered by data engineers. Could you also provide details on how to implement these scenarios?
- 1444 Views
- 3 replies
- 1 kudos
- 1 kudos
Generic topic. Here are few latest article to help you on thishttps://community.databricks.com/t5/get-started-guides/getting-started-with-databricks-build-a-simple-lakehouse/tac-p/139492#M29https://community.databricks.com/t5/announcements/big-book-o...
- 1 kudos
